GoogLeNet / Inception

A 22-layer CNN built from Inception modules: parallel 1×1, 3×3, 5×5 and pooling branches whose outputs are concatenated, with cheap 1×1 convolutions shrinking the depth first. It won ILSVRC-2014 with 6.67% top-5 error using about 6.8M parameters, roughly a ninth of AlexNet's 60M.

12 min read1258 wordscore0 viewsupdated 2026-10-11

Definition

GoogLeNet is the 22-layer convolutional network of Szegedy et al. (2014), "Going deeper with convolutions". It is a stack of Inception modules. Each module runs a 1×11\times1, a 3×33\times3 and a 5×55\times5 convolution and a 3×33\times3 max-pool in parallel on the same input and concatenates the four outputs along the channel axis. Before the expensive 3×33\times3 and 5×55\times5 convolutions, a 1×11\times1 convolution first reduces the number of channels. "Inception" is the module and the family of networks; GoogLeNet is the specific network the paper entered in ILSVRC-2014, its name a nod to LeNet.

After AlexNet (2012) the obvious way to get better ImageNet accuracy was to make the network bigger: more layers and more channels. That has two costs. More parameters overfit more easily, and compute grows quadratically when two chained conv layers are both widened. If you double the channels of two stacked 3×33\times3 layers, the second one costs 4×4\times as much. Szegedy and colleagues set themselves a budget of about 1.5 billion multiply-adds per image, so that the network could be used in practice, and asked how to spend it well.

Two ideas shaped the answer. Lin, Chen and Yan's Network-in-Network (2013) had used 1×11\times1 convolutions as small per-pixel MLPs across channels. GoogLeNet uses them mainly for dimension reduction: a 1×11\times1 conv from 192 to 16 channels before a 5×55\times5 conv cuts that conv's cost by about 10×. The second idea came from Arora et al.'s analysis of sparse networks: an ideal network would be sparsely connected, with correlated units grouped together, but sparse matrix arithmetic is slow on hardware. The Inception module approximates such a sparse structure with dense building blocks. Several filter sizes look at the same location in parallel, and the next layer decides which of them to use.

GoogLeNet won the ILSVRC-2014 classification task with 6.67% top-5 error on the test set, ahead of VGG (7.32%). That number comes from an ensemble of 7 networks with 144 crops per image. A single network with a single crop scored 10.07% top-5 on the validation set. The paper reports that GoogLeNet uses 12 times fewer parameters than the 2012 winner of Krizhevsky et al. (about 60M).

At a glance

Paper
Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, Rabinovich, CVPR 2015 (arXiv Sept. 2014)
Depth
22 layers with parameters (27 counting pooling), 9 Inception modules
Parameters
≈6.8M (sum of the paper's Table 1); torchvision's port: 6,624,904
Compute
≈1.5G multiply-adds at 224×224224\times224
Result
ILSVRC-2014 winner, 6.67% top-5 test error (7-model, 144-crop ensemble)
Key tricks
parallel branches, 1×11\times1 bottlenecks, global average pooling, two auxiliary classifiers during training
In PyTorch
torchvision.models.googlenet(weights="IMAGENET1K_V1"): 69.78% top-1, 89.53% top-5

The Inception module

Let xx be a feature map with CinC_{in} channels on an H×WH\times W grid. The module computes four branches that all keep the grid size (stride 1, "same" padding) and concatenates them:

y=[  Cn11×1(x)  ∥  Cn33×3(Rr3(x))  ∥  Cn55×5(Rr5(x))  ∥  Rp(maxpool3×3(x))  ],Cout=n1+n3+n5+py=\Big[\;C^{1\times1}_{n_1}(x)\;\Big\|\;C^{3\times3}_{n_3}\big(R_{r_3}(x)\big)\;\Big\|\;C^{5\times5}_{n_5}\big(R_{r_5}(x)\big)\;\Big\|\;R_{p}\big(\mathrm{maxpool}_{3\times3}(x)\big)\;\Big],\qquad C_{out}=n_1+n_3+n_5+p
Cnk×kC^{k\times k}_{n}
a k×kk\times k convolution with nn output channels, followed by ReLU
RrR_{r}
a 1×11\times1 "reduce" convolution to rr channels (+ ReLU): the bottleneck
∥\|
concatenation along the channel axis (the paper's "filter concatenation")
pp
"pool proj": the 1×11\times1 conv after the max-pool, which keeps the pool branch from passing all CinC_{in} channels through

Why the reductions matter becomes clear from the cost of a convolution. A k×kk\times k conv from CinC_{in} to CoutC_{out} channels on an H×WH\times W output has k2CinCoutk^2C_{in}C_{out} weights and needs k2CinCoutHWk^2C_{in}C_{out}HW multiply-adds. Putting a 1×11\times1 conv to rr channels in front replaces k2CinCoutk^2C_{in}C_{out} by Cinr+k2rCoutC_{in}r+k^2rC_{out}:

cost with reductioncost without=Cin r+k2 r Coutk2 Cin Cout=rk2Cout+rCin\frac{\text{cost with reduction}}{\text{cost without}}=\frac{C_{in}\,r+k^2\,r\,C_{out}}{k^2\,C_{in}\,C_{out}}=\frac{r}{k^2C_{out}}+\frac{r}{C_{in}}
rr
channels after the 1×11\times1 reduction (r3r_3 or r5r_5 in the module)
kk
kernel size of the expensive branch (3 or 5)
r/Cinr/C_{in}
dominant term: the reduction ratio. Module 3a's 5×55\times5 branch has r/Cin=16/192r/C_{in}=16/192

The pooling branch has a second problem. Max-pooling keeps the channel count, so without a projection every module would output at least as many channels as it receives, plus the conv branches. Depth would then grow from module to module, and the cost of each following module with it. The figure's "naive" mode shows this.

Backpropagation through a module is simple. The concatenation splits the incoming gradient g=∂L/∂yg=\partial\mathcal L/\partial y into four channel slices g1,…,g4g_1,\dots,g_4, one per branch. Because all four branches read the same xx, their input gradients add: ∂L/∂x=∑bJb⊤gb\partial\mathcal L/\partial x=\sum_b J_b^\top g_b, where JbJ_b is the Jacobian of branch bb. The max-pool branch routes its share only to the input position that was the maximum of each window.

GoogLeNet, 224×224 input · bars: output depth (top), mult-adds (bottom) · click a stagec164118Mc2192359.7M3a256128M3b480304.3M4a51273.6M4b51287.9M4c51299.9M4d528118.5M4e832170M5a83251.1M5b102470.7Mfc10001Minception (3a): 192×28² → 256×28² · 163,328 params · 128.0M mult-addsinput 192×28×281×1 → 6412,288 p9.6M · 64 ch1×1 → 963×3 → 128129,024 p101.2M · 128 ch1×1 → 165×5 → 3215,872 p12.4M · 32 ch3×3 max-pool → 1921×1 → 326,144 p4.8M · 32 chfilter concat: 64 + 128 + 32 + 32 = 256 channelstotal: 6,990,272 params · 1.58 G mult-addspaper Table 1 lists ≈6.8M params and ≈1.5G ops (its conv1 and 4c rows are lower than the config implies)paper Table 1 row: 159K params (K = 1024 weights), 128M ops; pale boxes = 1×1 reductions
fig.GoogLeNet layer by layer, computed from the paper's Table 1 configuration at 224×224. The top strip shows each stage's output depth and multiply-adds; click a stage to see the four branches of that Inception module. With the default 1×1 reductions, module 3a maps 192×28² to 256×28² with 163,328 weights and 128.0M mult-adds (the paper lists 159K = 159.5×1024 and 128M), and the whole network has 6,990,272 weights and 1.58G mult-adds. "Naive: no reductions" removes every 1×1 reduction and pool projection: the depth grows to 4,944 channels and the network needs 97.5M weights and 12.57G mult-adds, about 14× and 7.9× as much. "torchvision variant" adds BatchNorm and the fc bias and reproduces torchvision's 6,624,904 parameters exactly. "aux classifiers" marks 4a and 4d and adds their 6.38M training-only weights.

The whole network

GoogLeNet from input to output (224×224×3)

  1. 1Stem. 7×77\times7/2 conv to 64 channels, 3×33\times3/2 max-pool, then a 1×11\times1 conv to 64 and a 3×33\times3 conv to 192, another max-pool: 192×28×28192\times28\times28. This part is a conventional CNN, because the paper found Inception modules most useful at higher layers.
  2. 2Inception 3a, 3b at 28×2828\times28 (256 and 480 channels out), max-pool to 14×1414\times14.
  3. 3Inception 4a–4e at 14×1414\times14 (512, 512, 512, 528, 832 channels), max-pool to 7×77\times7.
  4. 4Inception 5a, 5b at 7×77\times7 (832 and 1024 channels).
  5. 5Head. 7×77\times7 global average pooling to a 1024-vector, 40% dropout, one fully connected layer to 1000 classes, softmax. Replacing AlexNet's large fully connected layers with average pooling improved top-1 accuracy by about 0.6% and removed most of the parameters: the single fc layer has 1024×1000≈11024\times1000\approx1M weights.

Branch widths change with depth: the share of 3×33\times3 and 5×55\times5 filters grows in later modules, because features become more abstract and spatially spread out. Module 5b has 384 + 384 + 128 + 128 = 1024 output channels, while 3a has 64 + 128 + 32 + 32 = 256.

Auxiliary classifiers

A 22-layer network was deep for 2014, and the paper worried that gradients would not reach the lower layers well. It attached two small auxiliary classifiers to the outputs of modules 4a and 4d. Each one is a 5×55\times5/3 average pool (giving 4×4×5124\times4\times512 or 4×4×5284\times4\times528), a 1×11\times1 conv to 128 channels, a 1024-unit fully connected layer, 70% dropout and a 1000-way softmax. Their losses are added with weight 0.3:

L=Lmain+0.3 (Laux,4a+Laux,4d),∂L∂h4a=∂Lmain∂h4a+0.3 ∂Laux,4a∂h4a+0.3 ∂Laux,4d∂h4a\mathcal L=\mathcal L_{\text{main}}+0.3\,\big(\mathcal L_{\text{aux},4a}+\mathcal L_{\text{aux},4d}\big),\qquad \frac{\partial\mathcal L}{\partial h_{4a}}=\frac{\partial\mathcal L_{\text{main}}}{\partial h_{4a}}+0.3\,\frac{\partial\mathcal L_{\text{aux},4a}}{\partial h_{4a}}+0.3\,\frac{\partial\mathcal L_{\text{aux},4d}}{\partial h_{4a}}
Laux\mathcal L_{\text{aux}}
cross-entropy of an auxiliary head, computed on the same labels
h4ah_{4a}
output of module 4a; the auxiliary head on it adds a gradient path only a few layers long
0.30.3
discount weight from the paper; at inference both heads are discarded

Worked example: module 3a with and without reductions

Module 3a receives 192×28×28192\times28\times28 and has n1=64n_1=64, r3=96r_3=96, n3=128n_3=128, r5=16r_5=16, n5=32n_5=32, p=32p=32. Counting conv weights only (as the paper does), with multiply-adds over the 28×28=78428\times28=784 output positions:

BranchWith 1×1 reductionsWeightsMult-addsNaive (no reductions)WeightsMult-adds
1×1192→6412,2889.6M192→6412,2889.6M
3×3192→96, 3×3 96→128129,024101.2M3×3 192→128221,184173.4M
5×5192→16, 5×5 16→3215,87212.4M5×5 192→32153,600120.4M
poolmax-pool, 192→326,1444.8Mmax-pool only00
module256 channels out163,328128.0M416 channels out387,072303.5M
Module 3a, weights and multiply-adds per branch (computed by the figure's lib and checked by hand)

Reading the table

  1. 1The 5×55\times5 branch: 25⋅192⋅32=153,60025\cdot192\cdot32=153{,}600 weights without reduction, versus 192⋅16+25⋅16⋅32=3,072+12,800=15,872192\cdot16+25\cdot16\cdot32=3{,}072+12{,}800=15{,}872 with it. That is 9.7× fewer, matching the formula: 16/(25⋅32)+16/192=0.10316/(25\cdot32)+16/192=0.103.
  2. 2Multiply-adds are weights × 784: 15,872⋅784=12.415{,}872\cdot784=12.4M versus 153,600⋅784=120.4153{,}600\cdot784=120.4M.
  3. 3The naive pool branch has no weights, but it passes all 192 input channels through, so the module outputs 64+128+32+192=41664+128+32+192=416 channels instead of 256. Module 3b then sees 416 input channels instead of 256, and the growth compounds.
  4. 4Over the full network the naive version reaches 4,944 output channels at 5b and costs 97.5M weights and 12.57G mult-adds, against 6.99M and 1.58G with reductions (figure, "naive" mode).
python
1import torch, torch.nn as nn
2from torchvision.models import googlenet, GoogLeNet_Weights
3
4def conv_bn(cin, cout, k):
5 return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2, bias=False),
6 nn.BatchNorm2d(cout), nn.ReLU(inplace=True))
7
8class Inception(nn.Module):
9 def __init__(self, cin, n1, r3, n3, r5, n5, pp):
10 super().__init__()
11 self.b1 = conv_bn(cin, n1, 1)
12 self.b2 = nn.Sequential(conv_bn(cin, r3, 1), conv_bn(r3, n3, 3)) # 1x1 reduce, then 3x3
13 self.b3 = nn.Sequential(conv_bn(cin, r5, 1), conv_bn(r5, n5, 5)) # 1x1 reduce, then 5x5
14 self.b4 = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1), conv_bn(cin, pp, 1))
15
16 def forward(self, x): # every branch keeps H x W
17 return torch.cat([self.b1(x), self.b2(x), self.b3(x), self.b4(x)], dim=1)
18
19m3a = Inception(192, 64, 96, 128, 16, 32, 32)
20print(m3a(torch.randn(1, 192, 28, 28)).shape) # torch.Size([1, 256, 28, 28])
21print(sum(p.numel() for p in m3a.parameters())) # 164064 = 163,328 weights + 736 BN
22
23# Pretrained GoogLeNet. The pretrained auxiliary heads are not shipped, so they are dropped.
24model = googlenet(weights=GoogLeNet_Weights.IMAGENET1K_V1).eval()
25print(sum(p.numel() for p in model.parameters())) # 6624904
26
27# Training from scratch with the auxiliary heads: train() mode returns three logits
28net = googlenet(weights=None, aux_logits=True, init_weights=True).train()
29out = net(torch.randn(2, 3, 224, 224)) # GoogLeNetOutputs(logits, aux_logits2, aux_logits1)
30y = torch.tensor([3, 7])
31ce = nn.functional.cross_entropy
32loss = ce(out.logits, y) + 0.3 * (ce(out.aux_logits1, y) + ce(out.aux_logits2, y))
An Inception module (BN variant, true 5×5) and the torchvision GoogLeNet

Use when

  • +About 1.5G multiply-adds and ~6.8M parameters for the best ILSVRC-2014 accuracy: far more efficient than AlexNet or VGG-16 (138M parameters).
  • +Multi-scale features at every location: 1×11\times1, 3×33\times3 and 5×55\times5 views side by side, and the next layer learns how to weight them.
  • +1×11\times1 reductions and global average pooling keep both compute and parameters low.

Avoid when

  • −Many hand-tuned branch widths per module: hard to scale or adapt, unlike VGG or ResNet, which repeat one block.
  • −Parallel branches and concatenations are less efficient on GPUs than one big dense conv of the same FLOPs.
  • −Auxiliary heads and a complex training recipe; superseded by BN and residual networks within two years.

Check yourself

Module 3a's 5×5 branch maps 192 channels to 32. Why does putting a 1×1 conv to 16 channels in front of it cut its cost by about 10×?

Check yourself

Without the 1×1 "pool projection", what goes wrong as Inception modules are stacked?

Remember

  • ✓An Inception module runs 1×11\times1, 3×33\times3, 5×55\times5 convs and a 3×33\times3 max-pool in parallel and concatenates them along channels.
  • ✓1×11\times1 "reduce" convs before the large kernels (from Network-in-Network) cut the cost ratio to r/(k2Cout)+r/Cinr/(k^2C_{out})+r/C_{in}: about 10× for module 3a's 5×55\times5 branch.
  • ✓GoogLeNet: 22 layers, 9 modules, ≈6.8M parameters and ≈1.5G multiply-adds, global average pooling instead of large fc layers.
  • ✓Two auxiliary classifiers (loss weight 0.3) on 4a and 4d during training; later shown to act mostly as regularizers.
  • ✓Won ILSVRC-2014 with 6.67% top-5 error (7-model, 144-crop ensemble); led to BN-Inception, Inception-v3, Xception and Inception-ResNet.
← back to Deep Learning roadmap