Definition
GoogLeNet is the 22-layer convolutional network of Szegedy et al. (2014), "Going deeper with convolutions". It is a stack of Inception modules. Each module runs a , a and a convolution and a max-pool in parallel on the same input and concatenates the four outputs along the channel axis. Before the expensive and convolutions, a convolution first reduces the number of channels. "Inception" is the module and the family of networks; GoogLeNet is the specific network the paper entered in ILSVRC-2014, its name a nod to LeNet.
After AlexNet (2012) the obvious way to get better ImageNet accuracy was to make the network bigger: more layers and more channels. That has two costs. More parameters overfit more easily, and compute grows quadratically when two chained conv layers are both widened. If you double the channels of two stacked layers, the second one costs as much. Szegedy and colleagues set themselves a budget of about 1.5 billion multiply-adds per image, so that the network could be used in practice, and asked how to spend it well.
Two ideas shaped the answer. Lin, Chen and Yan's Network-in-Network (2013) had used convolutions as small per-pixel MLPs across channels. GoogLeNet uses them mainly for dimension reduction: a conv from 192 to 16 channels before a conv cuts that conv's cost by about 10×. The second idea came from Arora et al.'s analysis of sparse networks: an ideal network would be sparsely connected, with correlated units grouped together, but sparse matrix arithmetic is slow on hardware. The Inception module approximates such a sparse structure with dense building blocks. Several filter sizes look at the same location in parallel, and the next layer decides which of them to use.
GoogLeNet won the ILSVRC-2014 classification task with 6.67% top-5 error on the test set, ahead of VGG (7.32%). That number comes from an ensemble of 7 networks with 144 crops per image. A single network with a single crop scored 10.07% top-5 on the validation set. The paper reports that GoogLeNet uses 12 times fewer parameters than the 2012 winner of Krizhevsky et al. (about 60M).
At a glance
- Paper
- Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, Rabinovich, CVPR 2015 (arXiv Sept. 2014)
- Depth
- 22 layers with parameters (27 counting pooling), 9 Inception modules
- Parameters
- ≈6.8M (sum of the paper's Table 1); torchvision's port: 6,624,904
- Compute
- ≈1.5G multiply-adds at
- Result
- ILSVRC-2014 winner, 6.67% top-5 test error (7-model, 144-crop ensemble)
- Key tricks
- parallel branches, bottlenecks, global average pooling, two auxiliary classifiers during training
- In PyTorch
torchvision.models.googlenet(weights="IMAGENET1K_V1"): 69.78% top-1, 89.53% top-5
The Inception module
Let be a feature map with channels on an grid. The module computes four branches that all keep the grid size (stride 1, "same" padding) and concatenates them:
- a convolution with output channels, followed by ReLU
- a "reduce" convolution to channels (+ ReLU): the bottleneck
- concatenation along the channel axis (the paper's "filter concatenation")
- "pool proj": the conv after the max-pool, which keeps the pool branch from passing all channels through
Why the reductions matter becomes clear from the cost of a convolution. A conv from to channels on an output has weights and needs multiply-adds. Putting a conv to channels in front replaces by :
- channels after the reduction ( or in the module)
- kernel size of the expensive branch (3 or 5)
- dominant term: the reduction ratio. Module 3a's branch has
The pooling branch has a second problem. Max-pooling keeps the channel count, so without a projection every module would output at least as many channels as it receives, plus the conv branches. Depth would then grow from module to module, and the cost of each following module with it. The figure's "naive" mode shows this.
Backpropagation through a module is simple. The concatenation splits the incoming gradient into four channel slices , one per branch. Because all four branches read the same , their input gradients add: , where is the Jacobian of branch . The max-pool branch routes its share only to the input position that was the maximum of each window.
The whole network
GoogLeNet from input to output (224×224×3)
- 1Stem. /2 conv to 64 channels, /2 max-pool, then a conv to 64 and a conv to 192, another max-pool: . This part is a conventional CNN, because the paper found Inception modules most useful at higher layers.
- 2Inception 3a, 3b at (256 and 480 channels out), max-pool to .
- 3Inception 4a–4e at (512, 512, 512, 528, 832 channels), max-pool to .
- 4Inception 5a, 5b at (832 and 1024 channels).
- 5Head. global average pooling to a 1024-vector, 40% dropout, one fully connected layer to 1000 classes, softmax. Replacing AlexNet's large fully connected layers with average pooling improved top-1 accuracy by about 0.6% and removed most of the parameters: the single fc layer has M weights.
Branch widths change with depth: the share of and filters grows in later modules, because features become more abstract and spatially spread out. Module 5b has 384 + 384 + 128 + 128 = 1024 output channels, while 3a has 64 + 128 + 32 + 32 = 256.
Auxiliary classifiers
A 22-layer network was deep for 2014, and the paper worried that gradients would not reach the lower layers well. It attached two small auxiliary classifiers to the outputs of modules 4a and 4d. Each one is a /3 average pool (giving or ), a conv to 128 channels, a 1024-unit fully connected layer, 70% dropout and a 1000-way softmax. Their losses are added with weight 0.3:
- cross-entropy of an auxiliary head, computed on the same labels
- output of module 4a; the auxiliary head on it adds a gradient path only a few layers long
- discount weight from the paper; at inference both heads are discarded
Worked example: module 3a with and without reductions
Module 3a receives and has , , , , , . Counting conv weights only (as the paper does), with multiply-adds over the output positions:
| Branch | With 1×1 reductions | Weights | Mult-adds | Naive (no reductions) | Weights | Mult-adds |
|---|---|---|---|---|---|---|
| 1×1 | 192→64 | 12,288 | 9.6M | 192→64 | 12,288 | 9.6M |
| 3×3 | 192→96, 3×3 96→128 | 129,024 | 101.2M | 3×3 192→128 | 221,184 | 173.4M |
| 5×5 | 192→16, 5×5 16→32 | 15,872 | 12.4M | 5×5 192→32 | 153,600 | 120.4M |
| pool | max-pool, 192→32 | 6,144 | 4.8M | max-pool only | 0 | 0 |
| module | 256 channels out | 163,328 | 128.0M | 416 channels out | 387,072 | 303.5M |
Reading the table
- 1The branch: weights without reduction, versus with it. That is 9.7× fewer, matching the formula: .
- 2Multiply-adds are weights × 784: M versus M.
- 3The naive pool branch has no weights, but it passes all 192 input channels through, so the module outputs channels instead of 256. Module 3b then sees 416 input channels instead of 256, and the growth compounds.
- 4Over the full network the naive version reaches 4,944 output channels at 5b and costs 97.5M weights and 12.57G mult-adds, against 6.99M and 1.58G with reductions (figure, "naive" mode).
1import torch, torch.nn as nn2from torchvision.models import googlenet, GoogLeNet_Weights34def conv_bn(cin, cout, k):5 return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2, bias=False),6 nn.BatchNorm2d(cout), nn.ReLU(inplace=True))78class Inception(nn.Module):9 def __init__(self, cin, n1, r3, n3, r5, n5, pp):10 super().__init__()11 self.b1 = conv_bn(cin, n1, 1)12 self.b2 = nn.Sequential(conv_bn(cin, r3, 1), conv_bn(r3, n3, 3)) # 1x1 reduce, then 3x313 self.b3 = nn.Sequential(conv_bn(cin, r5, 1), conv_bn(r5, n5, 5)) # 1x1 reduce, then 5x514 self.b4 = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1), conv_bn(cin, pp, 1))1516 def forward(self, x): # every branch keeps H x W17 return torch.cat([self.b1(x), self.b2(x), self.b3(x), self.b4(x)], dim=1)1819m3a = Inception(192, 64, 96, 128, 16, 32, 32)20print(m3a(torch.randn(1, 192, 28, 28)).shape) # torch.Size([1, 256, 28, 28])21print(sum(p.numel() for p in m3a.parameters())) # 164064 = 163,328 weights + 736 BN2223# Pretrained GoogLeNet. The pretrained auxiliary heads are not shipped, so they are dropped.24model = googlenet(weights=GoogLeNet_Weights.IMAGENET1K_V1).eval()25print(sum(p.numel() for p in model.parameters())) # 66249042627# Training from scratch with the auxiliary heads: train() mode returns three logits28net = googlenet(weights=None, aux_logits=True, init_weights=True).train()29out = net(torch.randn(2, 3, 224, 224)) # GoogLeNetOutputs(logits, aux_logits2, aux_logits1)30y = torch.tensor([3, 7])31ce = nn.functional.cross_entropy32loss = ce(out.logits, y) + 0.3 * (ce(out.aux_logits1, y) + ce(out.aux_logits2, y))Use when
- +About 1.5G multiply-adds and ~6.8M parameters for the best ILSVRC-2014 accuracy: far more efficient than AlexNet or VGG-16 (138M parameters).
- +Multi-scale features at every location: , and views side by side, and the next layer learns how to weight them.
- + reductions and global average pooling keep both compute and parameters low.
Avoid when
- −Many hand-tuned branch widths per module: hard to scale or adapt, unlike VGG or ResNet, which repeat one block.
- −Parallel branches and concatenations are less efficient on GPUs than one big dense conv of the same FLOPs.
- −Auxiliary heads and a complex training recipe; superseded by BN and residual networks within two years.
Check yourself
Module 3a's 5×5 branch maps 192 channels to 32. Why does putting a 1×1 conv to 16 channels in front of it cut its cost by about 10×?
Check yourself
Without the 1×1 "pool projection", what goes wrong as Inception modules are stacked?
Remember
- ✓An Inception module runs , , convs and a max-pool in parallel and concatenates them along channels.
- ✓ "reduce" convs before the large kernels (from Network-in-Network) cut the cost ratio to : about 10× for module 3a's branch.
- ✓GoogLeNet: 22 layers, 9 modules, ≈6.8M parameters and ≈1.5G multiply-adds, global average pooling instead of large fc layers.
- ✓Two auxiliary classifiers (loss weight 0.3) on 4a and 4d during training; later shown to act mostly as regularizers.
- ✓Won ILSVRC-2014 with 6.67% top-5 error (7-model, 144-crop ensemble); led to BN-Inception, Inception-v3, Xception and Inception-ResNet.