# GoogLeNet / Inception

> A 22-layer CNN built from Inception modules: parallel 1×1, 3×3, 5×5 and pooling branches whose outputs are concatenated, with cheap 1×1 convolutions shrinking the depth first. It won ILSVRC-2014 with 6.67% top-5 error using about 6.8M parameters, roughly a ninth of AlexNet's 60M.

Source: https://www.mlfoundry.in/learn/googlenet-inception  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 12 min read · updated 2026-10-11

**GoogLeNet:** is the 22-layer convolutional network of Szegedy et al. (2014), "Going deeper with convolutions". It is a stack of **Inception modules**. Each module runs a $1\times1$, a $3\times3$ and a $5\times5$ convolution and a $3\times3$ max-pool **in parallel** on the same input and concatenates the four outputs along the channel axis. Before the expensive $3\times3$ and $5\times5$ convolutions, a $1\times1$ convolution first reduces the number of channels. "Inception" is the module and the family of networks; GoogLeNet is the specific network the paper entered in ILSVRC-2014, its name a nod to LeNet.

After AlexNet (2012) the obvious way to get better ImageNet accuracy was to make the network bigger: more layers and more channels. That has two costs. More parameters overfit more easily, and compute grows quadratically when two chained conv layers are both widened. If you double the channels of two stacked $3\times3$ layers, the second one costs $4\times$ as much. Szegedy and colleagues set themselves a budget of about **1.5 billion multiply-adds** per image, so that the network could be used in practice, and asked how to spend it well.

Two ideas shaped the answer. Lin, Chen and Yan's **Network-in-Network** (2013) had used $1\times1$ convolutions as small per-pixel MLPs across channels. GoogLeNet uses them mainly for **dimension reduction**: a $1\times1$ conv from 192 to 16 channels before a $5\times5$ conv cuts that conv's cost by about 10×. The second idea came from Arora et al.'s analysis of sparse networks: an ideal network would be sparsely connected, with correlated units grouped together, but sparse matrix arithmetic is slow on hardware. The Inception module approximates such a sparse structure with dense building blocks. Several filter sizes look at the same location in parallel, and the next layer decides which of them to use.

GoogLeNet won the ILSVRC-2014 classification task with **6.67% top-5 error** on the test set, ahead of VGG (7.32%). That number comes from an ensemble of 7 networks with 144 crops per image. A single network with a single crop scored 10.07% top-5 on the validation set. The paper reports that GoogLeNet uses 12 times fewer parameters than the 2012 winner of Krizhevsky et al. (about 60M).

**At a glance**

- **Paper:** Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, Rabinovich, CVPR 2015 (arXiv Sept. 2014)
- **Depth:** 22 layers with parameters (27 counting pooling), 9 Inception modules
- **Parameters:** ≈6.8M (sum of the paper's Table 1); torchvision's port: 6,624,904
- **Compute:** ≈1.5G multiply-adds at $224\times224$
- **Result:** ILSVRC-2014 winner, 6.67% top-5 test error (7-model, 144-crop ensemble)
- **Key tricks:** parallel branches, $1\times1$ bottlenecks, global average pooling, two auxiliary classifiers during training
- **In PyTorch:** `torchvision.models.googlenet(weights="IMAGENET1K_V1")`: 69.78% top-1, 89.53% top-5

## The Inception module

Let $x$ be a feature map with $C_{in}$ channels on an $H\times W$ grid. The module computes four branches that all keep the grid size (stride 1, "same" padding) and concatenates them:

$$
y=\Big[\;C^{1\times1}_{n_1}(x)\;\Big\|\;C^{3\times3}_{n_3}\big(R_{r_3}(x)\big)\;\Big\|\;C^{5\times5}_{n_5}\big(R_{r_5}(x)\big)\;\Big\|\;R_{p}\big(\mathrm{maxpool}_{3\times3}(x)\big)\;\Big],\qquad C_{out}=n_1+n_3+n_5+p
$$

- $C^{k\times k}_{n}$: a $k\times k$ convolution with $n$ output channels, followed by ReLU
- $R_{r}$: a $1\times1$ "reduce" convolution to $r$ channels (+ ReLU): the bottleneck
- $\|$: concatenation along the channel axis (the paper's "filter concatenation")
- $p$: "pool proj": the $1\times1$ conv after the max-pool, which keeps the pool branch from passing all $C_{in}$ channels through

Why the reductions matter becomes clear from the cost of a convolution. A $k\times k$ conv from $C_{in}$ to $C_{out}$ channels on an $H\times W$ output has $k^2C_{in}C_{out}$ weights and needs $k^2C_{in}C_{out}HW$ multiply-adds. Putting a $1\times1$ conv to $r$ channels in front replaces $k^2C_{in}C_{out}$ by $C_{in}r+k^2rC_{out}$:

$$
\frac{\text{cost with reduction}}{\text{cost without}}=\frac{C_{in}\,r+k^2\,r\,C_{out}}{k^2\,C_{in}\,C_{out}}=\frac{r}{k^2C_{out}}+\frac{r}{C_{in}}
$$

- $r$: channels after the $1\times1$ reduction ($r_3$ or $r_5$ in the module)
- $k$: kernel size of the expensive branch (3 or 5)
- $r/C_{in}$: dominant term: the reduction ratio. Module 3a's $5\times5$ branch has $r/C_{in}=16/192$

The pooling branch has a second problem. Max-pooling keeps the channel count, so without a projection every module would output at least as many channels as it receives, plus the conv branches. Depth would then grow from module to module, and the cost of each following module with it. The figure's "naive" mode shows this.

Backpropagation through a module is simple. The concatenation splits the incoming gradient $g=\partial\mathcal L/\partial y$ into four channel slices $g_1,\dots,g_4$, one per branch. Because all four branches read the same $x$, their input gradients add: $\partial\mathcal L/\partial x=\sum_b J_b^\top g_b$, where $J_b$ is the Jacobian of branch $b$. The max-pool branch routes its share only to the input position that was the maximum of each window.

> Figure: GoogLeNet layer by layer, computed from the paper's Table 1 configuration at 224×224. The top strip shows each stage's output depth and multiply-adds; click a stage to see the four branches of that Inception module. With the default 1×1 reductions, module 3a maps 192×28² to 256×28² with 163,328 weights and 128.0M mult-adds (the paper lists 159K = 159.5×1024 and 128M), and the whole network has 6,990,272 weights and 1.58G mult-adds. "Naive: no reductions" removes every 1×1 reduction and pool projection: the depth grows to 4,944 channels and the network needs 97.5M weights and 12.57G mult-adds, about 14× and 7.9× as much. "torchvision variant" adds BatchNorm and the fc bias and reproduces torchvision's 6,624,904 parameters exactly. "aux classifiers" marks 4a and 4d and adds their 6.38M training-only weights.

## The whole network

**GoogLeNet from input to output (224×224×3)**

1. **Stem.** $7\times7$/2 conv to 64 channels, $3\times3$/2 max-pool, then a $1\times1$ conv to 64 and a $3\times3$ conv to 192, another max-pool: $192\times28\times28$. This part is a conventional CNN, because the paper found Inception modules most useful at higher layers.
2. **Inception 3a, 3b** at $28\times28$ (256 and 480 channels out), max-pool to $14\times14$.
3. **Inception 4a–4e** at $14\times14$ (512, 512, 512, 528, 832 channels), max-pool to $7\times7$.
4. **Inception 5a, 5b** at $7\times7$ (832 and 1024 channels).
5. **Head.** $7\times7$ global average pooling to a 1024-vector, 40% dropout, one fully connected layer to 1000 classes, softmax. Replacing AlexNet's large fully connected layers with average pooling improved top-1 accuracy by about 0.6% and removed most of the parameters: the single fc layer has $1024\times1000\approx1$M weights.

Branch widths change with depth: the share of $3\times3$ and $5\times5$ filters grows in later modules, because features become more abstract and spatially spread out. Module 5b has 384 + 384 + 128 + 128 = 1024 output channels, while 3a has 64 + 128 + 32 + 32 = 256.

## Auxiliary classifiers

A 22-layer network was deep for 2014, and the paper worried that gradients would not reach the lower layers well. It attached two small **auxiliary classifiers** to the outputs of modules 4a and 4d. Each one is a $5\times5$/3 average pool (giving $4\times4\times512$ or $4\times4\times528$), a $1\times1$ conv to 128 channels, a 1024-unit fully connected layer, 70% dropout and a 1000-way softmax. Their losses are added with weight 0.3:

$$
\mathcal L=\mathcal L_{\text{main}}+0.3\,\big(\mathcal L_{\text{aux},4a}+\mathcal L_{\text{aux},4d}\big),\qquad \frac{\partial\mathcal L}{\partial h_{4a}}=\frac{\partial\mathcal L_{\text{main}}}{\partial h_{4a}}+0.3\,\frac{\partial\mathcal L_{\text{aux},4a}}{\partial h_{4a}}+0.3\,\frac{\partial\mathcal L_{\text{aux},4d}}{\partial h_{4a}}
$$

- $\mathcal L_{\text{aux}}$: cross-entropy of an auxiliary head, computed on the same labels
- $h_{4a}$: output of module 4a; the auxiliary head on it adds a gradient path only a few layers long
- $0.3$: discount weight from the paper; at inference both heads are discarded

> **Did the auxiliary heads help?:** Later work by the same group (Szegedy et al. 2016, see **Inception-v3**) found that the heads do **not** speed up early training. Networks with and without them trained almost identically until late in training, when the version with heads pulled slightly ahead. They concluded the heads act mainly as a **regularizer**, not as a cure for vanishing gradients. Once **Batch normalization** arrived, deep networks no longer needed such tricks to train.

## Worked example: module 3a with and without reductions

Module 3a receives $192\times28\times28$ and has $n_1=64$, $r_3=96$, $n_3=128$, $r_5=16$, $n_5=32$, $p=32$. Counting conv weights only (as the paper does), with multiply-adds over the $28\times28=784$ output positions:

Module 3a, weights and multiply-adds per branch (computed by the figure's lib and checked by hand)

| Branch | With 1×1 reductions | Weights | Mult-adds | Naive (no reductions) | Weights | Mult-adds |
| --- | --- | --- | --- | --- | --- | --- |
| 1×1 | 192→64 | 12,288 | 9.6M | 192→64 | 12,288 | 9.6M |
| 3×3 | 192→96, 3×3 96→128 | 129,024 | 101.2M | 3×3 192→128 | 221,184 | 173.4M |
| 5×5 | 192→16, 5×5 16→32 | 15,872 | 12.4M | 5×5 192→32 | 153,600 | 120.4M |
| pool | max-pool, 192→32 | 6,144 | 4.8M | max-pool only | 0 | 0 |
| module | 256 channels out | 163,328 | 128.0M | 416 channels out | 387,072 | 303.5M |

**Reading the table**

1. The $5\times5$ branch: $25\cdot192\cdot32=153{,}600$ weights without reduction, versus $192\cdot16+25\cdot16\cdot32=3{,}072+12{,}800=15{,}872$ with it. That is 9.7× fewer, matching the formula: $16/(25\cdot32)+16/192=0.103$.
2. Multiply-adds are weights × 784: $15{,}872\cdot784=12.4$M versus $153{,}600\cdot784=120.4$M.
3. The naive pool branch has no weights, but it passes all 192 input channels through, so the module outputs $64+128+32+192=416$ channels instead of 256. Module 3b then sees 416 input channels instead of 256, and the growth compounds.
4. Over the full network the naive version reaches 4,944 output channels at 5b and costs 97.5M weights and 12.57G mult-adds, against 6.99M and 1.58G with reductions (figure, "naive" mode).

```python
import torch, torch.nn as nn
from torchvision.models import googlenet, GoogLeNet_Weights

def conv_bn(cin, cout, k):
    return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2, bias=False),
                         nn.BatchNorm2d(cout), nn.ReLU(inplace=True))

class Inception(nn.Module):
    def __init__(self, cin, n1, r3, n3, r5, n5, pp):
        super().__init__()
        self.b1 = conv_bn(cin, n1, 1)
        self.b2 = nn.Sequential(conv_bn(cin, r3, 1), conv_bn(r3, n3, 3))   # 1x1 reduce, then 3x3
        self.b3 = nn.Sequential(conv_bn(cin, r5, 1), conv_bn(r5, n5, 5))   # 1x1 reduce, then 5x5
        self.b4 = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1), conv_bn(cin, pp, 1))

    def forward(self, x):                      # every branch keeps H x W
        return torch.cat([self.b1(x), self.b2(x), self.b3(x), self.b4(x)], dim=1)

m3a = Inception(192, 64, 96, 128, 16, 32, 32)
print(m3a(torch.randn(1, 192, 28, 28)).shape)          # torch.Size([1, 256, 28, 28])
print(sum(p.numel() for p in m3a.parameters()))        # 164064 = 163,328 weights + 736 BN

# Pretrained GoogLeNet. The pretrained auxiliary heads are not shipped, so they are dropped.
model = googlenet(weights=GoogLeNet_Weights.IMAGENET1K_V1).eval()
print(sum(p.numel() for p in model.parameters()))      # 6624904

# Training from scratch with the auxiliary heads: train() mode returns three logits
net = googlenet(weights=None, aux_logits=True, init_weights=True).train()
out = net(torch.randn(2, 3, 224, 224))                 # GoogLeNetOutputs(logits, aux_logits2, aux_logits1)
y = torch.tensor([3, 7])
ce = nn.functional.cross_entropy
loss = ce(out.logits, y) + 0.3 * (ce(out.aux_logits1, y) + ce(out.aux_logits2, y))
```

An Inception module (BN variant, true 5×5) and the torchvision GoogLeNet

> **torchvision's GoogLeNet is not exactly the paper's:** torchvision ports the weights of a BatchNorm re-implementation, and in that port the "$5\times5$" branch actually uses a $3\times3$ kernel (a known quirk noted in the source code). Its 6,624,904 parameters include BN scales and shifts and the fc bias. If you compute FLOPs or receptive fields "by the paper", do not expect them to match this model exactly.

> **1×1 convolutions are the real invention:** Parallel filter sizes get the attention, but the $1\times1$ reductions are what made the module affordable, and they outlived it. ResNet's bottleneck block ($1\times1$ reduce, $3\times3$, $1\times1$ expand), SqueezeNet's squeeze layers and the pointwise half of a **Depthwise separable convolution** all use the same move: mix channels cheaply with $1\times1$ and spend the spatial computation on few channels.

> **What came next:** **Batch normalization** (Ioffe & Szegedy 2015) was introduced on an Inception network: BN-Inception, often called Inception-v2, added BN and replaced each $5\times5$ conv with two $3\times3$ convs. **Inception-v3** (2016) went further with factorized convolutions and label smoothing, **Xception** (2017) pushed the idea to depthwise separable convolutions, and Inception-v4 / Inception-ResNet (2016) added residual connections from **ResNet**.

**Strengths**

- About 1.5G multiply-adds and ~6.8M parameters for the best ILSVRC-2014 accuracy: far more efficient than AlexNet or VGG-16 (138M parameters).
- Multi-scale features at every location: $1\times1$, $3\times3$ and $5\times5$ views side by side, and the next layer learns how to weight them.
- $1\times1$ reductions and global average pooling keep both compute and parameters low.

**Limitations**

- Many hand-tuned branch widths per module: hard to scale or adapt, unlike VGG or ResNet, which repeat one block.
- Parallel branches and concatenations are less efficient on GPUs than one big dense conv of the same FLOPs.
- Auxiliary heads and a complex training recipe; superseded by BN and residual networks within two years.

**Question:** Module 3a's 5×5 branch maps 192 channels to 32. Why does putting a 1×1 conv to 16 channels in front of it cut its cost by about 10×?

- A. The 5×5 conv then runs on 16 instead of 192 input channels, and the extra 1×1 conv is cheap
- B. The 1×1 conv halves the spatial resolution
- C. The 1×1 conv makes the 5×5 kernel sparse
- D. ReLU after the 1×1 conv zeroes half the multiplications

**Answer:** A. The 5×5 conv then runs on 16 instead of 192 input channels, and the extra 1×1 conv is cheap — The cost of a conv is $k^2C_{in}C_{out}$ per position. With the reduction it is $192\cdot16+25\cdot16\cdot32=15{,}872$ instead of $25\cdot192\cdot32=153{,}600$, i.e. $r/(k^2C_{out})+r/C_{in}=0.103$ of the original. The grid size does not change.

**Question:** Without the 1×1 "pool projection", what goes wrong as Inception modules are stacked?

- A. The receptive field stops growing
- B. Each module outputs more channels than it receives, so depth and cost keep growing from module to module
- C. Max-pooling makes gradients vanish
- D. The pooling branch would change the spatial size

**Answer:** B. Each module outputs more channels than it receives, so depth and cost keep growing from module to module — Max-pooling keeps all $C_{in}$ channels, and the conv branches are concatenated on top, so $C_{out}>C_{in}$ in every module. In the naive figure mode depth grows from 192 to 4,944 channels and the network needs 7.9× the mult-adds.

**Key points**

- An Inception module runs $1\times1$, $3\times3$, $5\times5$ convs and a $3\times3$ max-pool in parallel and concatenates them along channels.
- $1\times1$ "reduce" convs before the large kernels (from Network-in-Network) cut the cost ratio to $r/(k^2C_{out})+r/C_{in}$: about 10× for module 3a's $5\times5$ branch.
- GoogLeNet: 22 layers, 9 modules, ≈6.8M parameters and ≈1.5G multiply-adds, global average pooling instead of large fc layers.
- Two auxiliary classifiers (loss weight 0.3) on 4a and 4d during training; later shown to act mostly as regularizers.
- Won ILSVRC-2014 with 6.67% top-5 error (7-model, 144-crop ensemble); led to BN-Inception, Inception-v3, Xception and Inception-ResNet.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
