# Inception-v3

> The 2016 redesign of Inception: replace 5×5 convs by two 3×3, and n×n by a 1×n then n×1 conv, shrink the grid with parallel stride-2 branches, and train with label smoothing and a BN auxiliary head. 21.2% top-1 single-crop error on ImageNet for about 5G multiply-adds.

Source: https://www.mlfoundry.in/learn/inception-v3  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 12 min read · updated 2026-10-11

**Inception-v3:** is the ImageNet network of Szegedy, Vanhoucke, Ioffe, Shlens and Wojna, "Rethinking the Inception Architecture for Computer Vision" (CVPR 2016). It keeps the multi-branch Inception module of **GoogLeNet / Inception** but builds every large kernel from smaller ones: **factorized convolutions**. It adds a cheaper way to reduce the grid size, takes $299\times299$ inputs, and is the paper in which **label smoothing** was introduced.

GoogLeNet was efficient but hard to change: its branch widths were tuned by hand, and simply scaling it up made the cost explode. The 2016 paper tries to turn the design into principles. It states four guidelines: avoid **representational bottlenecks** (do not shrink the representation sharply, especially early in the network); higher-dimensional representations are easier to process locally; spatial aggregation can be done on lower-dimensional embeddings without much loss, which is why $1\times1$ reductions work; and balance the width and depth of the network.

The naming is confusing. **Inception-v2** usually means **BN-Inception** from the **Batch normalization** paper (Ioffe & Szegedy 2015): GoogLeNet with BN after every conv and each $5\times5$ conv replaced by two $3\times3$ convs. The 2016 paper uses "Inception-v2" for its own intermediate factorized network and calls the final model, with all changes combined, **Inception-v3**. Later the same group published Inception-v4 and Inception-ResNet-v1/v2 (Szegedy, Ioffe, Vanhoucke & Alemi 2016), which add more modules and residual connections; an ensemble of them reached 3.08% top-5 test error.

**At a glance**

- **Paper:** Szegedy, Vanhoucke, Ioffe, Shlens, Wojna, CVPR 2016 (arXiv Dec. 2015)
- **Input:** $299\times299$; grids $35\times35\to17\times17\to8\times8$ inside the network
- **Factorizations:** $5\times5\to$ two $3\times3$; $n\times n\to1\times n$ then $n\times1$ ($n=7$ at $17\times17$)
- **Cost:** about 5G multiply-adds and under 25M parameters (paper); 23.8M without the auxiliary head
- **Result:** 21.2% top-1, 5.6% top-5 error, single crop (paper); 4-model multi-crop ensemble 17.3% / 3.5%
- **Training:** RMSProp, label smoothing ε = 0.1, BN-auxiliary classifier, gradient clipping at 2.0
- **In PyTorch:** `torchvision.models.inception_v3`: 27,161,264 params with aux head, 77.29% top-1, 5.71 GFLOPS

## Factorizing convolutions

A $5\times5$ conv sees the same $5\times5$ window as two stacked $3\times3$ convs, but costs $25/18\approx1.39\times$ as much. Replacing it saves 28% of the weights and multiply-adds, and adds one more nonlinearity. Pushing the idea further, any $n\times n$ conv can be replaced by a $1\times n$ conv followed by an $n\times1$ conv. For a $C\to C$ layer:

$$
\underbrace{n^2C^2}_{n\times n}\;\longrightarrow\;\underbrace{\tfrac{n-1}{2}\cdot 9C^2}_{\text{stacked }3\times3}\quad\text{or}\quad\underbrace{2nC^2}_{1\times n\,+\,n\times1},\qquad \text{saving}_{\text{asym}}=1-\frac{2}{n}
$$

- $n^2C^2$: weights (and mult-adds per output position) of one $n\times n$ conv with $C$ input and output channels
- $\tfrac{n-1}{2}$: number of $3\times3$ layers needed to cover an $n\times n$ window (odd $n$)
- $1-2/n$: share saved by the asymmetric pair: 33% for $n=3$, 60% for $n=5$, 71% for $n=7$

A factorized pair can represent exactly the **rank-1** (separable) $n\times n$ kernels $k=ab^\top$, plus whatever the ReLU and BN between the two convs add. Real kernels are not rank 1, but since the following layers can combine many such pairs, the network loses little. The paper reports that the asymmetric factorization works poorly in early layers and well on medium grids (between 12 and 20 pixels), so Inception-v3 uses $1\times7$ / $7\times1$ pairs only in its $17\times17$ modules. On the coarsest $8\times8$ grid it uses an **expanded filter bank**: a $1\times3$ and a $3\times1$ conv in parallel whose outputs are concatenated, which widens the representation instead of deepening it.

For training, a factorized pair is just two ordinary convolutions, so backprop goes through them one after the other: the $n\times1$ conv's input gradient is the $1\times n$ conv's output gradient. Nothing special is needed.

## Grid-size reduction without a bottleneck

To go from a $d\times d$ grid with $k$ channels to $d/2\times d/2$ with $2k$, there are two classical choices. Pool first and then expand with a conv: cheap, but the pooled $k$-channel map is a representational bottleneck. Expand first and then pool: no bottleneck, but the conv runs on the full $d\times d$ grid, $2d^2k^2$ operations instead of $2(d/2)^2k^2$, so 4× more. Inception-v3 does both halves at once. A stride-2 conv branch and a stride-2 pooling branch run **in parallel** and their outputs are concatenated. Mixed_6a, for example, takes $288\times35\times35$ to $17\times17$ and outputs $384+96+288=768$ channels.

> Figure: Inception-v3 module by module, computed from the torchvision layout at 299×299 (every conv has BN, so params = weights + 2 per channel). Solid bars are the current mode and outlines the other one. Factorized (default): 23,834,568 parameters and 5.71G mult-adds without the auxiliary head, 27,161,264 with it, which matches torchvision's inception_v3 exactly. Click a module: Mixed_6b's 7×7 branches are 1×7 → 7×1 pairs. "Undo factorizations" turns every pair back into one n×n kernel and every two 3×3 back into a 5×5: 44,759,304 parameters (1.88×) and 10.21G mult-adds (1.79×). The lower panel compares one n×n conv with C → C channels to its factorizations: at n = 7 and C = 192 the asymmetric pair has 516K weights instead of 1.81M (71% saved) and three 3×3 convs have 995K (45% saved).

## Training changes

**Label smoothing.** A one-hot target pushes the correct logit to be infinitely larger than all others, which encourages over-confident predictions and overfitting. The paper replaces the target $q(k)=\delta_{k,y}$ with a mixture of the one-hot vector and the uniform distribution $u(k)=1/K$:

$$
q'(k)=(1-\varepsilon)\,\delta_{k,y}+\frac{\varepsilon}{K},\qquad \mathcal L=-\sum_k q'(k)\log p(k)=(1-\varepsilon)\,H(\delta_y,p)+\varepsilon\,H(u,p),\qquad \frac{\partial\mathcal L}{\partial z_k}=p(k)-q'(k)
$$

- $\varepsilon$: smoothing weight; 0.1 in the paper
- $K$: number of classes (1000 for ImageNet)
- $z_k$: logit of class $k$; the gradient vanishes once $p=q'$, so the logits stay finite

With $\varepsilon=0.1$ and $K=1000$ the target puts $0.9001$ on the true class and $0.0001$ on each other class. The paper reports a consistent improvement of about 0.2% absolute in both top-1 and top-5 error. The **Label smoothing** article covers the technique in depth.

**BN-auxiliary.** Inception-v3 keeps a single auxiliary classifier, on the last $17\times17$ layer (Mixed_6e), and adds BatchNorm to it. The paper argues the head acts as a regularizer rather than speeding up early training, and reports that the BN version of the head improves top-1 by 0.4%. **Optimizer.** The best models used RMSProp with decay 0.9 and $\epsilon=1.0$, learning rate 0.045 decayed by 0.94 every two epochs, and gradient clipping at 2.0, trained with 50 replicas.

The paper's ablation adds these changes one at a time to its factorized "Inception-v2" network: RMSProp, then label smoothing, then the factorized $7\times7$, then BN-auxiliary. Each step lowers the single-crop error, ending at 21.2% top-1 and 5.6% top-5, the network reported as Inception-v3.

## Worked example: one 17×17 module

Mixed_6b takes $768\times17\times17$ and has a "7×7" branch: a $1\times1$ conv 768 → 128, then a $1\times7$ conv 128 → 128, then a $7\times1$ conv 128 → 192. All convs are followed by BN (2 parameters per channel). Multiply-adds are weights × $17^2=289$:

Mixed_6b's 7×7 branch, factorized versus a single 7×7 (values from the figure's lib)

| Layer | Factorized: weights | Unfactorized layer | Unfactorized: weights |
| --- | --- | --- | --- |
| 1×1, 768 → 128 | 98,304 | 1×1, 768 → 128 | 98,304 |
| 1×7, 128 → 128 | 114,688 | 7×7, 128 → 192 | 1,204,224 |
| 7×1, 128 → 192 | 172,032 |  |  |
| BN (γ, β) | 896 | BN (γ, β) | 640 |
| branch total | 385,920 (111.3M mult-adds) | branch total | 1,303,168 (376.4M mult-adds) |

**Checking the numbers**

1. $1\times7$: $7\cdot128\cdot128=114{,}688$; $7\times1$: $7\cdot128\cdot192=172{,}032$. Together $286{,}720$, versus $49\cdot128\cdot192=1{,}204{,}224$ for one $7\times7$: 4.2× fewer.
2. Branch with the $1\times1$ and BN: $98{,}304+286{,}720+2(128+128+192)=385{,}920$. Mult-adds: $(98{,}304+286{,}720)\cdot289=111.3$M.
3. Over the whole module (four branches, one of them a doubled $7\times7$ tower) the factorized version has 1,297,408 parameters and 374.1M mult-adds (figure).
4. Label smoothing on the same network: $\varepsilon=0.1$, $K=1000$ gives targets $1-0.1+0.1/1000=0.9001$ and $0.1/1000=0.0001$.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torchvision.models import inception_v3, Inception_V3_Weights

def conv_bn(cin, cout, k, pad):                     # torchvision's BasicConv2d
    return nn.Sequential(nn.Conv2d(cin, cout, k, padding=pad, bias=False),
                         nn.BatchNorm2d(cout, eps=0.001), nn.ReLU(inplace=True))

branch7 = nn.Sequential(                            # Mixed_6b, "7x7" branch
    conv_bn(768, 128, 1, 0),
    conv_bn(128, 128, (1, 7), (0, 3)),              # 1 x 7
    conv_bn(128, 192, (7, 1), (3, 0)))              # 7 x 1
print(sum(p.numel() for p in branch7.parameters())) # 385920
print(branch7(torch.randn(1, 768, 17, 17)).shape)   # torch.Size([1, 192, 17, 17])

# Label smoothing is built into cross_entropy: q' = (1 - eps) * onehot + eps / K
logits, y = torch.randn(8, 1000), torch.randint(0, 1000, (8,))
loss = F.cross_entropy(logits, y, label_smoothing=0.1)

# Pretrained model: 299 x 299 input; train() mode returns (logits, aux_logits)
weights = Inception_V3_Weights.IMAGENET1K_V1
model = inception_v3(weights=weights)
print(sum(p.numel() for p in model.parameters()))   # 27161264 (aux head included)
model.train()
out = model(torch.randn(2, 3, 299, 299))           # InceptionOutputs(logits, aux_logits)
loss = F.cross_entropy(out.logits, y[:2], label_smoothing=0.1) \
     + 0.4 * F.cross_entropy(out.aux_logits, y[:2], label_smoothing=0.1)
model.eval()                                        # eval() returns plain logits
preprocess = weights.transforms()                   # resize 342, center-crop 299
```

A factorized 7×7 branch, label smoothing, and torchvision's Inception-v3

> **299, not 224:** Inception-v3 expects $299\times299$ inputs, and torchvision's pretrained weights also set `transform_input=True`, which re-normalizes standard ImageNet-normalized tensors to the original Inception preprocessing. Feeding $224\times224$ crops still runs in eval mode (global average pooling hides the size change; the auxiliary head would fail in train mode), but the grids become $5\times5$ at the end instead of $8\times8$ and accuracy drops. Always use `weights.transforms()`. The 0.4 auxiliary loss weight in the code is a common choice, not a value from the paper.

> **Factorization is a low-rank prior:** A $1\times n$ followed by an $n\times1$ conv is a rank-1 constraint on each spatial filter. The bet is that natural-image features at medium resolution are well approximated by sums of separable filters, so most of the parameters of an $n\times n$ kernel are redundant. **Xception** and the **Depthwise separable convolution** make a related bet about the channel dimension: that spatial and cross-channel correlations can be learned separately.

> **Where Inception-v3 is still used:** Inception-v3 trained on ImageNet is the fixed feature extractor behind the **FID** (Fréchet Inception Distance) and Inception Score metrics for image generators. Those metrics depend on that specific network: computing them with a different backbone, or with different resizing, gives numbers that are not comparable with published ones.

**Strengths**

- Factorized kernels give large receptive fields at a fraction of the cost: $1\times7+7\times1$ uses 29% of a $7\times7$'s weights.
- Parallel stride-2 conv and pool branches shrink the grid without a representational bottleneck.
- Strong accuracy for its budget in 2016 (21.2% top-1 single crop at about 5G multiply-adds), and label smoothing became a standard training trick.

**Limitations**

- Many module types (A–E) with hand-chosen widths: harder to scale and modify than ResNet's single repeated block.
- Asymmetric convs and many narrow branches run less efficiently on GPUs than their FLOP count suggests.
- Non-standard 299×299 input and preprocessing; superseded by ResNet variants and EfficientNet at equal compute.

**Question:** A 7×7 conv with C input and output channels is replaced by a 1×7 conv followed by a 7×1 conv (both C → C). What fraction of the weights is saved?

- A. 2/7 ≈ 29%
- B. 5/7 ≈ 71%
- C. 1/2
- D. 6/7 ≈ 86%

**Answer:** B. 5/7 ≈ 71% — The pair has $7C^2+7C^2=14C^2$ weights versus $49C^2$: it keeps $14/49=2/7$ and saves $1-2/n=5/7\approx71\%$. In the figure at $C=192$: 516,096 versus 1,806,336.

**Question:** With label smoothing ε = 0.1 on K = 1000 classes, what happens to the cross-entropy gradient with respect to the true-class logit once the model predicts p(y) = 0.9001?

- A. It is still negative, pushing p(y) toward 1
- B. It is zero: the target is reached and the logits stop growing
- C. It becomes positive and large
- D. Label smoothing does not change the gradient

**Answer:** B. It is zero: the target is reached and the logits stop growing — $\partial\mathcal L/\partial z_k=p(k)-q'(k)$, and $q'(y)=0.9001$. With one-hot targets $q(y)=1$, so the gradient stays negative until $p(y)=1$, which requires infinitely large logit gaps. Smoothing gives a finite optimum.

**Key points**

- Inception-v3 (Szegedy et al. 2016) rebuilds Inception from small kernels: $5\times5\to$ two $3\times3$ (−28%), $n\times n\to1\times n+n\times1$ (saves $1-2/n$).
- Asymmetric factorization is used on the $17\times17$ grid ($n=7$); the $8\times8$ grid uses parallel $1\times3$ and $3\times1$ convs.
- Grid reduction runs a stride-2 conv branch and a stride-2 pool branch in parallel and concatenates them, avoiding a representational bottleneck.
- Label smoothing was introduced here: $q'=(1-\varepsilon)\,\text{onehot}+\varepsilon/K$ with $\varepsilon=0.1$, worth about 0.2% absolute.
- 299×299 input, one BN auxiliary head, RMSProp; 21.2% / 5.6% single-crop error. torchvision's version: 27.2M params with aux head, 5.71 GFLOPS.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
