Inception-v3

The 2016 redesign of Inception: replace 5×5 convs by two 3×3, and n×n by a 1×n then n×1 conv, shrink the grid with parallel stride-2 branches, and train with label smoothing and a BN auxiliary head. 21.2% top-1 single-crop error on ImageNet for about 5G multiply-adds.

12 min read1117 wordsadvanced0 viewsupdated 2026-10-11

Definition

Inception-v3 is the ImageNet network of Szegedy, Vanhoucke, Ioffe, Shlens and Wojna, "Rethinking the Inception Architecture for Computer Vision" (CVPR 2016). It keeps the multi-branch Inception module of GoogLeNet / Inception but builds every large kernel from smaller ones: factorized convolutions. It adds a cheaper way to reduce the grid size, takes 299×299299\times299 inputs, and is the paper in which label smoothing was introduced.

GoogLeNet was efficient but hard to change: its branch widths were tuned by hand, and simply scaling it up made the cost explode. The 2016 paper tries to turn the design into principles. It states four guidelines: avoid representational bottlenecks (do not shrink the representation sharply, especially early in the network); higher-dimensional representations are easier to process locally; spatial aggregation can be done on lower-dimensional embeddings without much loss, which is why 1×11\times1 reductions work; and balance the width and depth of the network.

The naming is confusing. Inception-v2 usually means BN-Inception from the Batch normalization paper (Ioffe & Szegedy 2015): GoogLeNet with BN after every conv and each 5×55\times5 conv replaced by two 3×33\times3 convs. The 2016 paper uses "Inception-v2" for its own intermediate factorized network and calls the final model, with all changes combined, Inception-v3. Later the same group published Inception-v4 and Inception-ResNet-v1/v2 (Szegedy, Ioffe, Vanhoucke & Alemi 2016), which add more modules and residual connections; an ensemble of them reached 3.08% top-5 test error.

At a glance

Paper
Szegedy, Vanhoucke, Ioffe, Shlens, Wojna, CVPR 2016 (arXiv Dec. 2015)
Input
299×299299\times299; grids 35×35→17×17→8×835\times35\to17\times17\to8\times8 inside the network
Factorizations
5×5→5\times5\to two 3×33\times3; n×n→1×nn\times n\to1\times n then n×1n\times1 (n=7n=7 at 17×1717\times17)
Cost
about 5G multiply-adds and under 25M parameters (paper); 23.8M without the auxiliary head
Result
21.2% top-1, 5.6% top-5 error, single crop (paper); 4-model multi-crop ensemble 17.3% / 3.5%
Training
RMSProp, label smoothing ε = 0.1, BN-auxiliary classifier, gradient clipping at 2.0
In PyTorch
torchvision.models.inception_v3: 27,161,264 params with aux head, 77.29% top-1, 5.71 GFLOPS

Factorizing convolutions

A 5×55\times5 conv sees the same 5×55\times5 window as two stacked 3×33\times3 convs, but costs 25/18≈1.39×25/18\approx1.39\times as much. Replacing it saves 28% of the weights and multiply-adds, and adds one more nonlinearity. Pushing the idea further, any n×nn\times n conv can be replaced by a 1×n1\times n conv followed by an n×1n\times1 conv. For a C→CC\to C layer:

n2C2⏟n×n  ⟶  n−12⋅9C2⏟stacked 3×3or2nC2⏟1×n + n×1,savingasym=1−2n\underbrace{n^2C^2}_{n\times n}\;\longrightarrow\;\underbrace{\tfrac{n-1}{2}\cdot 9C^2}_{\text{stacked }3\times3}\quad\text{or}\quad\underbrace{2nC^2}_{1\times n\,+\,n\times1},\qquad \text{saving}_{\text{asym}}=1-\frac{2}{n}
n2C2n^2C^2
weights (and mult-adds per output position) of one n×nn\times n conv with CC input and output channels
n−12\tfrac{n-1}{2}
number of 3×33\times3 layers needed to cover an n×nn\times n window (odd nn)
1−2/n1-2/n
share saved by the asymmetric pair: 33% for n=3n=3, 60% for n=5n=5, 71% for n=7n=7

A factorized pair can represent exactly the rank-1 (separable) n×nn\times n kernels k=ab⊤k=ab^\top, plus whatever the ReLU and BN between the two convs add. Real kernels are not rank 1, but since the following layers can combine many such pairs, the network loses little. The paper reports that the asymmetric factorization works poorly in early layers and well on medium grids (between 12 and 20 pixels), so Inception-v3 uses 1×71\times7 / 7×17\times1 pairs only in its 17×1717\times17 modules. On the coarsest 8×88\times8 grid it uses an expanded filter bank: a 1×31\times3 and a 3×13\times1 conv in parallel whose outputs are concatenated, which widens the representation instead of deepening it.

For training, a factorized pair is just two ordinary convolutions, so backprop goes through them one after the other: the n×1n\times1 conv's input gradient is the 1×n1\times n conv's output gradient. Nothing special is needed.

Grid-size reduction without a bottleneck

To go from a d×dd\times d grid with kk channels to d/2×d/2d/2\times d/2 with 2k2k, there are two classical choices. Pool first and then expand with a conv: cheap, but the pooled kk-channel map is a representational bottleneck. Expand first and then pool: no bottleneck, but the conv runs on the full d×dd\times d grid, 2d2k22d^2k^2 operations instead of 2(d/2)2k22(d/2)^2k^2, so 4× more. Inception-v3 does both halves at once. A stride-2 conv branch and a stride-2 pooling branch run in parallel and their outputs are concatenated. Mixed_6a, for example, takes 288×35×35288\times35\times35 to 17×1717\times17 and outputs 384+96+288=768384+96+288=768 channels.

Inception-v3 (torchvision layout), 299×299 · solid: factorized, outline: unfactorizedstem5b5c5d6a6b6c6d6eaux7a7b7cfc■ params ■ mult-adds per module · grid 35×35 → 17×17 → 8×8Mixed_6b (C): 768×17² → 768×17² · 1,297,408 params · 374.1M17×17 module: 7×7 → 1×7 + 7×1, c₇ = 128branchconvs (→ out channels)paramsmult-adds1×11×1→192147,84042.6M1×7, 7×11×1→128, 1×7→128, 7×1→192385,920111.3M(7×1, 1×7) × 21×1→128, 7×1→128, 1×7→128, 7×1→128, 1×7→192615,808177.6Mavg-pool, 1×11×1→192147,84042.6Mone 7×7 conv, 192 → 192 channels, versus its factorizations (weights per output pixel)7×7: 49·C² = 1.81Mbaseline1×7 + 7×1: 14·C² = 516Ksaves 71% (= 1 − 2/n)3× 3×3: 27·C² = 995Ksaves 45%, +2 nonlinearities23,834,568 params, 5.71 G mult-adds without aux; with aux 27,161,264torchvision inception_v3: 27,161,264 params, 5.71 GFLOPS ✓ match
click a module above
fig.Inception-v3 module by module, computed from the torchvision layout at 299×299 (every conv has BN, so params = weights + 2 per channel). Solid bars are the current mode and outlines the other one. Factorized (default): 23,834,568 parameters and 5.71G mult-adds without the auxiliary head, 27,161,264 with it, which matches torchvision's inception_v3 exactly. Click a module: Mixed_6b's 7×7 branches are 1×7 → 7×1 pairs. "Undo factorizations" turns every pair back into one n×n kernel and every two 3×3 back into a 5×5: 44,759,304 parameters (1.88×) and 10.21G mult-adds (1.79×). The lower panel compares one n×n conv with C → C channels to its factorizations: at n = 7 and C = 192 the asymmetric pair has 516K weights instead of 1.81M (71% saved) and three 3×3 convs have 995K (45% saved).

Training changes

Label smoothing. A one-hot target pushes the correct logit to be infinitely larger than all others, which encourages over-confident predictions and overfitting. The paper replaces the target q(k)=δk,yq(k)=\delta_{k,y} with a mixture of the one-hot vector and the uniform distribution u(k)=1/Ku(k)=1/K:

q′(k)=(1−ε) δk,y+εK,L=−∑kq′(k)log⁡p(k)=(1−ε) H(δy,p)+ε H(u,p),∂L∂zk=p(k)−q′(k)q'(k)=(1-\varepsilon)\,\delta_{k,y}+\frac{\varepsilon}{K},\qquad \mathcal L=-\sum_k q'(k)\log p(k)=(1-\varepsilon)\,H(\delta_y,p)+\varepsilon\,H(u,p),\qquad \frac{\partial\mathcal L}{\partial z_k}=p(k)-q'(k)
ε\varepsilon
smoothing weight; 0.1 in the paper
KK
number of classes (1000 for ImageNet)
zkz_k
logit of class kk; the gradient vanishes once p=q′p=q', so the logits stay finite

With ε=0.1\varepsilon=0.1 and K=1000K=1000 the target puts 0.90010.9001 on the true class and 0.00010.0001 on each other class. The paper reports a consistent improvement of about 0.2% absolute in both top-1 and top-5 error. The Label smoothing article covers the technique in depth.

BN-auxiliary. Inception-v3 keeps a single auxiliary classifier, on the last 17×1717\times17 layer (Mixed_6e), and adds BatchNorm to it. The paper argues the head acts as a regularizer rather than speeding up early training, and reports that the BN version of the head improves top-1 by 0.4%. Optimizer. The best models used RMSProp with decay 0.9 and ϵ=1.0\epsilon=1.0, learning rate 0.045 decayed by 0.94 every two epochs, and gradient clipping at 2.0, trained with 50 replicas.

The paper's ablation adds these changes one at a time to its factorized "Inception-v2" network: RMSProp, then label smoothing, then the factorized 7×77\times7, then BN-auxiliary. Each step lowers the single-crop error, ending at 21.2% top-1 and 5.6% top-5, the network reported as Inception-v3.

Worked example: one 17×17 module

Mixed_6b takes 768×17×17768\times17\times17 and has a "7×7" branch: a 1×11\times1 conv 768 → 128, then a 1×71\times7 conv 128 → 128, then a 7×17\times1 conv 128 → 192. All convs are followed by BN (2 parameters per channel). Multiply-adds are weights × 172=28917^2=289:

LayerFactorized: weightsUnfactorized layerUnfactorized: weights
1×1, 768 → 12898,3041×1, 768 → 12898,304
1×7, 128 → 128114,6887×7, 128 → 1921,204,224
7×1, 128 → 192172,032
BN (γ, β)896BN (γ, β)640
branch total385,920 (111.3M mult-adds)branch total1,303,168 (376.4M mult-adds)
Mixed_6b's 7×7 branch, factorized versus a single 7×7 (values from the figure's lib)

Checking the numbers

  1. 11×71\times7: 7⋅128⋅128=114,6887\cdot128\cdot128=114{,}688; 7×17\times1: 7⋅128⋅192=172,0327\cdot128\cdot192=172{,}032. Together 286,720286{,}720, versus 49⋅128⋅192=1,204,22449\cdot128\cdot192=1{,}204{,}224 for one 7×77\times7: 4.2× fewer.
  2. 2Branch with the 1×11\times1 and BN: 98,304+286,720+2(128+128+192)=385,92098{,}304+286{,}720+2(128+128+192)=385{,}920. Mult-adds: (98,304+286,720)⋅289=111.3(98{,}304+286{,}720)\cdot289=111.3M.
  3. 3Over the whole module (four branches, one of them a doubled 7×77\times7 tower) the factorized version has 1,297,408 parameters and 374.1M mult-adds (figure).
  4. 4Label smoothing on the same network: ε=0.1\varepsilon=0.1, K=1000K=1000 gives targets 1−0.1+0.1/1000=0.90011-0.1+0.1/1000=0.9001 and 0.1/1000=0.00010.1/1000=0.0001.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torchvision.models import inception_v3, Inception_V3_Weights
3
4def conv_bn(cin, cout, k, pad): # torchvision's BasicConv2d
5 return nn.Sequential(nn.Conv2d(cin, cout, k, padding=pad, bias=False),
6 nn.BatchNorm2d(cout, eps=0.001), nn.ReLU(inplace=True))
7
8branch7 = nn.Sequential( # Mixed_6b, "7x7" branch
9 conv_bn(768, 128, 1, 0),
10 conv_bn(128, 128, (1, 7), (0, 3)), # 1 x 7
11 conv_bn(128, 192, (7, 1), (3, 0))) # 7 x 1
12print(sum(p.numel() for p in branch7.parameters())) # 385920
13print(branch7(torch.randn(1, 768, 17, 17)).shape) # torch.Size([1, 192, 17, 17])
14
15# Label smoothing is built into cross_entropy: q' = (1 - eps) * onehot + eps / K
16logits, y = torch.randn(8, 1000), torch.randint(0, 1000, (8,))
17loss = F.cross_entropy(logits, y, label_smoothing=0.1)
18
19# Pretrained model: 299 x 299 input; train() mode returns (logits, aux_logits)
20weights = Inception_V3_Weights.IMAGENET1K_V1
21model = inception_v3(weights=weights)
22print(sum(p.numel() for p in model.parameters())) # 27161264 (aux head included)
23model.train()
24out = model(torch.randn(2, 3, 299, 299)) # InceptionOutputs(logits, aux_logits)
25loss = F.cross_entropy(out.logits, y[:2], label_smoothing=0.1) \
26 + 0.4 * F.cross_entropy(out.aux_logits, y[:2], label_smoothing=0.1)
27model.eval() # eval() returns plain logits
28preprocess = weights.transforms() # resize 342, center-crop 299
A factorized 7×7 branch, label smoothing, and torchvision's Inception-v3

Use when

  • +Factorized kernels give large receptive fields at a fraction of the cost: 1×7+7×11\times7+7\times1 uses 29% of a 7×77\times7's weights.
  • +Parallel stride-2 conv and pool branches shrink the grid without a representational bottleneck.
  • +Strong accuracy for its budget in 2016 (21.2% top-1 single crop at about 5G multiply-adds), and label smoothing became a standard training trick.

Avoid when

  • −Many module types (A–E) with hand-chosen widths: harder to scale and modify than ResNet's single repeated block.
  • −Asymmetric convs and many narrow branches run less efficiently on GPUs than their FLOP count suggests.
  • −Non-standard 299×299 input and preprocessing; superseded by ResNet variants and EfficientNet at equal compute.

Check yourself

A 7×7 conv with C input and output channels is replaced by a 1×7 conv followed by a 7×1 conv (both C → C). What fraction of the weights is saved?

Check yourself

With label smoothing ε = 0.1 on K = 1000 classes, what happens to the cross-entropy gradient with respect to the true-class logit once the model predicts p(y) = 0.9001?

Remember

  • ✓Inception-v3 (Szegedy et al. 2016) rebuilds Inception from small kernels: 5×5→5\times5\to two 3×33\times3 (−28%), n×n→1×n+n×1n\times n\to1\times n+n\times1 (saves 1−2/n1-2/n).
  • ✓Asymmetric factorization is used on the 17×1717\times17 grid (n=7n=7); the 8×88\times8 grid uses parallel 1×31\times3 and 3×13\times1 convs.
  • ✓Grid reduction runs a stride-2 conv branch and a stride-2 pool branch in parallel and concatenates them, avoiding a representational bottleneck.
  • ✓Label smoothing was introduced here: q′=(1−ε) onehot+ε/Kq'=(1-\varepsilon)\,\text{onehot}+\varepsilon/K with ε=0.1\varepsilon=0.1, worth about 0.2% absolute.
  • ✓299×299 input, one BN auxiliary head, RMSProp; 21.2% / 5.6% single-crop error. torchvision's version: 27.2M params with aux head, 5.71 GFLOPS.
← back to Deep Learning roadmap