Definition
Inception-v3 is the ImageNet network of Szegedy, Vanhoucke, Ioffe, Shlens and Wojna, "Rethinking the Inception Architecture for Computer Vision" (CVPR 2016). It keeps the multi-branch Inception module of GoogLeNet / Inception but builds every large kernel from smaller ones: factorized convolutions. It adds a cheaper way to reduce the grid size, takes inputs, and is the paper in which label smoothing was introduced.
GoogLeNet was efficient but hard to change: its branch widths were tuned by hand, and simply scaling it up made the cost explode. The 2016 paper tries to turn the design into principles. It states four guidelines: avoid representational bottlenecks (do not shrink the representation sharply, especially early in the network); higher-dimensional representations are easier to process locally; spatial aggregation can be done on lower-dimensional embeddings without much loss, which is why reductions work; and balance the width and depth of the network.
The naming is confusing. Inception-v2 usually means BN-Inception from the Batch normalization paper (Ioffe & Szegedy 2015): GoogLeNet with BN after every conv and each conv replaced by two convs. The 2016 paper uses "Inception-v2" for its own intermediate factorized network and calls the final model, with all changes combined, Inception-v3. Later the same group published Inception-v4 and Inception-ResNet-v1/v2 (Szegedy, Ioffe, Vanhoucke & Alemi 2016), which add more modules and residual connections; an ensemble of them reached 3.08% top-5 test error.
At a glance
- Paper
- Szegedy, Vanhoucke, Ioffe, Shlens, Wojna, CVPR 2016 (arXiv Dec. 2015)
- Input
- ; grids inside the network
- Factorizations
- two ; then ( at )
- Cost
- about 5G multiply-adds and under 25M parameters (paper); 23.8M without the auxiliary head
- Result
- 21.2% top-1, 5.6% top-5 error, single crop (paper); 4-model multi-crop ensemble 17.3% / 3.5%
- Training
- RMSProp, label smoothing ε = 0.1, BN-auxiliary classifier, gradient clipping at 2.0
- In PyTorch
torchvision.models.inception_v3: 27,161,264 params with aux head, 77.29% top-1, 5.71 GFLOPS
Factorizing convolutions
A conv sees the same window as two stacked convs, but costs as much. Replacing it saves 28% of the weights and multiply-adds, and adds one more nonlinearity. Pushing the idea further, any conv can be replaced by a conv followed by an conv. For a layer:
- weights (and mult-adds per output position) of one conv with input and output channels
- number of layers needed to cover an window (odd )
- share saved by the asymmetric pair: 33% for , 60% for , 71% for
A factorized pair can represent exactly the rank-1 (separable) kernels , plus whatever the ReLU and BN between the two convs add. Real kernels are not rank 1, but since the following layers can combine many such pairs, the network loses little. The paper reports that the asymmetric factorization works poorly in early layers and well on medium grids (between 12 and 20 pixels), so Inception-v3 uses / pairs only in its modules. On the coarsest grid it uses an expanded filter bank: a and a conv in parallel whose outputs are concatenated, which widens the representation instead of deepening it.
For training, a factorized pair is just two ordinary convolutions, so backprop goes through them one after the other: the conv's input gradient is the conv's output gradient. Nothing special is needed.
Grid-size reduction without a bottleneck
To go from a grid with channels to with , there are two classical choices. Pool first and then expand with a conv: cheap, but the pooled -channel map is a representational bottleneck. Expand first and then pool: no bottleneck, but the conv runs on the full grid, operations instead of , so 4× more. Inception-v3 does both halves at once. A stride-2 conv branch and a stride-2 pooling branch run in parallel and their outputs are concatenated. Mixed_6a, for example, takes to and outputs channels.
Training changes
Label smoothing. A one-hot target pushes the correct logit to be infinitely larger than all others, which encourages over-confident predictions and overfitting. The paper replaces the target with a mixture of the one-hot vector and the uniform distribution :
- smoothing weight; 0.1 in the paper
- number of classes (1000 for ImageNet)
- logit of class ; the gradient vanishes once , so the logits stay finite
With and the target puts on the true class and on each other class. The paper reports a consistent improvement of about 0.2% absolute in both top-1 and top-5 error. The Label smoothing article covers the technique in depth.
BN-auxiliary. Inception-v3 keeps a single auxiliary classifier, on the last layer (Mixed_6e), and adds BatchNorm to it. The paper argues the head acts as a regularizer rather than speeding up early training, and reports that the BN version of the head improves top-1 by 0.4%. Optimizer. The best models used RMSProp with decay 0.9 and , learning rate 0.045 decayed by 0.94 every two epochs, and gradient clipping at 2.0, trained with 50 replicas.
The paper's ablation adds these changes one at a time to its factorized "Inception-v2" network: RMSProp, then label smoothing, then the factorized , then BN-auxiliary. Each step lowers the single-crop error, ending at 21.2% top-1 and 5.6% top-5, the network reported as Inception-v3.
Worked example: one 17×17 module
Mixed_6b takes and has a "7×7" branch: a conv 768 → 128, then a conv 128 → 128, then a conv 128 → 192. All convs are followed by BN (2 parameters per channel). Multiply-adds are weights × :
| Layer | Factorized: weights | Unfactorized layer | Unfactorized: weights |
|---|---|---|---|
| 1×1, 768 → 128 | 98,304 | 1×1, 768 → 128 | 98,304 |
| 1×7, 128 → 128 | 114,688 | 7×7, 128 → 192 | 1,204,224 |
| 7×1, 128 → 192 | 172,032 | ||
| BN (γ, β) | 896 | BN (γ, β) | 640 |
| branch total | 385,920 (111.3M mult-adds) | branch total | 1,303,168 (376.4M mult-adds) |
Checking the numbers
- 1: ; : . Together , versus for one : 4.2× fewer.
- 2Branch with the and BN: . Mult-adds: M.
- 3Over the whole module (four branches, one of them a doubled tower) the factorized version has 1,297,408 parameters and 374.1M mult-adds (figure).
- 4Label smoothing on the same network: , gives targets and .
1import torch, torch.nn as nn, torch.nn.functional as F2from torchvision.models import inception_v3, Inception_V3_Weights34def conv_bn(cin, cout, k, pad): # torchvision's BasicConv2d5 return nn.Sequential(nn.Conv2d(cin, cout, k, padding=pad, bias=False),6 nn.BatchNorm2d(cout, eps=0.001), nn.ReLU(inplace=True))78branch7 = nn.Sequential( # Mixed_6b, "7x7" branch9 conv_bn(768, 128, 1, 0),10 conv_bn(128, 128, (1, 7), (0, 3)), # 1 x 711 conv_bn(128, 192, (7, 1), (3, 0))) # 7 x 112print(sum(p.numel() for p in branch7.parameters())) # 38592013print(branch7(torch.randn(1, 768, 17, 17)).shape) # torch.Size([1, 192, 17, 17])1415# Label smoothing is built into cross_entropy: q' = (1 - eps) * onehot + eps / K16logits, y = torch.randn(8, 1000), torch.randint(0, 1000, (8,))17loss = F.cross_entropy(logits, y, label_smoothing=0.1)1819# Pretrained model: 299 x 299 input; train() mode returns (logits, aux_logits)20weights = Inception_V3_Weights.IMAGENET1K_V121model = inception_v3(weights=weights)22print(sum(p.numel() for p in model.parameters())) # 27161264 (aux head included)23model.train()24out = model(torch.randn(2, 3, 299, 299)) # InceptionOutputs(logits, aux_logits)25loss = F.cross_entropy(out.logits, y[:2], label_smoothing=0.1) \26 + 0.4 * F.cross_entropy(out.aux_logits, y[:2], label_smoothing=0.1)27model.eval() # eval() returns plain logits28preprocess = weights.transforms() # resize 342, center-crop 299Use when
- +Factorized kernels give large receptive fields at a fraction of the cost: uses 29% of a 's weights.
- +Parallel stride-2 conv and pool branches shrink the grid without a representational bottleneck.
- +Strong accuracy for its budget in 2016 (21.2% top-1 single crop at about 5G multiply-adds), and label smoothing became a standard training trick.
Avoid when
- −Many module types (A–E) with hand-chosen widths: harder to scale and modify than ResNet's single repeated block.
- −Asymmetric convs and many narrow branches run less efficiently on GPUs than their FLOP count suggests.
- −Non-standard 299×299 input and preprocessing; superseded by ResNet variants and EfficientNet at equal compute.
Check yourself
A 7×7 conv with C input and output channels is replaced by a 1×7 conv followed by a 7×1 conv (both C → C). What fraction of the weights is saved?
Check yourself
With label smoothing ε = 0.1 on K = 1000 classes, what happens to the cross-entropy gradient with respect to the true-class logit once the model predicts p(y) = 0.9001?
Remember
- ✓Inception-v3 (Szegedy et al. 2016) rebuilds Inception from small kernels: two (−28%), (saves ).
- ✓Asymmetric factorization is used on the grid (); the grid uses parallel and convs.
- ✓Grid reduction runs a stride-2 conv branch and a stride-2 pool branch in parallel and concatenates them, avoiding a representational bottleneck.
- ✓Label smoothing was introduced here: with , worth about 0.2% absolute.
- ✓299×299 input, one BN auxiliary head, RMSProp; 21.2% / 5.6% single-crop error. torchvision's version: 27.2M params with aux head, 5.71 GFLOPS.