CNN architecture

Almost every CNN is one template: a strided stem, stages of conv-BN-ReLU blocks that halve the resolution and double the channels, global pooling and a linear head. Params pile up in the last stage; compute spreads evenly over the stages.

13 min read927 wordscore0 viewsupdated 2026-10-11

Definition

CNN architecture is the arrangement of layers that turns an image tensor (N,3,H,W)(N,3,H,W) into predictions. The standard template is a stem that quickly reduces resolution, a sequence of stages (each a stack of identical blocks working at one resolution), a global pooling layer that removes the spatial dimensions, and a small head that maps the pooled feature vector to outputs.

The template is the result of twenty-five years of trial and error. LeNet-5 (LeCun et al., 1998) already alternated convolutions and subsampling and finished with fully connected layers. AlexNet (2012) scaled the recipe to ImageNet on GPUs. VGGNet (2014) showed that a regular design built only from stacked 3×33\times3 convs works. ResNet (2015) added the identity shortcut that made 50 to 150 layers trainable. Since then most CNNs have been variations of the ResNet layout, with a different block inside each stage or a different rule for choosing depth and width.

Knowing the template is what makes new papers readable. "ResNeXt-50", "RegNetY-400MF" or "ConvNeXt-T" each name a block and a list of stage depths and widths. Given those, you can work out every tensor shape, the parameter count and the compute by hand.

The template

Stem → stages → pool → head (ResNet-50 at 224×224)

  1. 1Stem. A 7×77\times7 conv with stride 2 (64 channels, BN, ReLU), then a 3×33\times3 max-pool with stride 2: (3,224,224)→(64,56,56)(3,224,224)\to(64,56,56). Downsampling by 4 straight away keeps the expensive full-resolution processing to one cheap layer. Newer designs use three 3×33\times3 convs or a 4×44\times4 stride-4 "patchify" conv (ConvNeXt).
  2. 2Stages. Four stages at 56256^2, 28228^2, 14214^2, 727^2. Each is a stack of identical blocks, here bottlenecks (conv-BN-ReLU 1×11\times1, 3×33\times3, 1×11\times1 plus an identity shortcut). The first block of stages 2 to 4 has stride 2, and its shortcut uses a 1×11\times1 projection conv to match the new shape.
  3. 3Width rule. Whenever the resolution halves, the width doubles: 256, 512, 1024, 2048 output channels.
  4. 4Global average pooling. (2048,7,7)→(2048)(2048,7,7)\to(2048): the mean over all positions. This removes the dependence on input size and replaces the huge fully connected layers of AlexNet and VGG.
  5. 5Head. One linear layer 2048→10002048\to1000 for classification. Detection and segmentation keep the stage outputs instead and attach their own heads (feature pyramids, decoders).

The halve-and-double rule has a cost argument behind it. The ResNet paper puts it as: if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. A conv layer costs k2CinCoutHWk^2C_{in}C_{out}HW multiply-adds. Doubling both channel counts multiplies that by 4, and halving HH and WW divides it by 4. Each block therefore costs about the same, whichever stage it sits in. The number of activation values, C⋅H⋅WC\cdot H\cdot W, halves per stage, and the weights per layer, k2CinCoutk^2C_{in}C_{out}, grow by 4. Most of the memory sits in the early stages and most of the parameters in the late ones.

Counting parameters and compute

#params=k2 Cing Cout  (+ Cout bias)  (+ 2Cout BN),#MACs=k2 Cing Cout HoutWout\#\text{params}=k^2\,\frac{C_{in}}{g}\,C_{out}\;(+\,C_{out}\text{ bias})\;(+\,2C_{out}\text{ BN}),\qquad \#\text{MACs}=k^2\,\frac{C_{in}}{g}\,C_{out}\,H_{out}W_{out}
kk
kernel side; a k×kk\times k conv
gg
groups: g=1g=1 for a normal conv, g=Cing=C_{in} for a depthwise conv (see Depthwise separable convolution)
2Cout BN2C_{out}\text{ BN}
BatchNorm's learned γ,β\gamma,\beta; the running mean and variance are buffers, not parameters. A conv followed by BN usually has bias=False
HoutWoutH_{out}W_{out}
output positions: the same weights are applied at every one, so compute scales with image area and parameters do not
MACs\text{MACs}
multiply-accumulates. One MAC is 2 FLOPs, but papers and torchvision usually quote MACs and call them FLOPs
ResNet-50 · input 3×224×224 · click a stage224²stem6456²stage 125656²stage 251228²stage 3102414²stage 420487²head10001²height ∝ H×W side, thickness ∝ log₂ channels; labels: C, H²paramsmultiply-addsstem0%3%stage 11%16%stage 25%25%stage 328%36%stage 459%20%head8%0%stage 36 × [1×1 256, 3×3 256, 1×11024]out (N, 1024, 14, 14)params 7.10MMACs 1.46Gactivations 200.7k/imgtotal 25.56M params4.09 GMACs · 50 layersfp32 weights 97.5 MiB
fig.A ResNet drawn stage by stage: block height is the spatial side, thickness the log of the channel count. Parameters and multiply-adds are counted layer by layer from the torchvision configuration (convs, BN γ/β and the fc layer). ResNet-50 comes out at 25.56M params and 4.09 GMACs over 50 weight layers. Stage 4 holds 59% of the parameters but only 20% of the compute, stage 3 does the most work (36%), and the stem costs 3%. On ResNet-18, stages 2, 3 and 4 cost exactly 411.0M MACs each, the halve-and-double rule at work. Raise the resolution to 448 and the compute rises to 16.35 GMACs with the parameter count unchanged; width ×2 instead gives 98.0M params and 16.1 GMACs.
PartOutput (C, H, W)ParamsMACsShare of params / MACs
stem64, 56, 569,536118.0M0.0% / 2.9%
stage 1 (3 blocks)256, 56, 56215,808667.9M0.8% / 16.3%
stage 2 (4 blocks)512, 28, 281,219,5841027.6M4.8% / 25.1%
stage 3 (6 blocks)1024, 14, 147,098,3681464.3M27.8% / 35.8%
stage 4 (3 blocks)2048, 7, 714,964,736809.2M58.6% / 19.8%
pool + fc10002,049,0002.0M8.0% / 0.1%
total25,557,0324.09G
Per-stage accounting of ResNet-50 at 224×224, computed by the figure's lib. The totals match torchvision's documented 25,557,032 parameters and 4.09 GFLOPS (multiply-adds).

Worked example: one stage-3 bottleneck block (not the first)

  1. 1Input and output (1024,14,14)(1024,14,14), bottleneck width 256. The 1×11\times1 reduce conv has 1024⋅256=262,1441024\cdot256=262{,}144 weights, the 3×33\times3 conv 9⋅256⋅256=589,8249\cdot256\cdot256=589{,}824, and the 1×11\times1 expand conv 256⋅1024=262,144256\cdot1024=262{,}144.
  2. 2Weights: 1,114,1121{,}114{,}112. BN adds 2⋅(256+256+1024)=30722\cdot(256+256+1024)=3072, for 1,117,1841{,}117{,}184 parameters. The identity shortcut is free.
  3. 3Every weight is used at all 14⋅14=19614\cdot14=196 positions: 1,114,112⋅196=218.4M1{,}114{,}112\cdot196=218.4\text{M} MACs. Five such blocks (1091.8M) plus the stride-2 first block (372.5M: its reduce conv still runs at 28228^2, and it has a 1×11\times1 projection shortcut 512→1024512\to1024) give the table's 1464.3M.
  4. 4The stem's single 7×77\times7 conv has only 49⋅3⋅64=940849\cdot3\cdot64=9408 weights but runs at 1122112^2 positions: 118.0M MACs, more than half a bottleneck block. High resolution makes cheap layers expensive.

Design rules that survived

Where it's used

Small kernels, stacked

Two 3×33\times3 convs see 5×55\times5 with 18C218C^2 weights instead of 25C225C^2, and add an extra nonlinearity (VGGNet). Large kernels came back only with depthwise convs (ConvNeXt's 7×77\times7).

Halve resolution, double width

Keeps the cost per block roughly constant across stages (VGGNet, ResNet). RegNet found good networks have widths that grow roughly linearly across blocks, quantised to a few stages.

Bottlenecks and cheap convs

1×11\times1 convs to shrink channels before an expensive 3×33\times3 (GoogLeNet / Inception, ResNet); grouped and depthwise convs to cut it further (ResNeXt, Xception, MobileNet, ShuffleNet).

Normalise and add shortcuts

Conv-BN-ReLU as the basic unit, and identity shortcuts so depth does not hurt optimisation (ResNet, DenseNet's concatenations).

Global pooling, small head

Global average pooling (Network in Network, GoogLeNet) instead of large fc layers: VGG-16's three fc layers hold 123.6M of its 138.4M parameters, 89%.

Scale everything together

Grow depth, width and resolution in a fixed ratio rather than one at a time (EfficientNet's compound scaling).

Timeline of the architectures in this module

YearArchitectureWhat it added
1998LeNet-5conv → subsample → conv → subsample → fc: the original template, trained end to end on digits
2012AlexNetthe template at ImageNet scale: ReLU, dropout, data augmentation, two GPUs
2013ZFNetAlexNet tuned with deconvnet visualisations: smaller first-layer kernel and stride
2014VGGNetonly 3×3 convs, 16 to 19 layers, a fully regular design
2014GoogLeNet / Inceptionparallel 1×1, 3×3, 5×5 and pool branches with 1×1 bottlenecks; global average pooling
2015Inception-v3factorised convolutions (n×1 then 1×n), label smoothing, efficient grid reduction
2015ResNetidentity shortcuts and bottleneck blocks; 50 to 152 layers
2016SqueezeNetAlexNet-level accuracy with 50× fewer parameters via fire modules
2016ResNeXtcardinality: grouped convs inside the bottleneck
2016DenseNeteach layer receives the concatenated maps of all earlier layers in its block
2016XceptionInception taken to the limit: depthwise separable convs everywhere
2017MobileNetdepthwise separable convs and width/resolution multipliers for phones
2017ShuffleNetgrouped 1×1 convs plus channel shuffle
2019EfficientNetarchitecture search for a base net, then compound scaling of depth, width and resolution
2020RegNetdesign spaces: simple rules (linear width growth) instead of one searched model
2020Vision Transformers (ViT)no convolutions after a patchify stem: a Transformer over 16×16 patches
2021Swin Transformerattention in shifted local windows with a CNN-like four-stage hierarchy
2022ConvNeXta ResNet modernised step by step towards Swin: patchify stem, 7×7 depthwise convs, LayerNorm, GELU
2023ConvNeXt V2fully convolutional masked-autoencoder pre-training and global response normalisation
Year of first publication (arXiv or conference) and the main idea each one added

Read the list as variations on the template. LeNet-5 to VGGNet settle the stem-stages-head layout. GoogLeNet and ResNet change what goes inside a block, and ResNeXt, DenseNet, Xception, MobileNet and ShuffleNet change it again for accuracy per FLOP. EfficientNet and RegNet change how the stage depths and widths are chosen. ViT replaces the blocks with Transformer layers but keeps a stem, while Swin and ConvNeXt bring the four-stage hierarchy back to both families.

python
1import torch, torch.nn as nn
2from torch.utils.flop_counter import FlopCounterMode
3from torchvision.models import resnet50
4
5def conv_bn_relu(cin, cout, k=3, s=1):
6 return nn.Sequential(nn.Conv2d(cin, cout, k, s, padding=k // 2, bias=False),
7 nn.BatchNorm2d(cout), nn.ReLU(inplace=True))
8
9class TinyCNN(nn.Module):
10 """stem -> 3 stages (halve H,W / double C) -> global pool -> linear head"""
11 def __init__(self, classes=10, widths=(32, 64, 128), blocks=2):
12 super().__init__()
13 self.stem = conv_bn_relu(3, widths[0], k=3, s=2)
14 stages, cin = [], widths[0]
15 for i, w in enumerate(widths):
16 layers = [conv_bn_relu(cin, w, s=1 if i == 0 else 2)]
17 layers += [conv_bn_relu(w, w) for _ in range(blocks - 1)]
18 stages.append(nn.Sequential(*layers)); cin = w
19 self.stages = nn.Sequential(*stages)
20 self.pool = nn.AdaptiveAvgPool2d(1)
21 self.head = nn.Linear(cin, classes)
22 def forward(self, x):
23 return self.head(self.pool(self.stages(self.stem(x))).flatten(1))
24
25model = resnet50(weights=None)
26print(sum(p.numel() for p in model.parameters())) # 25557032
27
28with FlopCounterMode(display=False) as fc:
29 model(torch.randn(1, 3, 224, 224))
30print(fc.get_total_flops() / 2 / 1e9) # ~4.09 GMACs (counter counts 2 FLOPs per MAC)
31
32# Shapes stage by stage
33x = torch.randn(1, 3, 224, 224)
34for name in ["conv1", "maxpool", "layer1", "layer2", "layer3", "layer4"]:
35 x = getattr(model, name)(x) if name != "conv1" else model.relu(model.bn1(model.conv1(x)))
36 print(name, tuple(x.shape))
37# conv1 (1, 64, 112, 112) ... layer4 (1, 2048, 7, 7)
The template in a few lines, and how to count parameters and FLOPs

Use when

  • +One template covers LeNet to ConvNeXt: knowing it makes any CNN paper readable
  • +Params and MACs can be counted exactly from the stage list
  • +Stage outputs give a multi-scale feature pyramid for detection and segmentation
  • +Halve-and-double keeps cost even across stages and easy to scale

Avoid when

  • −Most parameters end up in the last stage, at the lowest resolution
  • −Early full-resolution stages dominate activation memory
  • −A fixed downsampling schedule (stride 32) loses small objects
  • −Many knobs (depths, widths, block type, resolution) interact; good settings came from large searches (EfficientNet, RegNet)

Check yourself

A stage ends with (256, 28, 28). The next stage halves the resolution and doubles the width, keeping 3×3 convs. Compared with a 3×3 conv in the old stage, a 3×3 conv in the new stage has…

Check yourself

Why does replacing VGG's fully connected layers with global average pooling and one linear layer save so many parameters?

Remember

  • ✓Template: strided stem → stages of identical conv-BN-ReLU (residual) blocks → global average pooling → linear head.
  • ✓Between stages, H and W halve and channels double, so each block costs about the same: k2CinCoutHWk^2C_{in}C_{out}HW MACs.
  • ✓Params =k2CinCout/g=k^2C_{in}C_{out}/g (+BN), MACs = params × output positions. ResNet-50 (torchvision): 25.56M params, 4.09 GMACs.
  • ✓Parameters concentrate in the last stage, memory in the first; resolution scales compute but not parameters.
  • ✓Every architecture in this module, from LeNet-5 to ConvNeXt V2, is a choice of block, stage depths, widths and stem within this template.
← back to Deep Learning roadmap