Definition
CNN architecture is the arrangement of layers that turns an image tensor into predictions. The standard template is a stem that quickly reduces resolution, a sequence of stages (each a stack of identical blocks working at one resolution), a global pooling layer that removes the spatial dimensions, and a small head that maps the pooled feature vector to outputs.
The template is the result of twenty-five years of trial and error. LeNet-5 (LeCun et al., 1998) already alternated convolutions and subsampling and finished with fully connected layers. AlexNet (2012) scaled the recipe to ImageNet on GPUs. VGGNet (2014) showed that a regular design built only from stacked convs works. ResNet (2015) added the identity shortcut that made 50 to 150 layers trainable. Since then most CNNs have been variations of the ResNet layout, with a different block inside each stage or a different rule for choosing depth and width.
Knowing the template is what makes new papers readable. "ResNeXt-50", "RegNetY-400MF" or "ConvNeXt-T" each name a block and a list of stage depths and widths. Given those, you can work out every tensor shape, the parameter count and the compute by hand.
The template
Stem → stages → pool → head (ResNet-50 at 224×224)
- 1Stem. A conv with stride 2 (64 channels, BN, ReLU), then a max-pool with stride 2: . Downsampling by 4 straight away keeps the expensive full-resolution processing to one cheap layer. Newer designs use three convs or a stride-4 "patchify" conv (ConvNeXt).
- 2Stages. Four stages at , , , . Each is a stack of identical blocks, here bottlenecks (conv-BN-ReLU , , plus an identity shortcut). The first block of stages 2 to 4 has stride 2, and its shortcut uses a projection conv to match the new shape.
- 3Width rule. Whenever the resolution halves, the width doubles: 256, 512, 1024, 2048 output channels.
- 4Global average pooling. : the mean over all positions. This removes the dependence on input size and replaces the huge fully connected layers of AlexNet and VGG.
- 5Head. One linear layer for classification. Detection and segmentation keep the stage outputs instead and attach their own heads (feature pyramids, decoders).
The halve-and-double rule has a cost argument behind it. The ResNet paper puts it as: if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. A conv layer costs multiply-adds. Doubling both channel counts multiplies that by 4, and halving and divides it by 4. Each block therefore costs about the same, whichever stage it sits in. The number of activation values, , halves per stage, and the weights per layer, , grow by 4. Most of the memory sits in the early stages and most of the parameters in the late ones.
Counting parameters and compute
- kernel side; a conv
- groups: for a normal conv, for a depthwise conv (see Depthwise separable convolution)
- BatchNorm's learned ; the running mean and variance are buffers, not parameters. A conv followed by BN usually has
bias=False - output positions: the same weights are applied at every one, so compute scales with image area and parameters do not
- multiply-accumulates. One MAC is 2 FLOPs, but papers and torchvision usually quote MACs and call them FLOPs
| Part | Output (C, H, W) | Params | MACs | Share of params / MACs |
|---|---|---|---|---|
| stem | 64, 56, 56 | 9,536 | 118.0M | 0.0% / 2.9% |
| stage 1 (3 blocks) | 256, 56, 56 | 215,808 | 667.9M | 0.8% / 16.3% |
| stage 2 (4 blocks) | 512, 28, 28 | 1,219,584 | 1027.6M | 4.8% / 25.1% |
| stage 3 (6 blocks) | 1024, 14, 14 | 7,098,368 | 1464.3M | 27.8% / 35.8% |
| stage 4 (3 blocks) | 2048, 7, 7 | 14,964,736 | 809.2M | 58.6% / 19.8% |
| pool + fc | 1000 | 2,049,000 | 2.0M | 8.0% / 0.1% |
| total | 25,557,032 | 4.09G |
Worked example: one stage-3 bottleneck block (not the first)
- 1Input and output , bottleneck width 256. The reduce conv has weights, the conv , and the expand conv .
- 2Weights: . BN adds , for parameters. The identity shortcut is free.
- 3Every weight is used at all positions: MACs. Five such blocks (1091.8M) plus the stride-2 first block (372.5M: its reduce conv still runs at , and it has a projection shortcut ) give the table's 1464.3M.
- 4The stem's single conv has only weights but runs at positions: 118.0M MACs, more than half a bottleneck block. High resolution makes cheap layers expensive.
Design rules that survived
Where it's used
Small kernels, stacked
Two convs see with weights instead of , and add an extra nonlinearity (VGGNet). Large kernels came back only with depthwise convs (ConvNeXt's ).
Halve resolution, double width
Keeps the cost per block roughly constant across stages (VGGNet, ResNet). RegNet found good networks have widths that grow roughly linearly across blocks, quantised to a few stages.
Bottlenecks and cheap convs
convs to shrink channels before an expensive (GoogLeNet / Inception, ResNet); grouped and depthwise convs to cut it further (ResNeXt, Xception, MobileNet, ShuffleNet).
Normalise and add shortcuts
Conv-BN-ReLU as the basic unit, and identity shortcuts so depth does not hurt optimisation (ResNet, DenseNet's concatenations).
Global pooling, small head
Global average pooling (Network in Network, GoogLeNet) instead of large fc layers: VGG-16's three fc layers hold 123.6M of its 138.4M parameters, 89%.
Scale everything together
Grow depth, width and resolution in a fixed ratio rather than one at a time (EfficientNet's compound scaling).
Timeline of the architectures in this module
| Year | Architecture | What it added |
|---|---|---|
| 1998 | LeNet-5 | conv → subsample → conv → subsample → fc: the original template, trained end to end on digits |
| 2012 | AlexNet | the template at ImageNet scale: ReLU, dropout, data augmentation, two GPUs |
| 2013 | ZFNet | AlexNet tuned with deconvnet visualisations: smaller first-layer kernel and stride |
| 2014 | VGGNet | only 3×3 convs, 16 to 19 layers, a fully regular design |
| 2014 | GoogLeNet / Inception | parallel 1×1, 3×3, 5×5 and pool branches with 1×1 bottlenecks; global average pooling |
| 2015 | Inception-v3 | factorised convolutions (n×1 then 1×n), label smoothing, efficient grid reduction |
| 2015 | ResNet | identity shortcuts and bottleneck blocks; 50 to 152 layers |
| 2016 | SqueezeNet | AlexNet-level accuracy with 50× fewer parameters via fire modules |
| 2016 | ResNeXt | cardinality: grouped convs inside the bottleneck |
| 2016 | DenseNet | each layer receives the concatenated maps of all earlier layers in its block |
| 2016 | Xception | Inception taken to the limit: depthwise separable convs everywhere |
| 2017 | MobileNet | depthwise separable convs and width/resolution multipliers for phones |
| 2017 | ShuffleNet | grouped 1×1 convs plus channel shuffle |
| 2019 | EfficientNet | architecture search for a base net, then compound scaling of depth, width and resolution |
| 2020 | RegNet | design spaces: simple rules (linear width growth) instead of one searched model |
| 2020 | Vision Transformers (ViT) | no convolutions after a patchify stem: a Transformer over 16×16 patches |
| 2021 | Swin Transformer | attention in shifted local windows with a CNN-like four-stage hierarchy |
| 2022 | ConvNeXt | a ResNet modernised step by step towards Swin: patchify stem, 7×7 depthwise convs, LayerNorm, GELU |
| 2023 | ConvNeXt V2 | fully convolutional masked-autoencoder pre-training and global response normalisation |
Read the list as variations on the template. LeNet-5 to VGGNet settle the stem-stages-head layout. GoogLeNet and ResNet change what goes inside a block, and ResNeXt, DenseNet, Xception, MobileNet and ShuffleNet change it again for accuracy per FLOP. EfficientNet and RegNet change how the stage depths and widths are chosen. ViT replaces the blocks with Transformer layers but keeps a stem, while Swin and ConvNeXt bring the four-stage hierarchy back to both families.
1import torch, torch.nn as nn2from torch.utils.flop_counter import FlopCounterMode3from torchvision.models import resnet5045def conv_bn_relu(cin, cout, k=3, s=1):6 return nn.Sequential(nn.Conv2d(cin, cout, k, s, padding=k // 2, bias=False),7 nn.BatchNorm2d(cout), nn.ReLU(inplace=True))89class TinyCNN(nn.Module):10 """stem -> 3 stages (halve H,W / double C) -> global pool -> linear head"""11 def __init__(self, classes=10, widths=(32, 64, 128), blocks=2):12 super().__init__()13 self.stem = conv_bn_relu(3, widths[0], k=3, s=2)14 stages, cin = [], widths[0]15 for i, w in enumerate(widths):16 layers = [conv_bn_relu(cin, w, s=1 if i == 0 else 2)]17 layers += [conv_bn_relu(w, w) for _ in range(blocks - 1)]18 stages.append(nn.Sequential(*layers)); cin = w19 self.stages = nn.Sequential(*stages)20 self.pool = nn.AdaptiveAvgPool2d(1)21 self.head = nn.Linear(cin, classes)22 def forward(self, x):23 return self.head(self.pool(self.stages(self.stem(x))).flatten(1))2425model = resnet50(weights=None)26print(sum(p.numel() for p in model.parameters())) # 255570322728with FlopCounterMode(display=False) as fc:29 model(torch.randn(1, 3, 224, 224))30print(fc.get_total_flops() / 2 / 1e9) # ~4.09 GMACs (counter counts 2 FLOPs per MAC)3132# Shapes stage by stage33x = torch.randn(1, 3, 224, 224)34for name in ["conv1", "maxpool", "layer1", "layer2", "layer3", "layer4"]:35 x = getattr(model, name)(x) if name != "conv1" else model.relu(model.bn1(model.conv1(x)))36 print(name, tuple(x.shape))37# conv1 (1, 64, 112, 112) ... layer4 (1, 2048, 7, 7)Use when
- +One template covers LeNet to ConvNeXt: knowing it makes any CNN paper readable
- +Params and MACs can be counted exactly from the stage list
- +Stage outputs give a multi-scale feature pyramid for detection and segmentation
- +Halve-and-double keeps cost even across stages and easy to scale
Avoid when
- −Most parameters end up in the last stage, at the lowest resolution
- −Early full-resolution stages dominate activation memory
- −A fixed downsampling schedule (stride 32) loses small objects
- −Many knobs (depths, widths, block type, resolution) interact; good settings came from large searches (EfficientNet, RegNet)
Check yourself
A stage ends with (256, 28, 28). The next stage halves the resolution and doubles the width, keeping 3×3 convs. Compared with a 3×3 conv in the old stage, a 3×3 conv in the new stage has…
Check yourself
Why does replacing VGG's fully connected layers with global average pooling and one linear layer save so many parameters?
Remember
- ✓Template: strided stem → stages of identical conv-BN-ReLU (residual) blocks → global average pooling → linear head.
- ✓Between stages, H and W halve and channels double, so each block costs about the same: MACs.
- ✓Params (+BN), MACs = params × output positions. ResNet-50 (torchvision): 25.56M params, 4.09 GMACs.
- ✓Parameters concentrate in the last stage, memory in the first; resolution scales compute but not parameters.
- ✓Every architecture in this module, from LeNet-5 to ConvNeXt V2, is a choice of block, stage depths, widths and stem within this template.