# CNN architecture

> Almost every CNN is one template: a strided stem, stages of conv-BN-ReLU blocks that halve the resolution and double the channels, global pooling and a linear head. Params pile up in the last stage; compute spreads evenly over the stages.

Source: https://www.mlfoundry.in/learn/cnn-architecture  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 13 min read · updated 2026-10-11

**CNN architecture:** is the arrangement of layers that turns an image tensor $(N,3,H,W)$ into predictions. The standard template is a **stem** that quickly reduces resolution, a sequence of **stages** (each a stack of identical blocks working at one resolution), a **global pooling** layer that removes the spatial dimensions, and a small **head** that maps the pooled feature vector to outputs.

The template is the result of twenty-five years of trial and error. LeNet-5 (LeCun et al., 1998) already alternated convolutions and subsampling and finished with fully connected layers. AlexNet (2012) scaled the recipe to ImageNet on GPUs. VGGNet (2014) showed that a regular design built only from stacked $3\times3$ convs works. ResNet (2015) added the identity shortcut that made 50 to 150 layers trainable. Since then most CNNs have been variations of the ResNet layout, with a different block inside each stage or a different rule for choosing depth and width.

Knowing the template is what makes new papers readable. "ResNeXt-50", "RegNetY-400MF" or "ConvNeXt-T" each name a block and a list of stage depths and widths. Given those, you can work out every tensor shape, the parameter count and the compute by hand.

## The template

**Stem → stages → pool → head (ResNet-50 at 224×224)**

1. **Stem.** A $7\times7$ conv with stride 2 (64 channels, BN, ReLU), then a $3\times3$ max-pool with stride 2: $(3,224,224)\to(64,56,56)$. Downsampling by 4 straight away keeps the expensive full-resolution processing to one cheap layer. Newer designs use three $3\times3$ convs or a $4\times4$ stride-4 "patchify" conv (ConvNeXt).
2. **Stages.** Four stages at $56^2$, $28^2$, $14^2$, $7^2$. Each is a stack of identical blocks, here bottlenecks (conv-BN-ReLU $1\times1$, $3\times3$, $1\times1$ plus an identity shortcut). The first block of stages 2 to 4 has stride 2, and its shortcut uses a $1\times1$ projection conv to match the new shape.
3. **Width rule.** Whenever the resolution halves, the width doubles: 256, 512, 1024, 2048 output channels.
4. **Global average pooling.** $(2048,7,7)\to(2048)$: the mean over all positions. This removes the dependence on input size and replaces the huge fully connected layers of AlexNet and VGG.
5. **Head.** One linear layer $2048\to1000$ for classification. Detection and segmentation keep the stage outputs instead and attach their own heads (feature pyramids, decoders).

The halve-and-double rule has a cost argument behind it. The ResNet paper puts it as: if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. A conv layer costs $k^2C_{in}C_{out}HW$ multiply-adds. Doubling both channel counts multiplies that by 4, and halving $H$ and $W$ divides it by 4. Each block therefore costs about the same, whichever stage it sits in. The number of activation values, $C\cdot H\cdot W$, halves per stage, and the weights per layer, $k^2C_{in}C_{out}$, grow by 4. Most of the memory sits in the early stages and most of the parameters in the late ones.

## Counting parameters and compute

$$
\#\text{params}=k^2\,\frac{C_{in}}{g}\,C_{out}\;(+\,C_{out}\text{ bias})\;(+\,2C_{out}\text{ BN}),\qquad \#\text{MACs}=k^2\,\frac{C_{in}}{g}\,C_{out}\,H_{out}W_{out}
$$

- $k$: kernel side; a $k\times k$ conv
- $g$: groups: $g=1$ for a normal conv, $g=C_{in}$ for a depthwise conv (see **Depthwise separable convolution**)
- $2C_{out}\text{ BN}$: BatchNorm's learned $\gamma,\beta$; the running mean and variance are buffers, not parameters. A conv followed by BN usually has `bias=False`
- $H_{out}W_{out}$: output positions: the same weights are applied at every one, so compute scales with image area and parameters do not
- $\text{MACs}$: multiply-accumulates. One MAC is 2 FLOPs, but papers and torchvision usually quote MACs and call them FLOPs

> Figure: A ResNet drawn stage by stage: block height is the spatial side, thickness the log of the channel count. Parameters and multiply-adds are counted layer by layer from the torchvision configuration (convs, BN γ/β and the fc layer). ResNet-50 comes out at 25.56M params and 4.09 GMACs over 50 weight layers. Stage 4 holds 59% of the parameters but only 20% of the compute, stage 3 does the most work (36%), and the stem costs 3%. On ResNet-18, stages 2, 3 and 4 cost exactly 411.0M MACs each, the halve-and-double rule at work. Raise the resolution to 448 and the compute rises to 16.35 GMACs with the parameter count unchanged; width ×2 instead gives 98.0M params and 16.1 GMACs.

Per-stage accounting of ResNet-50 at 224×224, computed by the figure's lib. The totals match torchvision's documented 25,557,032 parameters and 4.09 GFLOPS (multiply-adds).

| Part | Output (C, H, W) | Params | MACs | Share of params / MACs |
| --- | --- | --- | --- | --- |
| stem | 64, 56, 56 | 9,536 | 118.0M | 0.0% / 2.9% |
| stage 1 (3 blocks) | 256, 56, 56 | 215,808 | 667.9M | 0.8% / 16.3% |
| stage 2 (4 blocks) | 512, 28, 28 | 1,219,584 | 1027.6M | 4.8% / 25.1% |
| stage 3 (6 blocks) | 1024, 14, 14 | 7,098,368 | 1464.3M | 27.8% / 35.8% |
| stage 4 (3 blocks) | 2048, 7, 7 | 14,964,736 | 809.2M | 58.6% / 19.8% |
| pool + fc | 1000 | 2,049,000 | 2.0M | 8.0% / 0.1% |
| total |  | 25,557,032 | 4.09G |  |

**Worked example: one stage-3 bottleneck block (not the first)**

1. Input and output $(1024,14,14)$, bottleneck width 256. The $1\times1$ reduce conv has $1024\cdot256=262{,}144$ weights, the $3\times3$ conv $9\cdot256\cdot256=589{,}824$, and the $1\times1$ expand conv $256\cdot1024=262{,}144$.
2. Weights: $1{,}114{,}112$. BN adds $2\cdot(256+256+1024)=3072$, for $1{,}117{,}184$ parameters. The identity shortcut is free.
3. Every weight is used at all $14\cdot14=196$ positions: $1{,}114{,}112\cdot196=218.4\text{M}$ MACs. Five such blocks (1091.8M) plus the stride-2 first block (372.5M: its reduce conv still runs at $28^2$, and it has a $1\times1$ projection shortcut $512\to1024$) give the table's 1464.3M.
4. The stem's single $7\times7$ conv has only $49\cdot3\cdot64=9408$ weights but runs at $112^2$ positions: 118.0M MACs, more than half a bottleneck block. High resolution makes cheap layers expensive.

> **Two FLOP conventions and two ResNet-50s:** The **Residual neural network (ResNet)** article quotes the paper's 3.8 GFLOPs for ResNet-50. That model puts each stage's stride on the first $1\times1$ conv. torchvision ("ResNet v1.5") moves the stride onto the $3\times3$ conv, which costs more (4.09 GMACs) and is slightly more accurate. Both figures count multiply-adds. Tools that count a MAC as 2 FLOPs (e.g. `torch.utils.flop_counter`) report about 8.2 GFLOPs for the same network. Always check which convention a number uses.

## Design rules that survived

- **Small kernels, stacked:** Two $3\times3$ convs see $5\times5$ with $18C^2$ weights instead of $25C^2$, and add an extra nonlinearity (VGGNet). Large kernels came back only with depthwise convs (ConvNeXt's $7\times7$).
- **Halve resolution, double width:** Keeps the cost per block roughly constant across stages (VGGNet, ResNet). RegNet found good networks have widths that grow roughly linearly across blocks, quantised to a few stages.
- **Bottlenecks and cheap convs:** $1\times1$ convs to shrink channels before an expensive $3\times3$ (GoogLeNet / Inception, ResNet); grouped and depthwise convs to cut it further (ResNeXt, Xception, MobileNet, ShuffleNet).
- **Normalise and add shortcuts:** Conv-BN-ReLU as the basic unit, and identity shortcuts so depth does not hurt optimisation (ResNet, DenseNet's concatenations).
- **Global pooling, small head:** Global average pooling (Network in Network, GoogLeNet) instead of large fc layers: VGG-16's three fc layers hold 123.6M of its 138.4M parameters, 89%.
- **Scale everything together:** Grow depth, width and resolution in a fixed ratio rather than one at a time (EfficientNet's compound scaling).

## Timeline of the architectures in this module

Year of first publication (arXiv or conference) and the main idea each one added

| Year | Architecture | What it added |
| --- | --- | --- |
| 1998 | LeNet-5 | conv → subsample → conv → subsample → fc: the original template, trained end to end on digits |
| 2012 | AlexNet | the template at ImageNet scale: ReLU, dropout, data augmentation, two GPUs |
| 2013 | ZFNet | AlexNet tuned with deconvnet visualisations: smaller first-layer kernel and stride |
| 2014 | VGGNet | only 3×3 convs, 16 to 19 layers, a fully regular design |
| 2014 | GoogLeNet / Inception | parallel 1×1, 3×3, 5×5 and pool branches with 1×1 bottlenecks; global average pooling |
| 2015 | Inception-v3 | factorised convolutions (n×1 then 1×n), label smoothing, efficient grid reduction |
| 2015 | ResNet | identity shortcuts and bottleneck blocks; 50 to 152 layers |
| 2016 | SqueezeNet | AlexNet-level accuracy with 50× fewer parameters via fire modules |
| 2016 | ResNeXt | cardinality: grouped convs inside the bottleneck |
| 2016 | DenseNet | each layer receives the concatenated maps of all earlier layers in its block |
| 2016 | Xception | Inception taken to the limit: depthwise separable convs everywhere |
| 2017 | MobileNet | depthwise separable convs and width/resolution multipliers for phones |
| 2017 | ShuffleNet | grouped 1×1 convs plus channel shuffle |
| 2019 | EfficientNet | architecture search for a base net, then compound scaling of depth, width and resolution |
| 2020 | RegNet | design spaces: simple rules (linear width growth) instead of one searched model |
| 2020 | Vision Transformers (ViT) | no convolutions after a patchify stem: a Transformer over 16×16 patches |
| 2021 | Swin Transformer | attention in shifted local windows with a CNN-like four-stage hierarchy |
| 2022 | ConvNeXt | a ResNet modernised step by step towards Swin: patchify stem, 7×7 depthwise convs, LayerNorm, GELU |
| 2023 | ConvNeXt V2 | fully convolutional masked-autoencoder pre-training and global response normalisation |

Read the list as variations on the template. LeNet-5 to VGGNet settle the stem-stages-head layout. GoogLeNet and ResNet change what goes inside a block, and ResNeXt, DenseNet, Xception, MobileNet and ShuffleNet change it again for accuracy per FLOP. EfficientNet and RegNet change how the stage depths and widths are chosen. ViT replaces the blocks with Transformer layers but keeps a stem, while Swin and ConvNeXt bring the four-stage hierarchy back to both families.

```python
import torch, torch.nn as nn
from torch.utils.flop_counter import FlopCounterMode
from torchvision.models import resnet50

def conv_bn_relu(cin, cout, k=3, s=1):
    return nn.Sequential(nn.Conv2d(cin, cout, k, s, padding=k // 2, bias=False),
                         nn.BatchNorm2d(cout), nn.ReLU(inplace=True))

class TinyCNN(nn.Module):
    """stem -> 3 stages (halve H,W / double C) -> global pool -> linear head"""
    def __init__(self, classes=10, widths=(32, 64, 128), blocks=2):
        super().__init__()
        self.stem = conv_bn_relu(3, widths[0], k=3, s=2)
        stages, cin = [], widths[0]
        for i, w in enumerate(widths):
            layers = [conv_bn_relu(cin, w, s=1 if i == 0 else 2)]
            layers += [conv_bn_relu(w, w) for _ in range(blocks - 1)]
            stages.append(nn.Sequential(*layers)); cin = w
        self.stages = nn.Sequential(*stages)
        self.pool = nn.AdaptiveAvgPool2d(1)
        self.head = nn.Linear(cin, classes)
    def forward(self, x):
        return self.head(self.pool(self.stages(self.stem(x))).flatten(1))

model = resnet50(weights=None)
print(sum(p.numel() for p in model.parameters()))      # 25557032

with FlopCounterMode(display=False) as fc:
    model(torch.randn(1, 3, 224, 224))
print(fc.get_total_flops() / 2 / 1e9)                  # ~4.09 GMACs (counter counts 2 FLOPs per MAC)

# Shapes stage by stage
x = torch.randn(1, 3, 224, 224)
for name in ["conv1", "maxpool", "layer1", "layer2", "layer3", "layer4"]:
    x = getattr(model, name)(x) if name != "conv1" else model.relu(model.bn1(model.conv1(x)))
    print(name, tuple(x.shape))
# conv1 (1, 64, 112, 112) ... layer4 (1, 2048, 7, 7)
```

The template in a few lines, and how to count parameters and FLOPs

> **Where to spend a budget:** Compute scales with resolution squared, parameters with width squared, and both linearly with depth. To shrink a model for deployment, lower the width multiplier (parameters and MACs both fall roughly with its square; in the figure ×0.5 gives 6.9M params and 1.05 GMACs). To get more accuracy at a fixed compute budget, scale depth, width and resolution together (EfficientNet), not just one of them.

> **Backbone and head:** Everything before global pooling is called the **backbone**. Transfer learning reuses it: load ImageNet weights, replace `model.fc` with a new linear layer for your classes, and fine-tune. Detection (Faster R-CNN, RetinaNet) and segmentation models take the stage outputs at strides 8, 16 and 32 and build their own heads on top.

**Strengths**

- One template covers LeNet to ConvNeXt: knowing it makes any CNN paper readable
- Params and MACs can be counted exactly from the stage list
- Stage outputs give a multi-scale feature pyramid for detection and segmentation
- Halve-and-double keeps cost even across stages and easy to scale

**Limitations**

- Most parameters end up in the last stage, at the lowest resolution
- Early full-resolution stages dominate activation memory
- A fixed downsampling schedule (stride 32) loses small objects
- Many knobs (depths, widths, block type, resolution) interact; good settings came from large searches (EfficientNet, RegNet)

**Question:** A stage ends with (256, 28, 28). The next stage halves the resolution and doubles the width, keeping 3×3 convs. Compared with a 3×3 conv in the old stage, a 3×3 conv in the new stage has…

- A. 4× the parameters and about the same MACs
- B. the same parameters and 1/4 of the MACs
- C. 2× the parameters and 2× the MACs
- D. 4× the parameters and 4× the MACs

**Answer:** A. 4× the parameters and about the same MACs — Weights $9\cdot C_{in}C_{out}$ go from $9\cdot256^2$ to $9\cdot512^2$: ×4. MACs are weights × positions: ×4 for the weights, ÷4 for $14^2$ instead of $28^2$ positions, so about the same. This is the cost argument behind the halve-and-double rule.

**Question:** Why does replacing VGG's fully connected layers with global average pooling and one linear layer save so many parameters?

- A. Pooling has no parameters and the fc input shrinks from 512·7·7 = 25,088 to 512 values
- B. Global pooling reduces the number of conv layers
- C. Average pooling is cheaper than max-pooling
- D. It removes BatchNorm parameters

**Answer:** A. Pooling has no parameters and the fc input shrinks from 512·7·7 = 25,088 to 512 values — VGG-16's first fc layer alone has $25{,}088\cdot4096\approx102.8$M weights, because it connects every position of the $512\times7\times7$ map to 4096 units. After global pooling the head sees a 512-dim vector, and a linear layer to 1000 classes needs only about 0.5M weights.

**Key points**

- Template: strided stem → stages of identical conv-BN-ReLU (residual) blocks → global average pooling → linear head.
- Between stages, H and W halve and channels double, so each block costs about the same: $k^2C_{in}C_{out}HW$ MACs.
- Params $=k^2C_{in}C_{out}/g$ (+BN), MACs = params × output positions. ResNet-50 (torchvision): 25.56M params, 4.09 GMACs.
- Parameters concentrate in the last stage, memory in the first; resolution scales compute but not parameters.
- Every architecture in this module, from LeNet-5 to ConvNeXt V2, is a choice of block, stage depths, widths and stem within this template.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
