# ResNet

> The ResNet family: a stem, four stages at strides 4 to 32, basic blocks (ResNet-18/34) or 1×1–3×3–1×1 bottlenecks (50/101/152), and 1×1 projection shortcuts where shapes change. ResNet-50 has 25.6M parameters and 4.1G multiply-adds; a ResNet ensemble won ILSVRC-2015 with 3.57% top-5 error.

Source: https://www.mlfoundry.in/learn/resnet  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 13 min read · updated 2026-10-11

**ResNet:** is the family of ImageNet CNNs of He, Zhang, Ren and Sun, "Deep Residual Learning for Image Recognition" (CVPR 2016): ResNet-18, 34, 50, 101 and 152. Each is a stem, four **stages** of residual blocks that halve the grid and double the width, global average pooling and one fully connected layer. Only the block type and the number of blocks per stage change between depths. This article covers the architecture: the layouts, shortcut options, costs and later variants. Why residual blocks make deep nets trainable is covered in **Residual neural network (ResNet)**.

In 2015 networks had become deeper (VGG-19, GoogLeNet's 22 layers), but plain networks stopped improving beyond about 20–30 layers: deeper ones had higher *training* error. ResNet fixed this by letting each block learn a correction $F(x)$ added to an identity shortcut. That made depth cheap to use, and the paper went from 34 to 152 layers on ImageNet. An ensemble of ResNets, including two 152-layer models, reached **3.57% top-5 error** on the ILSVRC-2015 test set and won the classification task. The same backbones won the ImageNet detection and localization tasks and COCO detection and segmentation in 2015.

The layout followed simple rules taken from VGG: mostly $3\times3$ convs; within a stage every block has the same number of filters; when the grid is halved, the number of filters is doubled, so the cost per block stays about the same. Downsampling is done by a stride-2 conv at the start of each stage, not by pooling layers. Compared with VGG-19 (19.6G multiply-adds), ResNet-34 needs only 3.6G.

**At a glance**

- **Paper:** He, Zhang, Ren, Sun, CVPR 2016 (arXiv Dec. 2015)
- **Stages:** conv2_x–conv5_x at $56^2, 28^2, 14^2, 7^2$ (strides 4, 8, 16, 32 for a $224^2$ input)
- **Blocks:** basic $[3\times3, 3\times3]$ for 18/34; bottleneck $[1\times1, 3\times3, 1\times1]$ with 4× expansion for 50/101/152
- **ResNet-50:** [3, 4, 6, 3] bottlenecks; 25,557,032 params; 4.09 GFLOPS (torchvision; the paper's layout: 3.8G)
- **ILSVRC-2015:** 1st place, 3.57% top-5 test error (ensemble); ResNet-152 single model 4.49% top-5 (val, multi-scale)
- **Training (paper):** SGD, momentum 0.9, batch 256, lr 0.1 ÷ 10 on plateaus, weight decay $10^{-4}$, BN after every conv, no dropout
- **In PyTorch:** `torchvision.models.resnet18 … resnet152`; ResNet-50 V2 weights: 80.86% top-1

## Layouts: basic and bottleneck

The **basic block** is two $3\times3$ convs. The **bottleneck block** uses a $1\times1$ conv to reduce the width 4× (256 → 64 in conv2_x), a $3\times3$ conv at the reduced width, and a $1\times1$ conv to expand back. At 256 channels it has 17× fewer weights than two $3\times3$ convs, which is what lets ResNet-50 be deeper than ResNet-34 at almost the same cost. Counting conv and fc layers:

$$
\text{depth}=2+\ell\sum_{s=2}^{5}n_s,\qquad \ell=\begin{cases}2&\text{basic}\\3&\text{bottleneck}\end{cases},\qquad \text{MACs}(\text{conv})=k^2\,C_{in}\,C_{out}\,H_{out}W_{out}
$$

- $n_s$: number of blocks in stage $s$: [2,2,2,2] for 18, [3,4,6,3] for 34 and 50, [3,4,23,3] for 101, [3,8,36,3] for 152
- $2$: the $7\times7$ stem conv and the final fc layer
- $\text{MACs}$: multiply-accumulates of one conv layer; the paper and torchvision both call these FLOPs

So ResNet-50 has $2+3\cdot16=50$ layers and ResNet-34 has $2+2\cdot16=34$, with the same [3,4,6,3] stage layout. Doubling the width while halving the grid keeps a block's cost constant: $(2C)^2(H/2)^2=C^2H^2$. In ResNet-18 the last three stages each cost 411.0M multiply-adds.

ResNet family at 224×224. Params and FLOPs (multiply-adds) from torchvision's documented weights table; the paper's Table 1 FLOPs use the original stride placement. Top-1 accuracy: torchvision IMAGENET1K_V1 weights, single crop.

| Model | Block | Blocks per stage | Params | GFLOPS (torchvision) | GFLOPs (paper) | Top-1 (V1) |
| --- | --- | --- | --- | --- | --- | --- |
| ResNet-18 | basic | 2, 2, 2, 2 | 11,689,512 | 1.81 | 1.8 | 69.76% |
| ResNet-34 | basic | 3, 4, 6, 3 | 21,797,672 | 3.66 | 3.6 | 73.31% |
| ResNet-50 | bottleneck | 3, 4, 6, 3 | 25,557,032 | 4.09 | 3.8 | 76.13% |
| ResNet-101 | bottleneck | 3, 4, 23, 3 | 44,549,160 | 7.80 | 7.6 | 77.37% |
| ResNet-152 | bottleneck | 3, 8, 36, 3 | 60,192,808 | 11.51 | 11.3 | 78.31% |

## Shortcuts: options A, B and C

The add $y=F(x)+x$ needs matching shapes. At the first block of conv3_x–conv5_x the grid halves and the width doubles; in a bottleneck net, the first block of conv2_x also changes the width (64 → 256). The paper compared three choices for the shortcut:

$$
\text{A: }y=F(x)+\big[\,x_{::2,::2}\;\|\;\mathbf 0\,\big],\qquad \text{B: }y=F(x)+\begin{cases}x&\text{same shape}\\ \mathrm{BN}(W_s * x)&\text{shape changes}\end{cases},\qquad \text{C: }y=F(x)+\mathrm{BN}(W_s*x)\ \text{always}
$$

- $x_{::2,::2}\,\|\,\mathbf 0$: option A: subsample by 2 and pad the extra channels with zeros; no parameters
- $W_s$: a $1\times1$ conv (stride 2 where the grid halves): the projection shortcut
- $\text{B}$: the standard choice, used by the paper's deeper nets and by torchvision

On ResNet-34, 10-crop ImageNet top-1 error was 25.03% for A, 24.52% for B and 24.19% for C (paper, Table 3). B is better than A because the zero-padded channels do no residual learning. C is only marginally better than B, and the paper attributes that to its extra parameters. Since identity shortcuts are free, B became the standard. In bottleneck nets C is expensive: projections on all 16 blocks raise ResNet-50 from 25.6M to 40.1M parameters and from 4.09G to 6.56G multiply-adds (figure).

> Figure: ResNet-18/34/50/101/152 built from the real layout at 224×224. Bars are parameters and multiply-adds per stage; click a stage, then a block, to list its layers with output shape, parameters (conv weights + BN γ, β) and mult-adds. The default, ResNet-50 with projection shortcuts where the shape changes and the stride on the 3×3 (torchvision), totals 25,557,032 parameters and 4.09G mult-adds with 50 weighted layers, exactly torchvision's resnet50; every depth matches torchvision the same way. "Stride on 1×1" gives the paper's 3.86G (Table 1: 3.8G). Shortcut A drops ResNet-50 to 22,780,456 parameters; C raises it to 40,128,552 and 6.56G. "ResNet-D tweaks" (deep stem and avg-pool shortcuts) gives 25,576,264 parameters and 4.33G.

## Worked example: where ResNet-50 spends its budget

ResNet-50 per stage (torchvision layout; parameters include BN, computed by the figure's lib; totals match torchvision)

| Stage | Output | Blocks | Params | Share | Mult-adds |
| --- | --- | --- | --- | --- | --- |
| stem: 7×7/2 conv, max-pool | 64 × 56² | – | 9,536 | 0.04% | 118.0M |
| conv2_x | 256 × 56² | 3 | 215,808 | 0.8% | 667.9M |
| conv3_x | 512 × 28² | 4 | 1,219,584 | 4.8% | 1,027.6M |
| conv4_x | 1024 × 14² | 6 | 7,098,368 | 27.8% | 1,464.3M |
| conv5_x | 2048 × 7² | 3 | 14,964,736 | 58.6% | 809.2M |
| avg-pool + fc 2048 → 1000 | 1000 | – | 2,049,000 | 8.0% | 2.0M |

**The first block of conv5_x (1024 × 14² → 2048 × 7²)**

1. $1\times1$, 1024 → 512 at $14^2$: $1024\cdot512+2\cdot512=525{,}312$ parameters, $524{,}288\cdot196=102.8$M mult-adds.
2. $3\times3$/2, 512 → 512, output $7^2$: $9\cdot512^2+1024=2{,}360{,}320$ parameters, $2{,}359{,}296\cdot49=115.6$M.
3. $1\times1$, 512 → 2048: $1{,}048{,}576+4{,}096=1{,}052{,}672$ parameters, 51.4M.
4. Projection shortcut $1\times1$/2, 1024 → 2048: $2{,}097{,}152+4{,}096=2{,}101{,}248$ parameters, 102.8M. Block total 6,039,552.
5. The two identity blocks have 4,462,592 each, so conv5_x has $6{,}039{,}552+2\cdot4{,}462{,}592=14{,}964{,}736$: 58.6% of the parameters, but only 20% of the compute, because it runs on a $7\times7$ grid. conv4_x does the most work (1.46G).

## Variants: v1.5, v2, ResNet-D and better training

**Stride placement ("v1.5").** The paper puts the stride-2 of a bottleneck on its first $1\times1$ conv, which simply skips three out of four input positions. torchvision, and most code since, puts it on the $3\times3$ conv instead. This costs 4.09G instead of 3.86G multiply-adds for ResNet-50 and is somewhat more accurate; Bag of Tricks calls it ResNet-B.

**ResNet v2, pre-activation** (He et al., "Identity Mappings in Deep Residual Networks", ECCV 2016). The block becomes BN → ReLU → conv → BN → ReLU → conv, with nothing after the addition, so the shortcut path is a pure identity from input to output. This helps most for very deep nets: a 1001-layer ResNet reached 4.62% error on CIFAR-10, and on ImageNet a pre-activation ResNet-200 had 20.7% top-1 error versus 21.8% for the original (single $320\times320$ crop).

**ResNet-C and ResNet-D** (He, Zhang, Zhang, Zhang, Xie & Li, "Bag of Tricks for Image Classification", CVPR 2019). ResNet-C replaces the $7\times7$ stem with three $3\times3$ convs (32, 32, 64 channels). ResNet-D fixes the downsampling shortcut: a strided $1\times1$ conv ignores 3/4 of its input, so it becomes a $2\times2$ average pool followed by a $1\times1$ conv. With the paper's training setup, ResNet-50 went from 76.21% to 77.16% top-1 with B + C + D, at about 4.3G multiply-adds.

**Training, not architecture.** Much of what later architectures gained over ResNet came from better training. Wightman, Touvron and Jégou's "ResNet strikes back" (2021) trained an unchanged ResNet-50 for 600 epochs with LAMB, a binary cross-entropy loss, **Mixup**, **CutMix**, RandAugment, repeated augmentation and **Stochastic depth**, and reached **80.4%** top-1 at $224^2$, versus 76.1% for the original recipe. torchvision's IMAGENET1K_V2 weights for the same network, trained with a similar modern recipe, reach 80.86%.

```python
import torch, torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights
from torchvision.models.feature_extraction import create_feature_extractor

weights = ResNet50_Weights.IMAGENET1K_V2              # 80.858% top-1 (V1: 76.13%)
model = resnet50(weights=weights).eval()
print(sum(p.numel() for p in model.parameters()))      # 25557032
preprocess = weights.transforms()                      # resize 232, center-crop 224, normalize

# One feature map per stage: strides 4, 8, 16, 32 (used by FPN detectors)
fx = create_feature_extractor(model, return_nodes={"layer1": "c2", "layer2": "c3",
                                                   "layer3": "c4", "layer4": "c5"})
with torch.no_grad():
    feats = fx(torch.randn(1, 3, 224, 224))
print({k: tuple(v.shape) for k, v in feats.items()})
# {'c2': (1, 256, 56, 56), 'c3': (1, 512, 28, 28), 'c4': (1, 1024, 14, 14), 'c5': (1, 2048, 7, 7)}

print(model.layer4[0].downsample)                      # the option-B projection: 1x1 conv stride 2 + BN

# Fine-tune on 10 classes: new head, smaller learning rate for the backbone
model.fc = nn.Linear(model.fc.in_features, 10)         # 2048 -> 10
opt = torch.optim.SGD([
    {"params": [p for n, p in model.named_parameters() if not n.startswith("fc.")], "lr": 1e-3},
    {"params": model.fc.parameters(), "lr": 1e-2}], momentum=0.9, weight_decay=1e-4)

# Dense prediction: keep stride 8 by dilating the last two stages instead of striding
seg_backbone = resnet50(weights=weights, replace_stride_with_dilation=[False, True, True])
```

Using torchvision's ResNets: weights, feature maps per stage, fine-tuning

> **"ResNet-50" is not one network:** Papers compare against "ResNet-50" numbers anywhere from 75% to over 80% top-1. The differences come from the stride placement (v1 vs v1.5), the stem and shortcut tweaks (ResNet-C/D), and above all the training recipe: epochs, augmentation, label smoothing, EMA, test resolution. When comparing a new architecture to ResNet, check which variant and which recipe the baseline used. torchvision's V1 and V2 weights for the identical architecture differ by 4.7 points.

> **Why residual stages matter beyond ImageNet:** The four stages give feature maps at strides 4, 8, 16 and 32 with 256 to 2048 channels. That pyramid became the default input for detection (Faster R-CNN with FPN, RetinaNet), segmentation (DeepLab, Mask R-CNN) and many other tasks, because each stage can be read out directly. Even after vision transformers, ResNet-50 is the reference backbone for comparing training methods and self-supervised learning.

> **Successors:** **Wide ResNets** (2016) use fewer, wider blocks. **ResNeXt** (2017) replaces the $3\times3$ of the bottleneck with a grouped conv. **DenseNet** concatenates instead of adding. **SE-ResNet** adds channel attention. **ConvNeXt** (2022) re-designs ResNet step by step toward transformer-style choices. **Stochastic depth** drops whole residual blocks during training.

**Strengths**

- One block type and four numbers define the whole network; easy to scale from 18 to 152 layers.
- Identity shortcuts are free; bottlenecks make 50+ layers cost about as much as ResNet-34.
- Strong, well-understood backbone with a multi-scale feature pyramid; pretrained weights everywhere.
- Trains with plain SGD and BatchNorm; robust to hyperparameters.

**Limitations**

- Most parameters sit in the last stage (58.6% in conv5_x for ResNet-50), much of it redundant.
- The $7\times7$ stem and strided-$1\times1$ shortcuts discard information; fixed by ResNet-C/D.
- Depends on BatchNorm, which behaves poorly with very small batches.
- At equal FLOPs, newer designs (EfficientNet, RegNet, ConvNeXt) are more accurate, though partly due to training recipes.

**Question:** ResNet-34 and ResNet-50 use the same stage layout [3, 4, 6, 3]. Why is ResNet-50 deeper but only slightly more expensive (3.6G vs 3.8G multiply-adds in the paper)?

- A. ResNet-50 uses a smaller input resolution
- B. Each bottleneck does its 3×3 conv at a quarter of the block's width, with cheap 1×1 convs to reduce and expand
- C. ResNet-50 has no projection shortcuts
- D. ResNet-50 drops the fully connected layer

**Answer:** B. Each bottleneck does its 3×3 conv at a quarter of the block's width, with cheap 1×1 convs to reduce and expand — A bottleneck block has 3 layers instead of 2, so depth goes from $2+2\cdot16=34$ to $2+3\cdot16=50$. Its $3\times3$ runs at 64 channels inside a 256-channel block; at that width it has 17× fewer weights than two full $3\times3$ convs.

**Question:** In ResNet-50, which stage holds most of the parameters, and which does the most multiply-adds?

- A. conv2_x for both
- B. conv5_x holds most parameters; conv4_x does the most multiply-adds
- C. conv4_x for both
- D. The fc layer holds most parameters; the stem does the most compute

**Answer:** B. conv5_x holds most parameters; conv4_x does the most multiply-adds — conv5_x has 14.96M of 25.56M parameters (58.6%), because its convs are widest, but it runs on a $7\times7$ grid, so it costs only 809M multiply-adds. conv4_x has 6 blocks on a $14\times14$ grid and costs 1.46G, the most of any stage.

**Key points**

- ResNet = stem + four stages (grid $56\to7$, width doubles as the grid halves) + global average pool + fc; depth $=2+\ell\sum n_s$.
- Basic blocks (two $3\times3$) for 18/34; bottlenecks ($1\times1$ reduce, $3\times3$, $1\times1$ expand ×4) for 50/101/152 with [3,4,6,3], [3,4,23,3], [3,8,36,3].
- Shortcuts: identity where shapes match, 1×1 projection where they change (option B). A zero-pads, C projects everywhere; B is the standard.
- ResNet-50: 25,557,032 params, 4.09 GFLOPS (torchvision, stride on the 3×3) or 3.8G (paper); ILSVRC-2015 won with 3.57% top-5.
- Variants: pre-activation v2, ResNet-B/C/D from Bag of Tricks; with a modern recipe the same ResNet-50 reaches 80.4% (ResNet strikes back).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
