ResNet

The ResNet family: a stem, four stages at strides 4 to 32, basic blocks (ResNet-18/34) or 1×1–3×3–1×1 bottlenecks (50/101/152), and 1×1 projection shortcuts where shapes change. ResNet-50 has 25.6M parameters and 4.1G multiply-adds; a ResNet ensemble won ILSVRC-2015 with 3.57% top-5 error.

13 min read1127 wordscore0 viewsupdated 2026-10-11

Definition

ResNet is the family of ImageNet CNNs of He, Zhang, Ren and Sun, "Deep Residual Learning for Image Recognition" (CVPR 2016): ResNet-18, 34, 50, 101 and 152. Each is a stem, four stages of residual blocks that halve the grid and double the width, global average pooling and one fully connected layer. Only the block type and the number of blocks per stage change between depths. This article covers the architecture: the layouts, shortcut options, costs and later variants. Why residual blocks make deep nets trainable is covered in Residual neural network (ResNet).

In 2015 networks had become deeper (VGG-19, GoogLeNet's 22 layers), but plain networks stopped improving beyond about 20–30 layers: deeper ones had higher training error. ResNet fixed this by letting each block learn a correction F(x)F(x) added to an identity shortcut. That made depth cheap to use, and the paper went from 34 to 152 layers on ImageNet. An ensemble of ResNets, including two 152-layer models, reached 3.57% top-5 error on the ILSVRC-2015 test set and won the classification task. The same backbones won the ImageNet detection and localization tasks and COCO detection and segmentation in 2015.

The layout followed simple rules taken from VGG: mostly 3×33\times3 convs; within a stage every block has the same number of filters; when the grid is halved, the number of filters is doubled, so the cost per block stays about the same. Downsampling is done by a stride-2 conv at the start of each stage, not by pooling layers. Compared with VGG-19 (19.6G multiply-adds), ResNet-34 needs only 3.6G.

At a glance

Paper
He, Zhang, Ren, Sun, CVPR 2016 (arXiv Dec. 2015)
Stages
conv2_x–conv5_x at 562,282,142,7256^2, 28^2, 14^2, 7^2 (strides 4, 8, 16, 32 for a 2242224^2 input)
Blocks
basic [3×3,3×3][3\times3, 3\times3] for 18/34; bottleneck [1×1,3×3,1×1][1\times1, 3\times3, 1\times1] with 4× expansion for 50/101/152
ResNet-50
[3, 4, 6, 3] bottlenecks; 25,557,032 params; 4.09 GFLOPS (torchvision; the paper's layout: 3.8G)
ILSVRC-2015
1st place, 3.57% top-5 test error (ensemble); ResNet-152 single model 4.49% top-5 (val, multi-scale)
Training (paper)
SGD, momentum 0.9, batch 256, lr 0.1 ÷ 10 on plateaus, weight decay 10−410^{-4}, BN after every conv, no dropout
In PyTorch
torchvision.models.resnet18 … resnet152; ResNet-50 V2 weights: 80.86% top-1

Layouts: basic and bottleneck

The basic block is two 3×33\times3 convs. The bottleneck block uses a 1×11\times1 conv to reduce the width 4× (256 → 64 in conv2_x), a 3×33\times3 conv at the reduced width, and a 1×11\times1 conv to expand back. At 256 channels it has 17× fewer weights than two 3×33\times3 convs, which is what lets ResNet-50 be deeper than ResNet-34 at almost the same cost. Counting conv and fc layers:

depth=2+ℓ∑s=25ns,ℓ={2basic3bottleneck,MACs(conv)=k2 Cin Cout HoutWout\text{depth}=2+\ell\sum_{s=2}^{5}n_s,\qquad \ell=\begin{cases}2&\text{basic}\\3&\text{bottleneck}\end{cases},\qquad \text{MACs}(\text{conv})=k^2\,C_{in}\,C_{out}\,H_{out}W_{out}
nsn_s
number of blocks in stage ss: [2,2,2,2] for 18, [3,4,6,3] for 34 and 50, [3,4,23,3] for 101, [3,8,36,3] for 152
22
the 7×77\times7 stem conv and the final fc layer
MACs\text{MACs}
multiply-accumulates of one conv layer; the paper and torchvision both call these FLOPs

So ResNet-50 has 2+3⋅16=502+3\cdot16=50 layers and ResNet-34 has 2+2⋅16=342+2\cdot16=34, with the same [3,4,6,3] stage layout. Doubling the width while halving the grid keeps a block's cost constant: (2C)2(H/2)2=C2H2(2C)^2(H/2)^2=C^2H^2. In ResNet-18 the last three stages each cost 411.0M multiply-adds.

ModelBlockBlocks per stageParamsGFLOPS (torchvision)GFLOPs (paper)Top-1 (V1)
ResNet-18basic2, 2, 2, 211,689,5121.811.869.76%
ResNet-34basic3, 4, 6, 321,797,6723.663.673.31%
ResNet-50bottleneck3, 4, 6, 325,557,0324.093.876.13%
ResNet-101bottleneck3, 4, 23, 344,549,1607.807.677.37%
ResNet-152bottleneck3, 8, 36, 360,192,80811.5111.378.31%
ResNet family at 224×224. Params and FLOPs (multiply-adds) from torchvision's documented weights table; the paper's Table 1 FLOPs use the original stride placement. Top-1 accuracy: torchvision IMAGENET1K_V1 weights, single crop.

Shortcuts: options A, B and C

The add y=F(x)+xy=F(x)+x needs matching shapes. At the first block of conv3_x–conv5_x the grid halves and the width doubles; in a bottleneck net, the first block of conv2_x also changes the width (64 → 256). The paper compared three choices for the shortcut:

A: y=F(x)+[ x::2,::2  ∥  0 ],B: y=F(x)+{xsame shapeBN(Ws∗x)shape changes,C: y=F(x)+BN(Ws∗x) always\text{A: }y=F(x)+\big[\,x_{::2,::2}\;\|\;\mathbf 0\,\big],\qquad \text{B: }y=F(x)+\begin{cases}x&\text{same shape}\\ \mathrm{BN}(W_s * x)&\text{shape changes}\end{cases},\qquad \text{C: }y=F(x)+\mathrm{BN}(W_s*x)\ \text{always}
x::2,::2 ∥ 0x_{::2,::2}\,\|\,\mathbf 0
option A: subsample by 2 and pad the extra channels with zeros; no parameters
WsW_s
a 1×11\times1 conv (stride 2 where the grid halves): the projection shortcut
B\text{B}
the standard choice, used by the paper's deeper nets and by torchvision

On ResNet-34, 10-crop ImageNet top-1 error was 25.03% for A, 24.52% for B and 24.19% for C (paper, Table 3). B is better than A because the zero-padded channels do no residual learning. C is only marginally better than B, and the paper attributes that to its extra parameters. Since identity shortcuts are free, B became the standard. In bottleneck nets C is expensive: projections on all 16 blocks raise ResNet-50 from 25.6M to 40.1M parameters and from 4.09G to 6.56G multiply-adds (figure).

ResNet-50, 224×224 input · bottleneck blocks · click a stage, then a blockstem64×56²conv19.5K118.0Mconv2_x256×56²3 blocks215.8K667.9Mconv3_x512×28²4 blocks1.22M1.03Gconv4_x1024×14²6 blocks7.10M1.46Gconv5_x2048×7²3 blocks14.96M809.2Mhead1000fc2.05M2.0M■ params ■ multiply-adds (per stage)conv2_x: [1×1,64; 3×3,64; 1×1,256] × 3123block 1: 64×56² → 256×56², shortcut projectionlayeroutparamsmult-adds1×1, 6464×56²4,22412.8M3×3, 6464×56²36,992115.6M1×1, 256256×56²16,89651.4Mshortcut 1×1/1, 256256×56²16,89651.4Mtotal: 25,557,032 params · 4.09 G mult-adds · 50 weighted layerstorchvision resnet50: 25,557,032 params, 4.09 GFLOPS ✓ matchshortcut params in total: 2,776,576 · □ identity ■ 1×1 projection ■ zero-pad ■ avg-pool + 1×1
depth
shortcut
stride on
fig.ResNet-18/34/50/101/152 built from the real layout at 224×224. Bars are parameters and multiply-adds per stage; click a stage, then a block, to list its layers with output shape, parameters (conv weights + BN γ, β) and mult-adds. The default, ResNet-50 with projection shortcuts where the shape changes and the stride on the 3×3 (torchvision), totals 25,557,032 parameters and 4.09G mult-adds with 50 weighted layers, exactly torchvision's resnet50; every depth matches torchvision the same way. "Stride on 1×1" gives the paper's 3.86G (Table 1: 3.8G). Shortcut A drops ResNet-50 to 22,780,456 parameters; C raises it to 40,128,552 and 6.56G. "ResNet-D tweaks" (deep stem and avg-pool shortcuts) gives 25,576,264 parameters and 4.33G.

Worked example: where ResNet-50 spends its budget

StageOutputBlocksParamsShareMult-adds
stem: 7×7/2 conv, max-pool64 × 56²–9,5360.04%118.0M
conv2_x256 × 56²3215,8080.8%667.9M
conv3_x512 × 28²41,219,5844.8%1,027.6M
conv4_x1024 × 14²67,098,36827.8%1,464.3M
conv5_x2048 × 7²314,964,73658.6%809.2M
avg-pool + fc 2048 → 10001000–2,049,0008.0%2.0M
ResNet-50 per stage (torchvision layout; parameters include BN, computed by the figure's lib; totals match torchvision)

The first block of conv5_x (1024 × 14² → 2048 × 7²)

  1. 11×11\times1, 1024 → 512 at 14214^2: 1024⋅512+2⋅512=525,3121024\cdot512+2\cdot512=525{,}312 parameters, 524,288⋅196=102.8524{,}288\cdot196=102.8M mult-adds.
  2. 23×33\times3/2, 512 → 512, output 727^2: 9⋅5122+1024=2,360,3209\cdot512^2+1024=2{,}360{,}320 parameters, 2,359,296⋅49=115.62{,}359{,}296\cdot49=115.6M.
  3. 31×11\times1, 512 → 2048: 1,048,576+4,096=1,052,6721{,}048{,}576+4{,}096=1{,}052{,}672 parameters, 51.4M.
  4. 4Projection shortcut 1×11\times1/2, 1024 → 2048: 2,097,152+4,096=2,101,2482{,}097{,}152+4{,}096=2{,}101{,}248 parameters, 102.8M. Block total 6,039,552.
  5. 5The two identity blocks have 4,462,592 each, so conv5_x has 6,039,552+2⋅4,462,592=14,964,7366{,}039{,}552+2\cdot4{,}462{,}592=14{,}964{,}736: 58.6% of the parameters, but only 20% of the compute, because it runs on a 7×77\times7 grid. conv4_x does the most work (1.46G).

Variants: v1.5, v2, ResNet-D and better training

Stride placement ("v1.5"). The paper puts the stride-2 of a bottleneck on its first 1×11\times1 conv, which simply skips three out of four input positions. torchvision, and most code since, puts it on the 3×33\times3 conv instead. This costs 4.09G instead of 3.86G multiply-adds for ResNet-50 and is somewhat more accurate; Bag of Tricks calls it ResNet-B.

ResNet v2, pre-activation (He et al., "Identity Mappings in Deep Residual Networks", ECCV 2016). The block becomes BN → ReLU → conv → BN → ReLU → conv, with nothing after the addition, so the shortcut path is a pure identity from input to output. This helps most for very deep nets: a 1001-layer ResNet reached 4.62% error on CIFAR-10, and on ImageNet a pre-activation ResNet-200 had 20.7% top-1 error versus 21.8% for the original (single 320×320320\times320 crop).

ResNet-C and ResNet-D (He, Zhang, Zhang, Zhang, Xie & Li, "Bag of Tricks for Image Classification", CVPR 2019). ResNet-C replaces the 7×77\times7 stem with three 3×33\times3 convs (32, 32, 64 channels). ResNet-D fixes the downsampling shortcut: a strided 1×11\times1 conv ignores 3/4 of its input, so it becomes a 2×22\times2 average pool followed by a 1×11\times1 conv. With the paper's training setup, ResNet-50 went from 76.21% to 77.16% top-1 with B + C + D, at about 4.3G multiply-adds.

Training, not architecture. Much of what later architectures gained over ResNet came from better training. Wightman, Touvron and Jégou's "ResNet strikes back" (2021) trained an unchanged ResNet-50 for 600 epochs with LAMB, a binary cross-entropy loss, Mixup, CutMix, RandAugment, repeated augmentation and Stochastic depth, and reached 80.4% top-1 at 2242224^2, versus 76.1% for the original recipe. torchvision's IMAGENET1K_V2 weights for the same network, trained with a similar modern recipe, reach 80.86%.

python
1import torch, torch.nn as nn
2from torchvision.models import resnet50, ResNet50_Weights
3from torchvision.models.feature_extraction import create_feature_extractor
4
5weights = ResNet50_Weights.IMAGENET1K_V2 # 80.858% top-1 (V1: 76.13%)
6model = resnet50(weights=weights).eval()
7print(sum(p.numel() for p in model.parameters())) # 25557032
8preprocess = weights.transforms() # resize 232, center-crop 224, normalize
9
10# One feature map per stage: strides 4, 8, 16, 32 (used by FPN detectors)
11fx = create_feature_extractor(model, return_nodes={"layer1": "c2", "layer2": "c3",
12 "layer3": "c4", "layer4": "c5"})
13with torch.no_grad():
14 feats = fx(torch.randn(1, 3, 224, 224))
15print({k: tuple(v.shape) for k, v in feats.items()})
16# {'c2': (1, 256, 56, 56), 'c3': (1, 512, 28, 28), 'c4': (1, 1024, 14, 14), 'c5': (1, 2048, 7, 7)}
17
18print(model.layer4[0].downsample) # the option-B projection: 1x1 conv stride 2 + BN
19
20# Fine-tune on 10 classes: new head, smaller learning rate for the backbone
21model.fc = nn.Linear(model.fc.in_features, 10) # 2048 -> 10
22opt = torch.optim.SGD([
23 {"params": [p for n, p in model.named_parameters() if not n.startswith("fc.")], "lr": 1e-3},
24 {"params": model.fc.parameters(), "lr": 1e-2}], momentum=0.9, weight_decay=1e-4)
25
26# Dense prediction: keep stride 8 by dilating the last two stages instead of striding
27seg_backbone = resnet50(weights=weights, replace_stride_with_dilation=[False, True, True])
Using torchvision's ResNets: weights, feature maps per stage, fine-tuning

Use when

  • +One block type and four numbers define the whole network; easy to scale from 18 to 152 layers.
  • +Identity shortcuts are free; bottlenecks make 50+ layers cost about as much as ResNet-34.
  • +Strong, well-understood backbone with a multi-scale feature pyramid; pretrained weights everywhere.
  • +Trains with plain SGD and BatchNorm; robust to hyperparameters.

Avoid when

  • −Most parameters sit in the last stage (58.6% in conv5_x for ResNet-50), much of it redundant.
  • −The 7×77\times7 stem and strided-1×11\times1 shortcuts discard information; fixed by ResNet-C/D.
  • −Depends on BatchNorm, which behaves poorly with very small batches.
  • −At equal FLOPs, newer designs (EfficientNet, RegNet, ConvNeXt) are more accurate, though partly due to training recipes.

Check yourself

ResNet-34 and ResNet-50 use the same stage layout [3, 4, 6, 3]. Why is ResNet-50 deeper but only slightly more expensive (3.6G vs 3.8G multiply-adds in the paper)?

Check yourself

In ResNet-50, which stage holds most of the parameters, and which does the most multiply-adds?

Remember

  • ✓ResNet = stem + four stages (grid 56→756\to7, width doubles as the grid halves) + global average pool + fc; depth =2+ℓ∑ns=2+\ell\sum n_s.
  • ✓Basic blocks (two 3×33\times3) for 18/34; bottlenecks (1×11\times1 reduce, 3×33\times3, 1×11\times1 expand ×4) for 50/101/152 with [3,4,6,3], [3,4,23,3], [3,8,36,3].
  • ✓Shortcuts: identity where shapes match, 1×1 projection where they change (option B). A zero-pads, C projects everywhere; B is the standard.
  • ✓ResNet-50: 25,557,032 params, 4.09 GFLOPS (torchvision, stride on the 3×3) or 3.8G (paper); ILSVRC-2015 won with 3.57% top-5.
  • ✓Variants: pre-activation v2, ResNet-B/C/D from Bag of Tricks; with a modern recipe the same ResNet-50 reaches 80.4% (ResNet strikes back).
← back to Deep Learning roadmap