Residual neural network (ResNet)

Each block learns a correction F(x) added to an identity shortcut, so a stack of blocks can always fall back to the identity and gradients get a direct path to every layer.

11 min read1106 wordscore0 viewsupdated 2026-10-11

Definition

Residual neural network (ResNet) is a deep network built from residual blocks y=x+F(x)y=x+F(x): a few layers compute a correction F(x)F(x) that is added to the block's unchanged input through an identity shortcut. He, Zhang, Ren and Sun introduced it in 2015 and used it to train networks of 152 layers on ImageNet and over 1000 on CIFAR-10.

The degradation problem

By 2015 BatchNorm and He initialization had made 20–30-layer plain networks trainable. Going deeper should not hurt: a deeper net can copy a shallower one and set the extra layers to the identity. In practice it did hurt. On CIFAR-10 a plain 56-layer net had higher training error than a plain 20-layer net, and on ImageNet plain-34 had a higher validation top-1 error than plain-18 (28.54% vs 27.94%, 10-crop).

Higher training error means this is not overfitting. It is an optimization failure: SGD cannot find the "extra layers = identity" solution, because the identity is hard to represent with stacked ReLU layers, which start from small random weights. He et al. checked that the gradients of their BatchNorm plain nets did not vanish, so this was not simply vanishing gradients either.

The fix is to change what the layers are asked to learn. If the target mapping is H(x)H(x), let the layers fit the residual F(x)=H(x)−xF(x)=H(x)-x and add xx back. The identity then costs nothing: drive FF to zero. A layer that is not needed only has to learn to stay quiet.

The residual block

y=F(x,{Wi})+x,F(x)=W2 ReLU(BN(W1x))  (then BN, add, ReLU)y=F(x,\{W_i\})+x,\qquad F(x)=W_2\,\mathrm{ReLU}\big(\mathrm{BN}(W_1x)\big)\ \ \text{(then BN, add, ReLU)}
xx
block input (a feature map in a CNN)
FF
residual branch: two 3×33\times3 convolutions with BatchNorm in the basic block
+x+x
identity shortcut: no parameters, no extra FLOPs beyond the elementwise add

The addition needs F(x)F(x) and xx to have the same shape. When a stage halves the spatial size and doubles the channels, the shortcut becomes a projection y=F(x)+Wsxy=F(x)+W_sx, where WsW_s is a 1×11\times1 convolution with stride 2 (followed by BN). The paper compared options: zero-padding shortcuts (A), projections only when the shape changes (B), and projections everywhere (C). B and C were only marginally better than A, so the parameter-free identity is used wherever shapes match.

The two block types

  1. 1Basic block (ResNet-18/34): 3×33\times3 conv → BN → ReLU → 3×33\times3 conv → BN, add the shortcut, ReLU. Two layers per block.
  2. 2Bottleneck block (ResNet-50/101/152): 1×11\times1 conv reduces channels 4× (256 → 64), 3×33\times3 conv works at 64 channels, 1×11\times1 conv expands back to 256, then the add. Three layers per block.
  3. 3Bottleneck saving: at 256 channels it has 256⋅64+9⋅64⋅64+64⋅256=69,632256\cdot64+9\cdot64\cdot64+64\cdot256=69{,}632 conv weights, versus 2⋅9⋅256⋅256=1,179,6482\cdot9\cdot256\cdot256=1{,}179{,}648 for two full 3×33\times3 convs, about 17× fewer.
  4. 4Network layout: a 7×77\times7 stride-2 conv and max-pool, four stages at 64/128/256/512 base channels (each later stage starts with a stride-2 block and a projection shortcut), global average pooling, one fully connected layer.
ModelBlockBlocks per stageParamsFLOPsTop-1 / top-5 err.
ResNet-18basic2, 2, 2, 211.7M1.8G27.88% (top-1)
ResNet-34basic3, 4, 6, 321.8M3.6G24.52% / 7.46%
ResNet-50bottleneck3, 4, 6, 325.6M3.8G22.85% / 6.71%
ResNet-101bottleneck3, 4, 23, 344.5M7.6G21.75% / 6.05%
ResNet-152bottleneck3, 8, 36, 360.2M11.3G21.43% / 5.71%
Standard ImageNet ResNets. Depth counts conv + fc layers. FLOPs (multiply-adds) from He et al. 2015, Table 1; 10-crop validation errors from Tables 2–3 (ResNet-34 with projection option B); parameter counts from the torchvision layout.

ResNet-50 has 50 layers: 1+3⋅(3+4+6+3)+11 + 3\cdot(3+4+6+3) + 1. Swapping basic for bottleneck blocks adds 16 layers to ResNet-34 for only 0.2 GFLOPs more. An ensemble of ResNets reached 3.57% top-5 error on the ImageNet test set and won the ILSVRC 2015 classification task; the same backbones also won the ImageNet detection and localization and COCO detection and segmentation tracks.

Why gradients flow: the identity term

In the follow-up paper (He et al. 2016, "Identity Mappings in Deep Residual Networks") the ReLU after the addition moves into the branch (pre-activation: BN → ReLU → conv → BN → ReLU → conv). Then xl+1=xl+F(xl)x_{l+1}=x_l+F(x_l) exactly, and unrolling from any block ll to a deeper block LL gives a sum:

xL=xl+∑i=lL−1F(xi,Wi)⟹∂L∂xl=∂L∂xL(1+∂∂xl∑i=lL−1F(xi,Wi))x_L=x_l+\sum_{i=l}^{L-1}F(x_i,W_i)\quad\Longrightarrow\quad \frac{\partial\mathcal{L}}{\partial x_l}=\frac{\partial\mathcal{L}}{\partial x_L}\Big(1+\frac{\partial}{\partial x_l}\sum_{i=l}^{L-1}F(x_i,W_i)\Big)
xlx_l
input to block ll
∂L/∂xL\partial\mathcal{L}/\partial x_L
gradient at a deep block, passed to block l unchanged by the 1 term
∂∂xl∑F\frac{\partial}{\partial x_l}\sum F
contribution that goes through the residual branches

In a plain network the same gradient is a product of L−lL-l layer Jacobians, and a product of many factors drifts to 0 or ∞. Here the "1" is a direct path: the deep gradient reaches block ll without being multiplied by any weight. The branch term would have to equal exactly −1-1 for every sample in the batch to cancel it, which is very unlikely. Pre-activation let a 1001-layer ResNet reach 4.62% error on CIFAR-10.

simulated unit: pre-activation basic block ×28 · width 32 · 128 inputs x ∈ [−2, 2]hₗφWφW×β+hₗ₊₁identity shortcut: ∂hₗ₊₁/∂hₗ = I + β·∂F/∂hₗF(hₗ) = W·φ(W·φ(hₗ)), β = 0.189 · plain net = same boxes, no shortcutbackward: ‖∂y/∂hₖ‖ (log)1e-121e-81e-411e4k = 0 (input side)k = 56forward: RMS(hₖ) (log)1e-121e-81e-411e4k = 0 (input side)k = 56plain dy/dx vs x · lag-1 corr 0.71x = -2x = 2residual dy/dx vs x · lag-1 corr 0.93x = -2x = 2━ plain 56 layers ━ same weights + 28 shortcuts · strips scaled to own max
activation φ
W ~ N(0, s²/32)
branch β
‖∂y/∂h₀‖ / ‖∂y/∂h_L‖: plain 0.0396 · residual 1.81output RMS: 0.0812 · 1.90
fig.One real forward and backward pass through a 32-wide net, wired plain (rose) and with an identity shortcut every two layers (accent). Both use the same seeded weights. At the defaults (ReLU, He scale, 56 layers, branch scale β = 1/√28), the plain net's gradient shrinks 25× from output to input (ratio 0.040) and its signal falls to RMS 0.081. The residual net keeps both near 1 (1.81 and 1.90). Set s = 2 with tanh: the plain net's input gradient turns into noise (lag-1 correlation 0.04) while the residual one stays smooth (0.99). Set β = 1 to see why real ResNets need BatchNorm or a small branch scale.

The bottom strips show dy/dxdy/dx across 128 inputs. In deep plain nets the input gradient becomes shattered, close to white noise (Balduzzi et al. 2017), so a gradient step taken at one input says little about its neighbours. Residual nets keep it smooth. With β=1\beta=1 and no normalization, every block adds variance: at 56 layers the residual net's output RMS reaches about 1500. With β=1/B\beta=1/\sqrt{B} for BB blocks the growth stays bounded, since (1+1/B)B/2≤e≈1.65(1+1/B)^{B/2}\le\sqrt e\approx1.65 when the branch is roughly uncorrelated with its input. Real ResNets control this with BatchNorm, and many recipes go further by initialising the last BN scale in each branch to zero, so every block starts as the identity (Goyal et al. 2017; Fixup and SkipInit do the same without BN).

Worked example: paths through three blocks

Take a scalar linear residual chain xl+1=xl+wlxlx_{l+1}=x_l+w_lx_l with w=(0.5, −0.3, 0.2)w=(0.5,\,-0.3,\,0.2). Then ∂x3/∂x0=∏l(1+wl)=1.5⋅0.7⋅1.2=1.26\partial x_3/\partial x_0=\prod_l(1+w_l)=1.5\cdot0.7\cdot1.2=1.26. Expand the product: every term picks, at each block, either the shortcut (factor 1) or the branch (factor wlw_l). That gives 23=82^3=8 paths, grouped here by how many branches they pass through:

Branches usedPathsTermsSum
0111
130.5 − 0.3 + 0.20.4
23−0.15 + 0.10 − 0.06−0.11
310.5 · (−0.3) · 0.2−0.03
total81.26
Expanding (1 + w₁)(1 + w₂)(1 + w₃) into paths (exact arithmetic)

A plain chain xl+1=wlxlx_{l+1}=w_lx_l keeps only the last row: gradient −0.03-0.03, 42× smaller and with the opposite sign. With 50 blocks of w=0.1w=0.1 the plain product is 10−5010^{-50}, while the residual one is 1.150=117.41.1^{50}=117.4. With w=−0.1w=-0.1 it is 0.950=0.00520.9^{50}=0.0052. The shortcut removes the guaranteed collapse, but the branches still compound, which is why branch scale and normalization matter.

python
1import torch.nn as nn, torch.nn.functional as F
2
3class BasicBlock(nn.Module):
4 def __init__(self, cin, cout, stride=1):
5 super().__init__()
6 self.conv1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False); self.bn1 = nn.BatchNorm2d(cout)
7 self.conv2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False); self.bn2 = nn.BatchNorm2d(cout)
8 self.short = nn.Identity()
9 if stride != 1 or cin != cout: # shape changes: 1x1 projection
10 self.short = nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
11 nn.init.zeros_(self.bn2.weight) # block starts as identity (zero-init γ)
12
13 def forward(self, x):
14 out = F.relu(self.bn1(self.conv1(x)))
15 out = self.bn2(self.conv2(out))
16 return F.relu(out + self.short(x)) # y = F(x) + x
17
18class Bottleneck(nn.Module): # 1x1 reduce, 3x3, 1x1 expand (x4)
19 def __init__(self, cin, width, stride=1):
20 super().__init__()
21 cout = 4 * width
22 self.f = nn.Sequential(
23 nn.Conv2d(cin, width, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
24 nn.Conv2d(width, width, 3, stride, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
25 nn.Conv2d(width, cout, 1, bias=False), nn.BatchNorm2d(cout))
26 self.short = nn.Identity() if stride == 1 and cin == cout else nn.Sequential(
27 nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
28
29 def forward(self, x):
30 return F.relu(self.f(x) + self.short(x))
31
32print(sum(p.numel() for p in BasicBlock(64, 128, stride=2).parameters())) # 230144
33# torchvision.models.resnet50() has 25,557,032 parameters
Basic block with a projection shortcut, and the bottleneck block (torchvision layout)

Use when

  • +Very deep nets train with plain SGD; depth gives accuracy again instead of degradation.
  • +Identity shortcuts add no parameters and almost no compute.
  • +Robust: blocks can be dropped at test time (stochastic depth uses this during training).
  • +A simple, regular design that transfers well as a backbone for detection, segmentation and more.

Avoid when

  • −The add forces matching shapes, so stage transitions need projections.
  • −Without normalization or a small branch scale, the forward signal grows with depth.
  • −Many blocks only make small refinements: wide ResNets (Zagoruyko & Komodakis 2016) match much deeper thin ones with far fewer layers.
  • −The skip tensors must be kept in memory during the forward pass.

Check yourself

A plain 56-layer CNN has higher training error than a plain 20-layer one. What does this show?

Check yourself

With pre-activation blocks, ∂L/∂xl=∂L/∂xL (1+∂xl∑F)\partial\mathcal{L}/\partial x_l=\partial\mathcal{L}/\partial x_L\,(1+\partial_{x_l}\sum F). Why does this stop the gradient vanishing?

Remember

  • ✓Degradation: deeper plain nets had higher training error (56 vs 20 layers on CIFAR-10). That is an optimization problem, not overfitting.
  • ✓Residual block y=x+F(x)y=x+F(x): identity shortcuts where shapes match, 1×11\times1 projections where they change. Basic blocks use two 3×33\times3 convs; bottlenecks use 1×11\times1–3×33\times3–1×11\times1.
  • ✓Pre-activation gives xL=xl+∑Fx_L=x_l+\sum F, so ∂L/∂xl\partial\mathcal{L}/\partial x_l contains ∂L/∂xL\partial\mathcal{L}/\partial x_L directly.
  • ✓nn blocks give 2n2^n paths of Binomial length: a ResNet acts like an ensemble of mostly shallow paths.
  • ✓ResNet-18/34/50/101/152: 11.7M–60.2M params. The ResNet ensemble won ILSVRC 2015 with 3.57% top-5, and the residual stream is the backbone of transformers.
← back to Deep Learning roadmap