Definition
Residual neural network (ResNet) is a deep network built from residual blocks : a few layers compute a correction that is added to the block's unchanged input through an identity shortcut. He, Zhang, Ren and Sun introduced it in 2015 and used it to train networks of 152 layers on ImageNet and over 1000 on CIFAR-10.
The degradation problem
By 2015 BatchNorm and He initialization had made 20–30-layer plain networks trainable. Going deeper should not hurt: a deeper net can copy a shallower one and set the extra layers to the identity. In practice it did hurt. On CIFAR-10 a plain 56-layer net had higher training error than a plain 20-layer net, and on ImageNet plain-34 had a higher validation top-1 error than plain-18 (28.54% vs 27.94%, 10-crop).
Higher training error means this is not overfitting. It is an optimization failure: SGD cannot find the "extra layers = identity" solution, because the identity is hard to represent with stacked ReLU layers, which start from small random weights. He et al. checked that the gradients of their BatchNorm plain nets did not vanish, so this was not simply vanishing gradients either.
The fix is to change what the layers are asked to learn. If the target mapping is , let the layers fit the residual and add back. The identity then costs nothing: drive to zero. A layer that is not needed only has to learn to stay quiet.
The residual block
- block input (a feature map in a CNN)
- residual branch: two convolutions with BatchNorm in the basic block
- identity shortcut: no parameters, no extra FLOPs beyond the elementwise add
The addition needs and to have the same shape. When a stage halves the spatial size and doubles the channels, the shortcut becomes a projection , where is a convolution with stride 2 (followed by BN). The paper compared options: zero-padding shortcuts (A), projections only when the shape changes (B), and projections everywhere (C). B and C were only marginally better than A, so the parameter-free identity is used wherever shapes match.
The two block types
- 1Basic block (ResNet-18/34): conv → BN → ReLU → conv → BN, add the shortcut, ReLU. Two layers per block.
- 2Bottleneck block (ResNet-50/101/152): conv reduces channels 4× (256 → 64), conv works at 64 channels, conv expands back to 256, then the add. Three layers per block.
- 3Bottleneck saving: at 256 channels it has conv weights, versus for two full convs, about 17× fewer.
- 4Network layout: a stride-2 conv and max-pool, four stages at 64/128/256/512 base channels (each later stage starts with a stride-2 block and a projection shortcut), global average pooling, one fully connected layer.
| Model | Block | Blocks per stage | Params | FLOPs | Top-1 / top-5 err. |
|---|---|---|---|---|---|
| ResNet-18 | basic | 2, 2, 2, 2 | 11.7M | 1.8G | 27.88% (top-1) |
| ResNet-34 | basic | 3, 4, 6, 3 | 21.8M | 3.6G | 24.52% / 7.46% |
| ResNet-50 | bottleneck | 3, 4, 6, 3 | 25.6M | 3.8G | 22.85% / 6.71% |
| ResNet-101 | bottleneck | 3, 4, 23, 3 | 44.5M | 7.6G | 21.75% / 6.05% |
| ResNet-152 | bottleneck | 3, 8, 36, 3 | 60.2M | 11.3G | 21.43% / 5.71% |
ResNet-50 has 50 layers: . Swapping basic for bottleneck blocks adds 16 layers to ResNet-34 for only 0.2 GFLOPs more. An ensemble of ResNets reached 3.57% top-5 error on the ImageNet test set and won the ILSVRC 2015 classification task; the same backbones also won the ImageNet detection and localization and COCO detection and segmentation tracks.
Why gradients flow: the identity term
In the follow-up paper (He et al. 2016, "Identity Mappings in Deep Residual Networks") the ReLU after the addition moves into the branch (pre-activation: BN → ReLU → conv → BN → ReLU → conv). Then exactly, and unrolling from any block to a deeper block gives a sum:
- input to block
- gradient at a deep block, passed to block l unchanged by the 1 term
- contribution that goes through the residual branches
In a plain network the same gradient is a product of layer Jacobians, and a product of many factors drifts to 0 or ∞. Here the "1" is a direct path: the deep gradient reaches block without being multiplied by any weight. The branch term would have to equal exactly for every sample in the batch to cancel it, which is very unlikely. Pre-activation let a 1001-layer ResNet reach 4.62% error on CIFAR-10.
The bottom strips show across 128 inputs. In deep plain nets the input gradient becomes shattered, close to white noise (Balduzzi et al. 2017), so a gradient step taken at one input says little about its neighbours. Residual nets keep it smooth. With and no normalization, every block adds variance: at 56 layers the residual net's output RMS reaches about 1500. With for blocks the growth stays bounded, since when the branch is roughly uncorrelated with its input. Real ResNets control this with BatchNorm, and many recipes go further by initialising the last BN scale in each branch to zero, so every block starts as the identity (Goyal et al. 2017; Fixup and SkipInit do the same without BN).
Worked example: paths through three blocks
Take a scalar linear residual chain with . Then . Expand the product: every term picks, at each block, either the shortcut (factor 1) or the branch (factor ). That gives paths, grouped here by how many branches they pass through:
| Branches used | Paths | Terms | Sum |
|---|---|---|---|
| 0 | 1 | 1 | 1 |
| 1 | 3 | 0.5 − 0.3 + 0.2 | 0.4 |
| 2 | 3 | −0.15 + 0.10 − 0.06 | −0.11 |
| 3 | 1 | 0.5 · (−0.3) · 0.2 | −0.03 |
| total | 8 | 1.26 |
A plain chain keeps only the last row: gradient , 42× smaller and with the opposite sign. With 50 blocks of the plain product is , while the residual one is . With it is . The shortcut removes the guaranteed collapse, but the branches still compound, which is why branch scale and normalization matter.
1import torch.nn as nn, torch.nn.functional as F23class BasicBlock(nn.Module):4 def __init__(self, cin, cout, stride=1):5 super().__init__()6 self.conv1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False); self.bn1 = nn.BatchNorm2d(cout)7 self.conv2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False); self.bn2 = nn.BatchNorm2d(cout)8 self.short = nn.Identity()9 if stride != 1 or cin != cout: # shape changes: 1x1 projection10 self.short = nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))11 nn.init.zeros_(self.bn2.weight) # block starts as identity (zero-init γ)1213 def forward(self, x):14 out = F.relu(self.bn1(self.conv1(x)))15 out = self.bn2(self.conv2(out))16 return F.relu(out + self.short(x)) # y = F(x) + x1718class Bottleneck(nn.Module): # 1x1 reduce, 3x3, 1x1 expand (x4)19 def __init__(self, cin, width, stride=1):20 super().__init__()21 cout = 4 * width22 self.f = nn.Sequential(23 nn.Conv2d(cin, width, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),24 nn.Conv2d(width, width, 3, stride, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),25 nn.Conv2d(width, cout, 1, bias=False), nn.BatchNorm2d(cout))26 self.short = nn.Identity() if stride == 1 and cin == cout else nn.Sequential(27 nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))2829 def forward(self, x):30 return F.relu(self.f(x) + self.short(x))3132print(sum(p.numel() for p in BasicBlock(64, 128, stride=2).parameters())) # 23014433# torchvision.models.resnet50() has 25,557,032 parametersUse when
- +Very deep nets train with plain SGD; depth gives accuracy again instead of degradation.
- +Identity shortcuts add no parameters and almost no compute.
- +Robust: blocks can be dropped at test time (stochastic depth uses this during training).
- +A simple, regular design that transfers well as a backbone for detection, segmentation and more.
Avoid when
- −The add forces matching shapes, so stage transitions need projections.
- −Without normalization or a small branch scale, the forward signal grows with depth.
- −Many blocks only make small refinements: wide ResNets (Zagoruyko & Komodakis 2016) match much deeper thin ones with far fewer layers.
- −The skip tensors must be kept in memory during the forward pass.
Check yourself
A plain 56-layer CNN has higher training error than a plain 20-layer one. What does this show?
Check yourself
With pre-activation blocks, . Why does this stop the gradient vanishing?
Remember
- ✓Degradation: deeper plain nets had higher training error (56 vs 20 layers on CIFAR-10). That is an optimization problem, not overfitting.
- ✓Residual block : identity shortcuts where shapes match, projections where they change. Basic blocks use two convs; bottlenecks use ––.
- ✓Pre-activation gives , so contains directly.
- ✓ blocks give paths of Binomial length: a ResNet acts like an ensemble of mostly shallow paths.
- ✓ResNet-18/34/50/101/152: 11.7M–60.2M params. The ResNet ensemble won ILSVRC 2015 with 3.57% top-5, and the residual stream is the backbone of transformers.