# Residual neural network (ResNet)

> Each block learns a correction F(x) added to an identity shortcut, so a stack of blocks can always fall back to the identity and gradients get a direct path to every layer.

Source: https://www.mlfoundry.in/learn/residual-neural-network-resnet  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 11 min read · updated 2026-10-11

**Residual neural network (ResNet):** is a deep network built from **residual blocks** $y=x+F(x)$: a few layers compute a correction $F(x)$ that is added to the block's unchanged input through an **identity shortcut**. He, Zhang, Ren and Sun introduced it in 2015 and used it to train networks of 152 layers on ImageNet and over 1000 on CIFAR-10.

## The degradation problem

By 2015 BatchNorm and He initialization had made 20–30-layer plain networks trainable. Going deeper should not hurt: a deeper net can copy a shallower one and set the extra layers to the identity. In practice it did hurt. On CIFAR-10 a plain 56-layer net had **higher training error** than a plain 20-layer net, and on ImageNet plain-34 had a higher validation top-1 error than plain-18 (28.54% vs 27.94%, 10-crop).

Higher *training* error means this is not overfitting. It is an optimization failure: SGD cannot find the "extra layers = identity" solution, because the identity is hard to represent with stacked ReLU layers, which start from small random weights. He et al. checked that the gradients of their BatchNorm plain nets did not vanish, so this was not simply vanishing gradients either.

The fix is to change what the layers are asked to learn. If the target mapping is $H(x)$, let the layers fit the **residual** $F(x)=H(x)-x$ and add $x$ back. The identity then costs nothing: drive $F$ to zero. A layer that is not needed only has to learn to stay quiet.

## The residual block

$$
y=F(x,\{W_i\})+x,\qquad F(x)=W_2\,\mathrm{ReLU}\big(\mathrm{BN}(W_1x)\big)\ \ \text{(then BN, add, ReLU)}
$$

- $x$: block input (a feature map in a CNN)
- $F$: residual branch: two $3\times3$ convolutions with BatchNorm in the basic block
- $+x$: identity shortcut: no parameters, no extra FLOPs beyond the elementwise add

The addition needs $F(x)$ and $x$ to have the same shape. When a stage halves the spatial size and doubles the channels, the shortcut becomes a **projection** $y=F(x)+W_sx$, where $W_s$ is a $1\times1$ convolution with stride 2 (followed by BN). The paper compared options: zero-padding shortcuts (A), projections only when the shape changes (B), and projections everywhere (C). B and C were only marginally better than A, so the parameter-free identity is used wherever shapes match.

**The two block types**

1. **Basic block** (ResNet-18/34): $3\times3$ conv → BN → ReLU → $3\times3$ conv → BN, add the shortcut, ReLU. Two layers per block.
2. **Bottleneck block** (ResNet-50/101/152): $1\times1$ conv reduces channels 4× (256 → 64), $3\times3$ conv works at 64 channels, $1\times1$ conv expands back to 256, then the add. Three layers per block.
3. Bottleneck saving: at 256 channels it has $256\cdot64+9\cdot64\cdot64+64\cdot256=69{,}632$ conv weights, versus $2\cdot9\cdot256\cdot256=1{,}179{,}648$ for two full $3\times3$ convs, about 17× fewer.
4. Network layout: a $7\times7$ stride-2 conv and max-pool, four stages at 64/128/256/512 base channels (each later stage starts with a stride-2 block and a projection shortcut), global average pooling, one fully connected layer.

Standard ImageNet ResNets. Depth counts conv + fc layers. FLOPs (multiply-adds) from He et al. 2015, Table 1; 10-crop validation errors from Tables 2–3 (ResNet-34 with projection option B); parameter counts from the torchvision layout.

| Model | Block | Blocks per stage | Params | FLOPs | Top-1 / top-5 err. |
| --- | --- | --- | --- | --- | --- |
| ResNet-18 | basic | 2, 2, 2, 2 | 11.7M | 1.8G | 27.88% (top-1) |
| ResNet-34 | basic | 3, 4, 6, 3 | 21.8M | 3.6G | 24.52% / 7.46% |
| ResNet-50 | bottleneck | 3, 4, 6, 3 | 25.6M | 3.8G | 22.85% / 6.71% |
| ResNet-101 | bottleneck | 3, 4, 23, 3 | 44.5M | 7.6G | 21.75% / 6.05% |
| ResNet-152 | bottleneck | 3, 8, 36, 3 | 60.2M | 11.3G | 21.43% / 5.71% |

ResNet-50 has 50 layers: $1 + 3\cdot(3+4+6+3) + 1$. Swapping basic for bottleneck blocks adds 16 layers to ResNet-34 for only 0.2 GFLOPs more. An ensemble of ResNets reached **3.57% top-5 error** on the ImageNet test set and won the ILSVRC 2015 classification task; the same backbones also won the ImageNet detection and localization and COCO detection and segmentation tracks.

## Why gradients flow: the identity term

In the follow-up paper (He et al. 2016, "Identity Mappings in Deep Residual Networks") the ReLU after the addition moves into the branch (**pre-activation**: BN → ReLU → conv → BN → ReLU → conv). Then $x_{l+1}=x_l+F(x_l)$ exactly, and unrolling from any block $l$ to a deeper block $L$ gives a sum:

$$
x_L=x_l+\sum_{i=l}^{L-1}F(x_i,W_i)\quad\Longrightarrow\quad \frac{\partial\mathcal{L}}{\partial x_l}=\frac{\partial\mathcal{L}}{\partial x_L}\Big(1+\frac{\partial}{\partial x_l}\sum_{i=l}^{L-1}F(x_i,W_i)\Big)
$$

- $x_l$: input to block $l$
- $\partial\mathcal{L}/\partial x_L$: gradient at a deep block, passed to block l unchanged by the 1 term
- $\frac{\partial}{\partial x_l}\sum F$: contribution that goes through the residual branches

In a plain network the same gradient is a **product** of $L-l$ layer Jacobians, and a product of many factors drifts to 0 or ∞. Here the "1" is a direct path: the deep gradient reaches block $l$ without being multiplied by any weight. The branch term would have to equal exactly $-1$ for every sample in the batch to cancel it, which is very unlikely. Pre-activation let a 1001-layer ResNet reach 4.62% error on CIFAR-10.

> Figure: One real forward and backward pass through a 32-wide net, wired plain (rose) and with an identity shortcut every two layers (accent). Both use the same seeded weights. At the defaults (ReLU, He scale, 56 layers, branch scale β = 1/√28), the plain net's gradient shrinks 25× from output to input (ratio 0.040) and its signal falls to RMS 0.081. The residual net keeps both near 1 (1.81 and 1.90). Set s = 2 with tanh: the plain net's input gradient turns into noise (lag-1 correlation 0.04) while the residual one stays smooth (0.99). Set β = 1 to see why real ResNets need BatchNorm or a small branch scale.

The bottom strips show $dy/dx$ across 128 inputs. In deep plain nets the input gradient becomes **shattered**, close to white noise (Balduzzi et al. 2017), so a gradient step taken at one input says little about its neighbours. Residual nets keep it smooth. With $\beta=1$ and no normalization, every block adds variance: at 56 layers the residual net's output RMS reaches about 1500. With $\beta=1/\sqrt{B}$ for $B$ blocks the growth stays bounded, since $(1+1/B)^{B/2}\le\sqrt e\approx1.65$ when the branch is roughly uncorrelated with its input. Real ResNets control this with BatchNorm, and many recipes go further by initialising the last BN scale in each branch to zero, so every block starts as the identity (Goyal et al. 2017; Fixup and SkipInit do the same without BN).

## Worked example: paths through three blocks

Take a scalar linear residual chain $x_{l+1}=x_l+w_lx_l$ with $w=(0.5,\,-0.3,\,0.2)$. Then $\partial x_3/\partial x_0=\prod_l(1+w_l)=1.5\cdot0.7\cdot1.2=1.26$. Expand the product: every term picks, at each block, either the shortcut (factor 1) or the branch (factor $w_l$). That gives $2^3=8$ paths, grouped here by how many branches they pass through:

Expanding (1 + w₁)(1 + w₂)(1 + w₃) into paths (exact arithmetic)

| Branches used | Paths | Terms | Sum |
| --- | --- | --- | --- |
| 0 | 1 | 1 | 1 |
| 1 | 3 | 0.5 − 0.3 + 0.2 | 0.4 |
| 2 | 3 | −0.15 + 0.10 − 0.06 | −0.11 |
| 3 | 1 | 0.5 · (−0.3) · 0.2 | −0.03 |
| total | 8 |  | 1.26 |

A plain chain $x_{l+1}=w_lx_l$ keeps only the last row: gradient $-0.03$, 42× smaller and with the opposite sign. With 50 blocks of $w=0.1$ the plain product is $10^{-50}$, while the residual one is $1.1^{50}=117.4$. With $w=-0.1$ it is $0.9^{50}=0.0052$. The shortcut removes the guaranteed collapse, but the branches still compound, which is why branch scale and normalization matter.

> **ResNets behave like ensembles of shallow paths:** Veit, Wilber and Belongie (2016) read the same expansion for real networks: $n$ blocks give $2^n$ paths, with path lengths distributed Binomial$(n,\tfrac12)$. Deleting a single block from a trained ResNet barely changes its accuracy, while deleting a layer from VGG breaks it. In a 54-block ResNet-110, paths of length 5–17 carry almost all of the gradient yet are only 0.45% of the $2^{54}$ paths. So the network trains mostly through fairly shallow paths.

```python
import torch.nn as nn, torch.nn.functional as F

class BasicBlock(nn.Module):
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False); self.bn1 = nn.BatchNorm2d(cout)
        self.conv2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False);     self.bn2 = nn.BatchNorm2d(cout)
        self.short = nn.Identity()
        if stride != 1 or cin != cout:                 # shape changes: 1x1 projection
            self.short = nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
        nn.init.zeros_(self.bn2.weight)                # block starts as identity (zero-init γ)

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + self.short(x))             # y = F(x) + x

class Bottleneck(nn.Module):                           # 1x1 reduce, 3x3, 1x1 expand (x4)
    def __init__(self, cin, width, stride=1):
        super().__init__()
        cout = 4 * width
        self.f = nn.Sequential(
            nn.Conv2d(cin, width, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
            nn.Conv2d(width, width, 3, stride, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
            nn.Conv2d(width, cout, 1, bias=False), nn.BatchNorm2d(cout))
        self.short = nn.Identity() if stride == 1 and cin == cout else nn.Sequential(
            nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))

    def forward(self, x):
        return F.relu(self.f(x) + self.short(x))

print(sum(p.numel() for p in BasicBlock(64, 128, stride=2).parameters()))   # 230144
# torchvision.models.resnet50() has 25,557,032 parameters
```

Basic block with a projection shortcut, and the bottleneck block (torchvision layout)

> **Do not put things on the shortcut:** The identity-mappings paper tried scaling the shortcut by 0.5, gating it, adding $1\times1$ convs or dropout on it. On ResNet-110/CIFAR-10 every variant trained to a higher error than the plain identity (6.61%). Anything that multiplies the shortcut brings back the product of factors the identity was there to remove. Put scaling and normalization inside $F$ instead.

> **Residuals everywhere:** The idea outlived CNNs. Every transformer sublayer is residual, and most modern ones use the pre-LN form $x\leftarrow x+\mathrm{Attn}(\mathrm{LN}(x))$, then $x\leftarrow x+\mathrm{MLP}(\mathrm{LN}(x))$. This form is the pre-activation ResNet idea: normalization sits in the branch and the residual stream stays a clean sum. That clean sum is what lets 100-layer language models train.

**Strengths**

- Very deep nets train with plain SGD; depth gives accuracy again instead of degradation.
- Identity shortcuts add no parameters and almost no compute.
- Robust: blocks can be dropped at test time (stochastic depth uses this during training).
- A simple, regular design that transfers well as a backbone for detection, segmentation and more.

**Limitations**

- The add forces matching shapes, so stage transitions need projections.
- Without normalization or a small branch scale, the forward signal grows with depth.
- Many blocks only make small refinements: wide ResNets (Zagoruyko & Komodakis 2016) match much deeper thin ones with far fewer layers.
- The skip tensors must be kept in memory during the forward pass.

**Question:** A plain 56-layer CNN has higher training error than a plain 20-layer one. What does this show?

- A. The 56-layer net overfits
- B. Optimization fails: SGD cannot find a solution the deeper net can represent
- C. The deeper net has fewer parameters
- D. The learning rate must be larger for deep nets

**Answer:** B. Optimization fails: SGD cannot find a solution the deeper net can represent — Overfitting would lower training error and raise test error. Here *training* error is worse, although the deeper net could copy the shallow one by setting its extra layers to the identity. ResNets make that solution easy to find: just set $F=0$.

**Question:** With pre-activation blocks, $\partial\mathcal{L}/\partial x_l=\partial\mathcal{L}/\partial x_L\,(1+\partial_{x_l}\sum F)$. Why does this stop the gradient vanishing?

- A. The branch gradients are always positive
- B. The 1 term passes the deep gradient straight to every earlier block, without a product of weight Jacobians
- C. BatchNorm rescales the gradient to unit norm
- D. The shortcut has its own learned weights

**Answer:** B. The 1 term passes the deep gradient straight to every earlier block, without a product of weight Jacobians — The gradient is a sum, not only a product. The identity path contributes $\partial\mathcal{L}/\partial x_L$ unchanged. The branch term would have to be exactly $-1$ for every sample to cancel it.

**Key points**

- Degradation: deeper plain nets had *higher training error* (56 vs 20 layers on CIFAR-10). That is an optimization problem, not overfitting.
- Residual block $y=x+F(x)$: identity shortcuts where shapes match, $1\times1$ projections where they change. Basic blocks use two $3\times3$ convs; bottlenecks use $1\times1$–$3\times3$–$1\times1$.
- Pre-activation gives $x_L=x_l+\sum F$, so $\partial\mathcal{L}/\partial x_l$ contains $\partial\mathcal{L}/\partial x_L$ directly.
- $n$ blocks give $2^n$ paths of Binomial length: a ResNet acts like an ensemble of mostly shallow paths.
- ResNet-18/34/50/101/152: 11.7M–60.2M params. The ResNet ensemble won ILSVRC 2015 with 3.57% top-5, and the residual stream is the backbone of transformers.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
