# Vanishing gradients

> Backprop multiplies one local derivative per layer; if each is below 1 — like sigmoid's maximum 0.25 — the gradient shrinks exponentially with depth.

Source: https://www.mlfoundry.in/learn/vanishing-gradients  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Vanishing gradients:** is the failure mode where the gradient reaching the early layers of a deep network (or the early time steps of a recurrent one) is exponentially smaller than the gradient at the output, so those layers barely change during training.

## Why depth shrinks the signal

Backpropagation is the chain rule applied layer by layer. To get the gradient at layer $l$, you start from the loss and multiply by one local Jacobian for every layer between $l$ and the output.

A product of $L$ factors behaves like $\gamma^L$. If the typical factor $\gamma$ is below 1, the product decays **exponentially** in depth, not linearly. Twenty layers at $\gamma = 0.9$ already give $0.9^{20} \approx 0.12$; at $\gamma = 0.25$ they give $9.1\times10^{-13}$.

## The product of Jacobians

Take a plain fully connected net: $z^{(k)} = W^{(k)}h^{(k-1)} + b^{(k)}$ and $h^{(k)} = \varphi(z^{(k)})$, for $k = 1,\dots,L$. One step of the chain rule is $\partial \mathcal{L}/\partial h^{(k-1)} = W^{(k)\top} D^{(k)}\,\partial \mathcal{L}/\partial h^{(k)}$. Unrolling it:

$$
\frac{\partial \mathcal{L}}{\partial h^{(l)}} = W^{(l+1)\top}D^{(l+1)}\,W^{(l+2)\top}D^{(l+2)}\cdots W^{(L)\top}D^{(L)}\;\frac{\partial \mathcal{L}}{\partial h^{(L)}},\qquad D^{(k)}=\operatorname{diag}\!\big(\varphi'(z^{(k)})\big)
$$

- $h^{(k)}$: activations of layer $k$ ($h^{(0)}$ is the input)
- $z^{(k)}$: pre-activations of layer $k$
- $W^{(k)}$: weight matrix of layer $k$
- $\varphi'$: derivative of the activation function, applied elementwise
- $D^{(k)}$: diagonal matrix of those slopes at layer $k$
- $L$: number of layers
- $\mathcal{L}$: the loss

The weight gradient is that signal times the layer's input: $\partial \mathcal{L}/\partial W^{(l)} = \delta^{(l)}h^{(l-1)\top}$ with $\delta^{(l)} = D^{(l)}\,\partial \mathcal{L}/\partial h^{(l)}$. If $\delta^{(l)}$ is tiny, so is the update to $W^{(l)}$.

Taking norms gives an upper bound that makes the exponential explicit:

$$
\Big\|\frac{\partial \mathcal{L}}{\partial h^{(l)}}\Big\| \le \prod_{k=l+1}^{L}\|W^{(k)}\|\,\max|\varphi'|\;\cdot\;\Big\|\frac{\partial \mathcal{L}}{\partial h^{(L)}}\Big\|
$$

- $\|W^{(k)}\|$: spectral norm (largest singular value) of the weight matrix
- $\max|\varphi'|$: largest slope of the activation: 0.25 for sigmoid, 1 for tanh and ReLU

## Sigmoid: at most 0.25 per layer

The sigmoid's derivative is $\sigma'(x) = \sigma(x)(1-\sigma(x))$. It peaks at $x = 0$ with value $0.5\cdot0.5 = 0.25$. So with $\|W^{(k)}\| \le 1$, each sigmoid layer shrinks the gradient bound by at least $4\times$.

**Saturation** makes it worse. For $|x|$ large the sigmoid is flat: $\sigma'(2) = 0.105$, $\sigma'(5) = 0.0066$. Tanh has slope 1 at 0 but saturates the same way ($\tanh'(3) = 0.0099$). A saturated unit passes almost no gradient back, however large its weights.

A one-unit-per-layer chain with weight 1: the gradient factor after k layers (computed in Node).

| layers k | sigmoid at z = 0: 0.25ᵏ | sigmoid at z = 2: 0.105ᵏ | ReLU, active unit: 1ᵏ |
| --- | --- | --- | --- |
| 1 | 0.25 | 0.105 | 1 |
| 5 | $9.8\times10^{-4}$ | $1.3\times10^{-5}$ | 1 |
| 10 | $9.5\times10^{-7}$ | $1.6\times10^{-10}$ | 1 |
| 20 | $9.1\times10^{-13}$ | $2.6\times10^{-20}$ | 1 |

> Figure: A seeded simulation, not a formula: one forward and backward pass through a real net of 32-unit layers with weights W ~ N(0, s²/32), on a fixed batch of 16 inputs. Bars are ‖δ⁽ˡ⁾‖ per layer on a log scale. Start with sigmoid at depth 20 and watch the bars fall by about 4× per layer toward the input. Switch to ReLU: at s = 1 it still decays (×0.70 per layer), at s = √2 (He) it is flat, and above that it explodes. Push tanh's scale up and watch the violet saturation strip fill in.

Read off the figure at depth 20 (one seeded draw, so the second digit depends on the seed).

| activation | scale s | ‖δ⁽¹⁾‖ / ‖δ⁽²⁰⁾‖ | per-layer factor |
| --- | --- | --- | --- |
| sigmoid | 1 | $1.1\times10^{-12}$ | 0.235 |
| sigmoid | 4 | $1.0\times10^{-4}$ | 0.617 |
| tanh | 1 | 0.20 | 0.919 |
| ReLU | 1 | $1.1\times10^{-3}$ | 0.700 $\approx 1/\sqrt2$ |
| ReLU | $\sqrt2$ (He) | 0.83 | 0.990 |
| ReLU | 2 | $6.0\times10^{2}$ | 1.40 (exploding) |

The sigmoid row sits right at the 0.25 bound: its average slope in this net is 0.234. ReLU passes slope 1 through the active half of its units and 0 through the rest, so with $\operatorname{Var}(W) = 1/n$ each layer keeps about $1/\sqrt2$ of the signal. He initialization's factor of $\sqrt2$ in the standard deviation exactly cancels that (see **Weight initialization**).

## The symptom: early layers don't learn

The loss still falls at first, because the last layers train fine. The first layers stay close to their random initialization, so the network behaves like a shallow model sitting on random features. Training stalls early and adding depth makes results worse, not better.

Diagnose it directly: after `loss.backward()`, log the gradient norm of every layer. A gradient norm that falls steadily from output to input by orders of magnitude is the signature.

```python
import torch
import torch.nn as nn

layers = []
for _ in range(20):
    layers += [nn.Linear(32, 32), nn.Sigmoid()]
net = nn.Sequential(*layers, nn.Linear(32, 1))

x, y = torch.randn(16, 32), torch.randn(16, 1)
loss = nn.functional.mse_loss(net(x), y)
loss.backward()

for name, p in net.named_parameters():
    if name.endswith("weight"):
        print(f"{name:10s} {p.grad.norm():.1e}")   # shrinks toward layer 0
```

Per-layer gradient norms of a 20-layer sigmoid MLP. Swap nn.Sigmoid for nn.ReLU and compare.

## Recurrent networks: depth in time

An RNN applies the same recurrence $h_t = \varphi(W_{hh}h_{t-1} + W_{xh}x_t)$ at every step. Backpropagation through time multiplies one Jacobian per step, and it is the **same** matrix each time:

$$
\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} D_k\,W_{hh}
$$

- $D_k$: $\operatorname{diag}(\varphi'(\cdot))$ at time step $k$
- $W_{hh}$: recurrent weight matrix, shared across time

If the largest singular value of $W_{hh}$ times $\max|\varphi'|$ is below 1, this shrinks exponentially in $T - t$ (Pascanu et al., 2013). The error at step $T$ then carries no information back to inputs 50 or 100 steps earlier, so plain RNNs cannot learn long-range dependencies.

## The fixes

| fix | why it helps |
| --- | --- |
| **ReLU-family activations** | slope exactly 1 for active units, so $D^{(k)}$ does not shrink the signal; no saturation for $z > 0$ |
| **He / Xavier initialization** | sets $\operatorname{Var}(W)$ so each layer's factor is about 1 at the start of training |
| **Residual connections** | $h^{(l+1)} = h^{(l)} + F(h^{(l)})$ makes each Jacobian $I + \partial F/\partial h$, an identity path the gradient can always use |
| **Normalization (BatchNorm, LayerNorm)** | keeps pre-activations near zero mean and unit scale, out of the saturated regions |
| **LSTM / GRU gates** | the cell state is updated additively, $c_t = f_t\odot c_{t-1} + \dots$, so its gradient is scaled by the forget gate $f_t$, which can stay near 1 |

> **It is not only a sigmoid problem:** ReLU removes saturation for active units, but the weights still contribute a factor per layer. In the figure, ReLU at $s = 1$ loses $10^{3}\times$ over 20 layers. The activation and the initialization have to be chosen together.

> **Vanishing and exploding are the same mechanism:** Both come from multiplying many per-layer factors. Below 1 the product vanishes, above 1 it explodes (see **Exploding gradients**). The goal of every fix above is to keep the per-layer factor close to 1.

**Question:** A 10-layer chain uses sigmoid units, each operating exactly at z = 0 with weight 1. By what factor is the gradient at the first layer scaled relative to the last?

- A. 0.25
- B. 2.5
- C. about 1e−3
- D. about 9.5e−7

**Answer:** D. about 9.5e−7 — Each layer contributes $\sigma'(0)\cdot 1 = 0.25$, and the factors multiply: $0.25^{10} = 9.54\times10^{-7}$. Adding them up instead of multiplying would give 2.5, which is not how the chain rule works.

**Question:** Why do residual connections help gradients reach early layers?

- A. They make every activation linear
- B. Each block's Jacobian becomes I + ∂F/∂h, so there is an identity path that is not multiplied by small factors
- C. They reduce the number of parameters
- D. They clip the gradient norm

**Answer:** B. Each block's Jacobian becomes I + ∂F/∂h, so there is an identity path that is not multiplied by small factors — With $h^{(l+1)} = h^{(l)} + F(h^{(l)})$, the local Jacobian is $I + \partial F/\partial h$. Even if $\partial F/\partial h$ is small, the identity term carries the gradient through unchanged.

**Key points**

- Backprop multiplies one Jacobian $W^{(k)\top}D^{(k)}$ per layer, so gradient size scales like $\gamma^{\text{depth}}$.
- Sigmoid's slope is at most 0.25: $0.25^{10} \approx 9.5\times10^{-7}$. Saturation pushes it toward 0.
- Symptom: the last layers train, the first layers stay random. Log per-layer gradient norms to see it.
- RNNs repeat the same $W_{hh}$ over time, so long-range gradients vanish when its largest singular value times $\max|\varphi'|$ is below 1.
- Fixes: ReLU, He/Xavier init, residual connections, normalization, LSTM/GRU gates.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
