Vanishing gradients

Backprop multiplies one local derivative per layer; if each is below 1 — like sigmoid's maximum 0.25 — the gradient shrinks exponentially with depth.

7 min read629 wordscore1 viewsupdated 2026-10-09

Definition

Vanishing gradients is the failure mode where the gradient reaching the early layers of a deep network (or the early time steps of a recurrent one) is exponentially smaller than the gradient at the output, so those layers barely change during training.

Why depth shrinks the signal

Backpropagation is the chain rule applied layer by layer. To get the gradient at layer ll, you start from the loss and multiply by one local Jacobian for every layer between ll and the output.

A product of LL factors behaves like γL\gamma^L. If the typical factor γ\gamma is below 1, the product decays exponentially in depth, not linearly. Twenty layers at γ=0.9\gamma = 0.9 already give 0.920≈0.120.9^{20} \approx 0.12; at γ=0.25\gamma = 0.25 they give 9.1×10−139.1\times10^{-13}.

The product of Jacobians

Take a plain fully connected net: z(k)=W(k)h(k−1)+b(k)z^{(k)} = W^{(k)}h^{(k-1)} + b^{(k)} and h(k)=φ(z(k))h^{(k)} = \varphi(z^{(k)}), for k=1,…,Lk = 1,\dots,L. One step of the chain rule is ∂L/∂h(k−1)=W(k)⊤D(k) ∂L/∂h(k)\partial \mathcal{L}/\partial h^{(k-1)} = W^{(k)\top} D^{(k)}\,\partial \mathcal{L}/\partial h^{(k)}. Unrolling it:

∂L∂h(l)=W(l+1)⊤D(l+1) W(l+2)⊤D(l+2)⋯W(L)⊤D(L)  ∂L∂h(L),D(k)=diag⁡ ⁣(φ′(z(k)))\frac{\partial \mathcal{L}}{\partial h^{(l)}} = W^{(l+1)\top}D^{(l+1)}\,W^{(l+2)\top}D^{(l+2)}\cdots W^{(L)\top}D^{(L)}\;\frac{\partial \mathcal{L}}{\partial h^{(L)}},\qquad D^{(k)}=\operatorname{diag}\!\big(\varphi'(z^{(k)})\big)
h(k)h^{(k)}
activations of layer kk (h(0)h^{(0)} is the input)
z(k)z^{(k)}
pre-activations of layer kk
W(k)W^{(k)}
weight matrix of layer kk
φ′\varphi'
derivative of the activation function, applied elementwise
D(k)D^{(k)}
diagonal matrix of those slopes at layer kk
LL
number of layers
L\mathcal{L}
the loss

The weight gradient is that signal times the layer's input: ∂L/∂W(l)=δ(l)h(l−1)⊤\partial \mathcal{L}/\partial W^{(l)} = \delta^{(l)}h^{(l-1)\top} with δ(l)=D(l) ∂L/∂h(l)\delta^{(l)} = D^{(l)}\,\partial \mathcal{L}/\partial h^{(l)}. If δ(l)\delta^{(l)} is tiny, so is the update to W(l)W^{(l)}.

Taking norms gives an upper bound that makes the exponential explicit:

∥∂L∂h(l)∥≤∏k=l+1L∥W(k)∥ max⁡∣φ′∣  ⋅  ∥∂L∂h(L)∥\Big\|\frac{\partial \mathcal{L}}{\partial h^{(l)}}\Big\| \le \prod_{k=l+1}^{L}\|W^{(k)}\|\,\max|\varphi'|\;\cdot\;\Big\|\frac{\partial \mathcal{L}}{\partial h^{(L)}}\Big\|
∥W(k)∥\|W^{(k)}\|
spectral norm (largest singular value) of the weight matrix
max⁡∣φ′∣\max|\varphi'|
largest slope of the activation: 0.25 for sigmoid, 1 for tanh and ReLU

Sigmoid: at most 0.25 per layer

The sigmoid's derivative is σ′(x)=σ(x)(1−σ(x))\sigma'(x) = \sigma(x)(1-\sigma(x)). It peaks at x=0x = 0 with value 0.5⋅0.5=0.250.5\cdot0.5 = 0.25. So with ∥W(k)∥≤1\|W^{(k)}\| \le 1, each sigmoid layer shrinks the gradient bound by at least 4×4\times.

Saturation makes it worse. For ∣x∣|x| large the sigmoid is flat: σ′(2)=0.105\sigma'(2) = 0.105, σ′(5)=0.0066\sigma'(5) = 0.0066. Tanh has slope 1 at 0 but saturates the same way (tanh⁡′(3)=0.0099\tanh'(3) = 0.0099). A saturated unit passes almost no gradient back, however large its weights.

layers ksigmoid at z = 0: 0.25ᵏsigmoid at z = 2: 0.105ᵏReLU, active unit: 1ᵏ
10.250.1051
59.8×10−49.8\times10^{-4}1.3×10−51.3\times10^{-5}1
109.5×10−79.5\times10^{-7}1.6×10−101.6\times10^{-10}1
209.1×10−139.1\times10^{-13}2.6×10−202.6\times10^{-20}1
A one-unit-per-layer chain with weight 1: the gradient factor after k layers (computed in Node).
‖δ⁽ˡ⁾‖ = ‖∂L/∂z⁽ˡ⁾‖ per layer (log scale) · 32 units/layer, batch 16, sigmoid← backprop flows from the loss toward layer 11e-161e-121e-81e-411e41e8last layer 0.23912468101214161820φ′≈0layer 1 (input side) → layer 20 (next to the loss)strip: share of units with φ′(z) < 5% of max φ′■ within 1000× of the last layer ■ < 1/1000 (vanished) ■ > 1000× (exploded)
activation
W ~ N(0, s²/32)
‖δ⁽¹⁾‖ / ‖δ⁽ᴸ⁾‖ = 1.1e-12per layer ≈ ×0.235mean φ′(z) 0.234φ′≈0 units 0%
fig.A seeded simulation, not a formula: one forward and backward pass through a real net of 32-unit layers with weights W ~ N(0, s²/32), on a fixed batch of 16 inputs. Bars are ‖δ⁽ˡ⁾‖ per layer on a log scale. Start with sigmoid at depth 20 and watch the bars fall by about 4× per layer toward the input. Switch to ReLU: at s = 1 it still decays (×0.70 per layer), at s = √2 (He) it is flat, and above that it explodes. Push tanh's scale up and watch the violet saturation strip fill in.
activationscale s‖δ⁽¹⁾‖ / ‖δ⁽²⁰⁾‖per-layer factor
sigmoid11.1×10−121.1\times10^{-12}0.235
sigmoid41.0×10−41.0\times10^{-4}0.617
tanh10.200.919
ReLU11.1×10−31.1\times10^{-3}0.700 ≈1/2\approx 1/\sqrt2
ReLU2\sqrt2 (He)0.830.990
ReLU26.0×1026.0\times10^{2}1.40 (exploding)
Read off the figure at depth 20 (one seeded draw, so the second digit depends on the seed).

The sigmoid row sits right at the 0.25 bound: its average slope in this net is 0.234. ReLU passes slope 1 through the active half of its units and 0 through the rest, so with Var⁡(W)=1/n\operatorname{Var}(W) = 1/n each layer keeps about 1/21/\sqrt2 of the signal. He initialization's factor of 2\sqrt2 in the standard deviation exactly cancels that (see Weight initialization).

The symptom: early layers don't learn

The loss still falls at first, because the last layers train fine. The first layers stay close to their random initialization, so the network behaves like a shallow model sitting on random features. Training stalls early and adding depth makes results worse, not better.

Diagnose it directly: after loss.backward(), log the gradient norm of every layer. A gradient norm that falls steadily from output to input by orders of magnitude is the signature.

python
1import torch
2import torch.nn as nn
3
4layers = []
5for _ in range(20):
6 layers += [nn.Linear(32, 32), nn.Sigmoid()]
7net = nn.Sequential(*layers, nn.Linear(32, 1))
8
9x, y = torch.randn(16, 32), torch.randn(16, 1)
10loss = nn.functional.mse_loss(net(x), y)
11loss.backward()
12
13for name, p in net.named_parameters():
14 if name.endswith("weight"):
15 print(f"{name:10s} {p.grad.norm():.1e}") # shrinks toward layer 0
Per-layer gradient norms of a 20-layer sigmoid MLP. Swap nn.Sigmoid for nn.ReLU and compare.

Recurrent networks: depth in time

An RNN applies the same recurrence ht=φ(Whhht−1+Wxhxt)h_t = \varphi(W_{hh}h_{t-1} + W_{xh}x_t) at every step. Backpropagation through time multiplies one Jacobian per step, and it is the same matrix each time:

∂hT∂ht=∏k=t+1TDk Whh\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} D_k\,W_{hh}
DkD_k
diag⁡(φ′(⋅))\operatorname{diag}(\varphi'(\cdot)) at time step kk
WhhW_{hh}
recurrent weight matrix, shared across time

If the largest singular value of WhhW_{hh} times max⁡∣φ′∣\max|\varphi'| is below 1, this shrinks exponentially in T−tT - t (Pascanu et al., 2013). The error at step TT then carries no information back to inputs 50 or 100 steps earlier, so plain RNNs cannot learn long-range dependencies.

The fixes

fixwhy it helps
ReLU-family activationsslope exactly 1 for active units, so D(k)D^{(k)} does not shrink the signal; no saturation for z>0z > 0
He / Xavier initializationsets Var⁡(W)\operatorname{Var}(W) so each layer's factor is about 1 at the start of training
Residual connectionsh(l+1)=h(l)+F(h(l))h^{(l+1)} = h^{(l)} + F(h^{(l)}) makes each Jacobian I+∂F/∂hI + \partial F/\partial h, an identity path the gradient can always use
Normalization (BatchNorm, LayerNorm)keeps pre-activations near zero mean and unit scale, out of the saturated regions
LSTM / GRU gatesthe cell state is updated additively, ct=ft⊙ct−1+…c_t = f_t\odot c_{t-1} + \dots, so its gradient is scaled by the forget gate ftf_t, which can stay near 1

Check yourself

A 10-layer chain uses sigmoid units, each operating exactly at z = 0 with weight 1. By what factor is the gradient at the first layer scaled relative to the last?

Check yourself

Why do residual connections help gradients reach early layers?

Remember

  • ✓Backprop multiplies one Jacobian W(k)⊤D(k)W^{(k)\top}D^{(k)} per layer, so gradient size scales like γdepth\gamma^{\text{depth}}.
  • ✓Sigmoid's slope is at most 0.25: 0.2510≈9.5×10−70.25^{10} \approx 9.5\times10^{-7}. Saturation pushes it toward 0.
  • ✓Symptom: the last layers train, the first layers stay random. Log per-layer gradient norms to see it.
  • ✓RNNs repeat the same WhhW_{hh} over time, so long-range gradients vanish when its largest singular value times max⁡∣φ′∣\max|\varphi'| is below 1.
  • ✓Fixes: ReLU, He/Xavier init, residual connections, normalization, LSTM/GRU gates.
← back to Deep Learning roadmap