Definition
Vanishing gradients is the failure mode where the gradient reaching the early layers of a deep network (or the early time steps of a recurrent one) is exponentially smaller than the gradient at the output, so those layers barely change during training.
Why depth shrinks the signal
Backpropagation is the chain rule applied layer by layer. To get the gradient at layer , you start from the loss and multiply by one local Jacobian for every layer between and the output.
A product of factors behaves like . If the typical factor is below 1, the product decays exponentially in depth, not linearly. Twenty layers at already give ; at they give .
The product of Jacobians
Take a plain fully connected net: and , for . One step of the chain rule is . Unrolling it:
- activations of layer ( is the input)
- pre-activations of layer
- weight matrix of layer
- derivative of the activation function, applied elementwise
- diagonal matrix of those slopes at layer
- number of layers
- the loss
The weight gradient is that signal times the layer's input: with . If is tiny, so is the update to .
Taking norms gives an upper bound that makes the exponential explicit:
- spectral norm (largest singular value) of the weight matrix
- largest slope of the activation: 0.25 for sigmoid, 1 for tanh and ReLU
Sigmoid: at most 0.25 per layer
The sigmoid's derivative is . It peaks at with value . So with , each sigmoid layer shrinks the gradient bound by at least .
Saturation makes it worse. For large the sigmoid is flat: , . Tanh has slope 1 at 0 but saturates the same way (). A saturated unit passes almost no gradient back, however large its weights.
| layers k | sigmoid at z = 0: 0.25ᵏ | sigmoid at z = 2: 0.105ᵏ | ReLU, active unit: 1ᵏ |
|---|---|---|---|
| 1 | 0.25 | 0.105 | 1 |
| 5 | 1 | ||
| 10 | 1 | ||
| 20 | 1 |
| activation | scale s | ‖δ⁽¹⁾‖ / ‖δ⁽²⁰⁾‖ | per-layer factor |
|---|---|---|---|
| sigmoid | 1 | 0.235 | |
| sigmoid | 4 | 0.617 | |
| tanh | 1 | 0.20 | 0.919 |
| ReLU | 1 | 0.700 | |
| ReLU | (He) | 0.83 | 0.990 |
| ReLU | 2 | 1.40 (exploding) |
The sigmoid row sits right at the 0.25 bound: its average slope in this net is 0.234. ReLU passes slope 1 through the active half of its units and 0 through the rest, so with each layer keeps about of the signal. He initialization's factor of in the standard deviation exactly cancels that (see Weight initialization).
The symptom: early layers don't learn
The loss still falls at first, because the last layers train fine. The first layers stay close to their random initialization, so the network behaves like a shallow model sitting on random features. Training stalls early and adding depth makes results worse, not better.
Diagnose it directly: after loss.backward(), log the gradient norm of every layer. A gradient norm that falls steadily from output to input by orders of magnitude is the signature.
1import torch2import torch.nn as nn34layers = []5for _ in range(20):6 layers += [nn.Linear(32, 32), nn.Sigmoid()]7net = nn.Sequential(*layers, nn.Linear(32, 1))89x, y = torch.randn(16, 32), torch.randn(16, 1)10loss = nn.functional.mse_loss(net(x), y)11loss.backward()1213for name, p in net.named_parameters():14 if name.endswith("weight"):15 print(f"{name:10s} {p.grad.norm():.1e}") # shrinks toward layer 0Recurrent networks: depth in time
An RNN applies the same recurrence at every step. Backpropagation through time multiplies one Jacobian per step, and it is the same matrix each time:
- at time step
- recurrent weight matrix, shared across time
If the largest singular value of times is below 1, this shrinks exponentially in (Pascanu et al., 2013). The error at step then carries no information back to inputs 50 or 100 steps earlier, so plain RNNs cannot learn long-range dependencies.
The fixes
| fix | why it helps |
|---|---|
| ReLU-family activations | slope exactly 1 for active units, so does not shrink the signal; no saturation for |
| He / Xavier initialization | sets so each layer's factor is about 1 at the start of training |
| Residual connections | makes each Jacobian , an identity path the gradient can always use |
| Normalization (BatchNorm, LayerNorm) | keeps pre-activations near zero mean and unit scale, out of the saturated regions |
| LSTM / GRU gates | the cell state is updated additively, , so its gradient is scaled by the forget gate , which can stay near 1 |
Check yourself
A 10-layer chain uses sigmoid units, each operating exactly at z = 0 with weight 1. By what factor is the gradient at the first layer scaled relative to the last?
Check yourself
Why do residual connections help gradients reach early layers?
Remember
- ✓Backprop multiplies one Jacobian per layer, so gradient size scales like .
- ✓Sigmoid's slope is at most 0.25: . Saturation pushes it toward 0.
- ✓Symptom: the last layers train, the first layers stay random. Log per-layer gradient norms to see it.
- ✓RNNs repeat the same over time, so long-range gradients vanish when its largest singular value times is below 1.
- ✓Fixes: ReLU, He/Xavier init, residual connections, normalization, LSTM/GRU gates.