Exploding gradients

If each layer multiplies the gradient by more than 1, it grows exponentially with depth; clip the gradient norm to keep updates bounded.

7 min read585 wordscore0 viewsupdated 2026-10-09

Definition

Exploding gradients is the failure mode where gradient norms grow exponentially as they are backpropagated through many layers or time steps, so a single update can move the parameters so far that training diverges.

The same product, factors above 1

Backpropagation multiplies one local Jacobian per layer (the derivation is in Vanishing gradients). If the typical factor γ\gamma is above 1, the gradient at the first layer is about γL\gamma^{L} times the gradient at the last.

∥∂L∂h(l)∥≈γ L−l ∥∂L∂h(L)∥,γ≈∥W(k)⊤D(k)∥\Big\|\frac{\partial \mathcal{L}}{\partial h^{(l)}}\Big\| \approx \gamma^{\,L-l}\,\Big\|\frac{\partial \mathcal{L}}{\partial h^{(L)}}\Big\|,\qquad \gamma \approx \|W^{(k)\top}D^{(k)}\|
L\mathcal{L}
the loss
h(l)h^{(l)}
activations of layer ll; LL is the number of layers
D(k)D^{(k)}
diag⁡(φ′(z(k)))\operatorname{diag}(\varphi'(z^{(k)})), the activation slopes at layer kk
γ\gamma
typical per-layer gain of the backward pass

Exponentials are unforgiving: a gain of only 1.51.5 per layer over 20 layers gives 1.520≈33251.5^{20} \approx 3325. In the vanishing-gradients figure, ReLU with weight scale 2 (instead of He's 2\sqrt2) grows the gradient by 1.4×1.4\times per layer, about 600×600\times over 20 layers.

Recurrent networks

An RNN multiplies by the same recurrent matrix at every time step, ∂hT/∂ht=∏k=t+1TDkWhh\partial h_T/\partial h_t = \prod_{k=t+1}^{T} D_k W_{hh}. For a linear RNN this is Whh T−tW_{hh}^{\,T-t}, which grows like ρ(Whh)T−t\rho(W_{hh})^{T-t} along its top eigenvector when the spectral radius ρ>1\rho > 1. Over 50 steps, 1.150≈1171.1^{50} \approx 117.

With tanh (∣φ′∣≤1|\varphi'| \le 1), a largest singular value of WhhW_{hh} above 1 is necessary for explosion but not sufficient, because saturated units damp the product (Pascanu et al., 2013). In practice RNNs swing between both regimes during training.

Symptoms

At a glance

Loss spikes
the training loss jumps up by orders of magnitude after a smooth stretch, then slowly recovers or never does
NaN / inf
weights or activations overflow; in fp16 anything above 6550465504 is inf, and inf - inf is nan
Huge gradient norm
the total gradient norm logged per step shows sudden spikes far above its usual range
Huge weights
parameter norms grow quickly, and the model's outputs saturate or blow up

Cliffs in the loss landscape

Pascanu et al. (2013) showed that RNN loss surfaces contain cliffs: narrow regions where the loss changes very steeply. Away from a cliff the gradient is modest; on its face the gradient is enormous.

Gradient descent moves by η ∥g∥\eta\,\|g\|. On the cliff face that step can be dozens of times larger than the distance to the minimum, so the parameters are catapulted far away and much of the progress so far is lost. The direction of gg is reasonable. Its length is the problem.

loss surface (darker = higher) · plain GD · GD + clip ‖g‖ ≤ 1.00cliff0246012θ₁ →start×‖∇L‖ at each step (log scale) · dashed = clip threshold c0.111010001020304050step → (filled dot = this update was clipped)
plain GD: max loss 27.3 · final 1.58 · largest step 6.80clipped: max loss 6.09 · final 0.50 · largest step 0.30 · clipped 29/50
optimum × at θ = (-0.37, 0), loss 0.293 · every step is θ ← θ − η·g·min(1, c/‖g‖)
fig.A hand-built 2-D loss: a valley whose optimum (×) sits at the foot of a steep wall, where the wall slopes at 40 while gradients in the valley are between about 0.5 and 3. Both runs start at the same point with the same learning rate. Plain gradient descent (rose) steps onto the wall and is thrown to the right, again and again; the clipped run (mint) bounces off the wall by at most η·c. Raise the learning rate to make the jumps larger, raise c until clipping stops acting, and scrub the step slider to replay the trajectories.
stepθ₁loss‖g‖step length η‖g‖
30.0320.941.230.37
4−0.3080.550.800.24
5−0.508 (on the wall)1.4521.86.54
66.03525.67.142.14
73.89412.65.001.50
Plain gradient descent in the figure at η = 0.3 (values from the figure's code, run in Node). One step on the wall face throws θ₁ past its starting value (2.2) and sends the loss above its starting value (6.09).

With clipping at c=1c = 1 the same situation produces a step of at most ηc=0.3\eta c = 0.3. Over 50 steps the plain run reaches a maximum loss of 27.3 and ends at 1.58. The clipped run never exceeds its starting loss of 6.09 and ends at 0.50, close to the optimum's 0.29.

Gradient clipping

Clip by norm rescales the whole gradient vector when its norm exceeds a threshold cc, and leaves it alone otherwise:

g  ←  g⋅min⁡ ⁣(1, c∥g∥)g \;\leftarrow\; g\cdot\min\!\Big(1,\ \frac{c}{\|g\|}\Big)
gg
the gradient of all parameters, concatenated into one vector
∥g∥\|g\|
its L2 norm (the global norm across all parameters)
cc
threshold (max_norm), often 1.0 for transformers

Clip by value clamps each component to [−c,c][-c, c] independently. That changes the direction whenever some components are clipped and others are not. Take g=(30,40)g = (30, 40) with ∥g∥=50\|g\| = 50 and c=5c = 5:

methodresultnormdirection
none(30, 40)5053.1°
by norm, c = 5(3, 4)553.1° (unchanged)
by value, c = 5(5, 5)7.0745° (rotated)

Clipping by norm keeps the descent direction and only shortens the step, so it is the default. It is a safety cap, not a fix: when it triggers on most steps, the learning rate or the model is the real problem.

python
1import torch
2
3for x, y in loader:
4 optimizer.zero_grad()
5 loss = loss_fn(model(x), y)
6 loss.backward() # 1. gradients exist now
7 total = torch.nn.utils.clip_grad_norm_( # 2. clip them in place
8 model.parameters(), max_norm=1.0) # returns the norm *before* clipping
9 optimizer.step() # 3. update with clipped gradients
10 log("grad_norm", total.item())
11
12# With mixed precision, unscale first so the threshold applies to the true gradients:
13# scaler.scale(loss).backward()
14# scaler.unscale_(optimizer)
15# torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
16# scaler.step(optimizer); scaler.update()
Clip after backward() and before step(). clip_grad_value_ is the clip-by-value variant.

Other fixes

fixwhy it helps
Lower learning ratethe step is η∥g∥\eta\|g\|, so a smaller η\eta shrinks every jump, including those on cliffs
Careful initializationHe / Xavier variance keeps the per-layer gain near 1; for RNNs, orthogonal WhhW_{hh} starts with all singular values equal to 1
NormalizationBatchNorm / LayerNorm re-scale activations every layer, so gains cannot compound unchecked
Learning-rate warmupstart with a tiny learning rate while gradients and optimizer statistics (e.g. Adam's second moment) are still unreliable, then ramp up
Gated RNNs (LSTM, GRU)the additive cell-state path avoids repeated multiplication by the same matrix

Check yourself

In a PyTorch training step, where does clip_grad_norm_ go?

Check yourself

The gradient is g = (6, 8) and you clip by norm with c = 2. What is the clipped gradient?

Remember

  • ✓A per-layer gain above 1 compounds: 1.520≈33251.5^{20} \approx 3325.
  • ✓RNNs reuse WhhW_{hh} at every step, so gradients can explode when its spectral radius (or largest singular value) exceeds 1.
  • ✓Symptoms: loss spikes, NaN/inf, spiking gradient norms. Cliffs in the loss surface turn one step into a catapult.
  • ✓Clip by norm, g←g⋅min⁡(1,c/∥g∥)g \leftarrow g\cdot\min(1, c/\|g\|): it bounds the step and keeps the direction.
  • ✓In PyTorch: backward(), then clip_grad_norm_, then step(). Also: lower LR, good init, normalization, warmup.
← back to Deep Learning roadmap