Definition
Exploding gradients is the failure mode where gradient norms grow exponentially as they are backpropagated through many layers or time steps, so a single update can move the parameters so far that training diverges.
The same product, factors above 1
Backpropagation multiplies one local Jacobian per layer (the derivation is in Vanishing gradients). If the typical factor is above 1, the gradient at the first layer is about times the gradient at the last.
- the loss
- activations of layer ; is the number of layers
- , the activation slopes at layer
- typical per-layer gain of the backward pass
Exponentials are unforgiving: a gain of only per layer over 20 layers gives . In the vanishing-gradients figure, ReLU with weight scale 2 (instead of He's ) grows the gradient by per layer, about over 20 layers.
Recurrent networks
An RNN multiplies by the same recurrent matrix at every time step, . For a linear RNN this is , which grows like along its top eigenvector when the spectral radius . Over 50 steps, .
With tanh (), a largest singular value of above 1 is necessary for explosion but not sufficient, because saturated units damp the product (Pascanu et al., 2013). In practice RNNs swing between both regimes during training.
Symptoms
At a glance
- Loss spikes
- the training loss jumps up by orders of magnitude after a smooth stretch, then slowly recovers or never does
- NaN / inf
- weights or activations overflow; in fp16 anything above is
inf, andinf - infisnan - Huge gradient norm
- the total gradient norm logged per step shows sudden spikes far above its usual range
- Huge weights
- parameter norms grow quickly, and the model's outputs saturate or blow up
Cliffs in the loss landscape
Pascanu et al. (2013) showed that RNN loss surfaces contain cliffs: narrow regions where the loss changes very steeply. Away from a cliff the gradient is modest; on its face the gradient is enormous.
Gradient descent moves by . On the cliff face that step can be dozens of times larger than the distance to the minimum, so the parameters are catapulted far away and much of the progress so far is lost. The direction of is reasonable. Its length is the problem.
| step | θ₁ | loss | ‖g‖ | step length η‖g‖ |
|---|---|---|---|---|
| 3 | 0.032 | 0.94 | 1.23 | 0.37 |
| 4 | −0.308 | 0.55 | 0.80 | 0.24 |
| 5 | −0.508 (on the wall) | 1.45 | 21.8 | 6.54 |
| 6 | 6.035 | 25.6 | 7.14 | 2.14 |
| 7 | 3.894 | 12.6 | 5.00 | 1.50 |
With clipping at the same situation produces a step of at most . Over 50 steps the plain run reaches a maximum loss of 27.3 and ends at 1.58. The clipped run never exceeds its starting loss of 6.09 and ends at 0.50, close to the optimum's 0.29.
Gradient clipping
Clip by norm rescales the whole gradient vector when its norm exceeds a threshold , and leaves it alone otherwise:
- the gradient of all parameters, concatenated into one vector
- its L2 norm (the global norm across all parameters)
- threshold (
max_norm), often 1.0 for transformers
Clip by value clamps each component to independently. That changes the direction whenever some components are clipped and others are not. Take with and :
| method | result | norm | direction |
|---|---|---|---|
| none | (30, 40) | 50 | 53.1° |
| by norm, c = 5 | (3, 4) | 5 | 53.1° (unchanged) |
| by value, c = 5 | (5, 5) | 7.07 | 45° (rotated) |
Clipping by norm keeps the descent direction and only shortens the step, so it is the default. It is a safety cap, not a fix: when it triggers on most steps, the learning rate or the model is the real problem.
1import torch23for x, y in loader:4 optimizer.zero_grad()5 loss = loss_fn(model(x), y)6 loss.backward() # 1. gradients exist now7 total = torch.nn.utils.clip_grad_norm_( # 2. clip them in place8 model.parameters(), max_norm=1.0) # returns the norm *before* clipping9 optimizer.step() # 3. update with clipped gradients10 log("grad_norm", total.item())1112# With mixed precision, unscale first so the threshold applies to the true gradients:13# scaler.scale(loss).backward()14# scaler.unscale_(optimizer)15# torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)16# scaler.step(optimizer); scaler.update()Other fixes
| fix | why it helps |
|---|---|
| Lower learning rate | the step is , so a smaller shrinks every jump, including those on cliffs |
| Careful initialization | He / Xavier variance keeps the per-layer gain near 1; for RNNs, orthogonal starts with all singular values equal to 1 |
| Normalization | BatchNorm / LayerNorm re-scale activations every layer, so gains cannot compound unchecked |
| Learning-rate warmup | start with a tiny learning rate while gradients and optimizer statistics (e.g. Adam's second moment) are still unreliable, then ramp up |
| Gated RNNs (LSTM, GRU) | the additive cell-state path avoids repeated multiplication by the same matrix |
Check yourself
In a PyTorch training step, where does clip_grad_norm_ go?
Check yourself
The gradient is g = (6, 8) and you clip by norm with c = 2. What is the clipped gradient?
Remember
- ✓A per-layer gain above 1 compounds: .
- ✓RNNs reuse at every step, so gradients can explode when its spectral radius (or largest singular value) exceeds 1.
- ✓Symptoms: loss spikes, NaN/inf, spiking gradient norms. Cliffs in the loss surface turn one step into a catapult.
- ✓Clip by norm, : it bounds the step and keeps the direction.
- ✓In PyTorch:
backward(), thenclip_grad_norm_, thenstep(). Also: lower LR, good init, normalization, warmup.