# Exploding gradients

> If each layer multiplies the gradient by more than 1, it grows exponentially with depth; clip the gradient norm to keep updates bounded.

Source: https://www.mlfoundry.in/learn/exploding-gradients  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Exploding gradients:** is the failure mode where gradient norms grow exponentially as they are backpropagated through many layers or time steps, so a single update can move the parameters so far that training diverges.

## The same product, factors above 1

Backpropagation multiplies one local Jacobian per layer (the derivation is in **Vanishing gradients**). If the typical factor $\gamma$ is above 1, the gradient at the first layer is about $\gamma^{L}$ times the gradient at the last.

$$
\Big\|\frac{\partial \mathcal{L}}{\partial h^{(l)}}\Big\| \approx \gamma^{\,L-l}\,\Big\|\frac{\partial \mathcal{L}}{\partial h^{(L)}}\Big\|,\qquad \gamma \approx \|W^{(k)\top}D^{(k)}\|
$$

- $\mathcal{L}$: the loss
- $h^{(l)}$: activations of layer $l$; $L$ is the number of layers
- $D^{(k)}$: $\operatorname{diag}(\varphi'(z^{(k)}))$, the activation slopes at layer $k$
- $\gamma$: typical per-layer gain of the backward pass

Exponentials are unforgiving: a gain of only $1.5$ per layer over 20 layers gives $1.5^{20} \approx 3325$. In the vanishing-gradients figure, ReLU with weight scale 2 (instead of He's $\sqrt2$) grows the gradient by $1.4\times$ per layer, about $600\times$ over 20 layers.

## Recurrent networks

An RNN multiplies by the same recurrent matrix at every time step, $\partial h_T/\partial h_t = \prod_{k=t+1}^{T} D_k W_{hh}$. For a linear RNN this is $W_{hh}^{\,T-t}$, which grows like $\rho(W_{hh})^{T-t}$ along its top eigenvector when the spectral radius $\rho > 1$. Over 50 steps, $1.1^{50} \approx 117$.

With tanh ($|\varphi'| \le 1$), a largest singular value of $W_{hh}$ above 1 is **necessary** for explosion but not sufficient, because saturated units damp the product (Pascanu et al., 2013). In practice RNNs swing between both regimes during training.

## Symptoms

- **Loss spikes:** the training loss jumps up by orders of magnitude after a smooth stretch, then slowly recovers or never does
- **NaN / inf:** weights or activations overflow; in fp16 anything above $65504$ is `inf`, and `inf - inf` is `nan`
- **Huge gradient norm:** the total gradient norm logged per step shows sudden spikes far above its usual range
- **Huge weights:** parameter norms grow quickly, and the model's outputs saturate or blow up

## Cliffs in the loss landscape

Pascanu et al. (2013) showed that RNN loss surfaces contain **cliffs**: narrow regions where the loss changes very steeply. Away from a cliff the gradient is modest; on its face the gradient is enormous.

Gradient descent moves by $\eta\,\|g\|$. On the cliff face that step can be dozens of times larger than the distance to the minimum, so the parameters are catapulted far away and much of the progress so far is lost. The direction of $g$ is reasonable. Its length is the problem.

> Figure: A hand-built 2-D loss: a valley whose optimum (×) sits at the foot of a steep wall, where the wall slopes at 40 while gradients in the valley are between about 0.5 and 3. Both runs start at the same point with the same learning rate. Plain gradient descent (rose) steps onto the wall and is thrown to the right, again and again; the clipped run (mint) bounces off the wall by at most η·c. Raise the learning rate to make the jumps larger, raise c until clipping stops acting, and scrub the step slider to replay the trajectories.

Plain gradient descent in the figure at η = 0.3 (values from the figure's code, run in Node). One step on the wall face throws θ₁ past its starting value (2.2) and sends the loss above its starting value (6.09).

| step | θ₁ | loss | ‖g‖ | step length η‖g‖ |
| --- | --- | --- | --- | --- |
| 3 | 0.032 | 0.94 | 1.23 | 0.37 |
| 4 | −0.308 | 0.55 | 0.80 | 0.24 |
| 5 | −0.508 (on the wall) | 1.45 | 21.8 | 6.54 |
| 6 | 6.035 | 25.6 | 7.14 | 2.14 |
| 7 | 3.894 | 12.6 | 5.00 | 1.50 |

With clipping at $c = 1$ the same situation produces a step of at most $\eta c = 0.3$. Over 50 steps the plain run reaches a maximum loss of 27.3 and ends at 1.58. The clipped run never exceeds its starting loss of 6.09 and ends at 0.50, close to the optimum's 0.29.

## Gradient clipping

**Clip by norm** rescales the whole gradient vector when its norm exceeds a threshold $c$, and leaves it alone otherwise:

$$
g \;\leftarrow\; g\cdot\min\!\Big(1,\ \frac{c}{\|g\|}\Big)
$$

- $g$: the gradient of all parameters, concatenated into one vector
- $\|g\|$: its L2 norm (the global norm across all parameters)
- $c$: threshold (`max_norm`), often 1.0 for transformers

**Clip by value** clamps each component to $[-c, c]$ independently. That changes the direction whenever some components are clipped and others are not. Take $g = (30, 40)$ with $\|g\| = 50$ and $c = 5$:

| method | result | norm | direction |
| --- | --- | --- | --- |
| none | (30, 40) | 50 | 53.1° |
| by norm, c = 5 | (3, 4) | 5 | 53.1° (unchanged) |
| by value, c = 5 | (5, 5) | 7.07 | 45° (rotated) |

Clipping by norm keeps the descent direction and only shortens the step, so it is the default. It is a safety cap, not a fix: when it triggers on most steps, the learning rate or the model is the real problem.

```python
import torch

for x, y in loader:
    optimizer.zero_grad()
    loss = loss_fn(model(x), y)
    loss.backward()                                   # 1. gradients exist now
    total = torch.nn.utils.clip_grad_norm_(           # 2. clip them in place
        model.parameters(), max_norm=1.0)             #    returns the norm *before* clipping
    optimizer.step()                                  # 3. update with clipped gradients
    log("grad_norm", total.item())

# With mixed precision, unscale first so the threshold applies to the true gradients:
#   scaler.scale(loss).backward()
#   scaler.unscale_(optimizer)
#   torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
#   scaler.step(optimizer); scaler.update()
```

Clip after backward() and before step(). clip_grad_value_ is the clip-by-value variant.

> **Clipping in the wrong place:** Before `backward()` there is nothing to clip (or you clip the previous step's gradients). After `optimizer.step()` the large update has already happened. With gradient accumulation, clip once, after the last micro-batch's `backward()`. With AMP, call `scaler.unscale_(optimizer)` first, or you are clipping gradients multiplied by the loss scale.

## Other fixes

| fix | why it helps |
| --- | --- |
| **Lower learning rate** | the step is $\eta\\|g\\|$, so a smaller $\eta$ shrinks every jump, including those on cliffs |
| **Careful initialization** | He / Xavier variance keeps the per-layer gain near 1; for RNNs, orthogonal $W_{hh}$ starts with all singular values equal to 1 |
| **Normalization** | BatchNorm / LayerNorm re-scale activations every layer, so gains cannot compound unchecked |
| **Learning-rate warmup** | start with a tiny learning rate while gradients and optimizer statistics (e.g. Adam's second moment) are still unreliable, then ramp up |
| **Gated RNNs (LSTM, GRU)** | the additive cell-state path avoids repeated multiplication by the same matrix |

> **Pick c from data:** Log the total gradient norm (the value `clip_grad_norm_` returns) for a few hundred steps without clipping. Set $c$ a little above the typical range, so clipping only fires on the spikes.

**Question:** In a PyTorch training step, where does clip_grad_norm_ go?

- A. Before loss.backward()
- B. After loss.backward() and before optimizer.step()
- C. After optimizer.step()
- D. Before optimizer.zero_grad()

**Answer:** B. After loss.backward() and before optimizer.step() — It rescales the `.grad` tensors in place, so they must already exist (after `backward()`), and it must run before `step()` reads them to update the weights.

**Question:** The gradient is g = (6, 8) and you clip by norm with c = 2. What is the clipped gradient?

- A. (2, 2)
- B. (1.2, 1.6)
- C. (0.6, 0.8)
- D. (6, 8)

**Answer:** B. (1.2, 1.6) — $\|g\| = 10 > 2$, so $g \leftarrow g\cdot 2/10 = (1.2, 1.6)$. Its norm is 2 and its direction is unchanged. $(2, 2)$ is clip-by-value, which rotates the direction.

**Key points**

- A per-layer gain above 1 compounds: $1.5^{20} \approx 3325$.
- RNNs reuse $W_{hh}$ at every step, so gradients can explode when its spectral radius (or largest singular value) exceeds 1.
- Symptoms: loss spikes, NaN/inf, spiking gradient norms. Cliffs in the loss surface turn one step into a catapult.
- Clip by norm, $g \leftarrow g\cdot\min(1, c/\|g\|)$: it bounds the step and keeps the direction.
- In PyTorch: `backward()`, then `clip_grad_norm_`, then `step()`. Also: lower LR, good init, normalization, warmup.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
