# Gradient clipping

> Rescale the gradient when its global norm exceeds a threshold c, so one step can move the weights at most η·c. It keeps the direction, costs almost nothing, and turns rare gradient spikes from cliffs and bad batches into ordinary steps.

Source: https://www.mlfoundry.in/learn/gradient-clipping  
Section: Deep Learning › Optimization Algorithms  
Level: core · 11 min read · updated 2026-10-11

**Gradient clipping:** caps the size of the gradient before the optimizer uses it. **Clipping by norm** rescales the whole gradient vector $g$ to length $c$ whenever $\|g\|>c$ and leaves it alone otherwise. **Clipping by value** clamps every component to $[-c,c]$ separately.

A gradient step is only trustworthy close to where the gradient was measured. When $\|g\|$ is suddenly a hundred or a thousand times its usual size, $\eta g$ jumps far outside that region and can undo many steps of progress in one update. Clipping bounds the step at $\eta c$ and changes nothing on ordinary steps. That is why almost every recurrent network and every large language model is trained with it.

Element-wise clipping was already used in recurrent language models (Mikolov, 2012). **Pascanu, Mikolov and Bengio (2013)**, *On the difficulty of training recurrent neural networks*, explained why it is needed and proposed clipping the **norm**. Their picture: the loss surface of an RNN has **cliffs**, narrow walls where a small change in the weights changes the long-run dynamics of the hidden state. The gradient on a wall is huge, but it is only accurate in a tiny neighbourhood. Following it at full length throws the parameters far away. Rescaling keeps the direction and drops the length. Language models adopted the same rule: GPT-3 and LLaMA clip the global norm at 1.0.

## The two rules

$$
\text{by norm: }\; g\;\leftarrow\; g\cdot\min\!\Big(1,\ \frac{c}{\|g\|_2}\Big)\qquad\qquad \text{by value: }\; g_i\;\leftarrow\;\operatorname{clip}(g_i,\,-c,\,c)
$$

- $g$: the gradient of **all** parameters, concatenated into one vector
- $\|g\|_2$: the global norm, $\sqrt{\sum_p\|g_p\|^2}$ over every parameter tensor $p$
- $c$: the threshold (`max_norm` / `clip_value` in PyTorch)

After norm clipping the update is $-\eta\,g\min(1,c/\|g\|)$, so its length is $\eta\min(\|g\|,c)\le\eta c$. The rule is a projection of $g$ onto the ball of radius $c$. On a step where $\|g\|\le c$ it is the identity.

The norm is **global**: one factor for the whole model. Every parameter is scaled by the same amount, so the update keeps the exact direction of the true gradient. Clipping each tensor separately, or each element, changes the relative sizes of the parts and therefore the direction.

**At a glance**

- **Where:** after `loss.backward()`, before `optimizer.step()`
- **Typical c:** 1.0 for transformers and LLMs; 0.25 to 5 for LSTMs and GRUs; tune from logged norms otherwise
- **Cost:** one reduction over all gradients and one in-place multiply
- **Changes:** only steps with $\|g\|>c$; the direction is kept, the length becomes $c$
- **Not:** a fix for vanishing gradients: it can only shrink
- **Log:** the pre-clip norm that `clip_grad_norm_` returns, every step

## A cliff in a one-unit RNN

The figure uses a recurrent net with one unit, $x_t=w\tanh(x_{t-1})+b$, run for $T=30$ steps from $x_0=0.5$, with loss $L=(x_{30}-0.6)^2$. For $w>1$ the map has two attractors, one positive and one negative. Every start in the figure sits on the plateau of the negative one ($x_{30}$ between $-2.6$ and $-3.8$, loss 10.4 to 18.9). Gradient descent drifts toward the region where the negative attractor disappears. Near that boundary, $x_{30}$ is extremely sensitive to $(w,b)$: that is the cliff.

> Figure: Twelve gradient-descent runs on the one-unit RNN, 60 steps each at learning rate 0.03. Rose runs use raw gradients, mint runs clip by norm at c = 12, just above the plateau's typical norm of 9 to 12. Three raw runs (starting at w = 2.2, 2.7 and 2.8) land on the wall, see gradient norms of 350 to 4335 and are thrown out of the window; the worst reaches loss 29756. With clipping, none is thrown, only 0.7% of steps are clipped, and all 12 reach L < 0.01. Drop c to 1: 91% of steps are clipped and only 4 of 12 runs converge in 60 steps. Change the learning rate and different runs hit the wall: whether a step lands on a cliff is luck.

Three things to read off the figure. The spikes are **rare**: across all raw steps the median norm is 0.56 and the 90th percentile 10.2, but the 99th percentile is 293. They are **unpredictable**: move the learning rate slider and different runs hit the wall. And with a threshold just above the normal range, clipping fires only on those spikes.

## Worked example: one step onto the wall

Follow the run that starts at $(w,b)=(2.2,\,-0.95)$ with $\eta=0.03$. Its first four steps have $\|g\|$ between 7.9 and 10.7. At step 4 it reaches $(1.4249,\,-0.1596)$ with loss 1.104, right on the wall, and the gradient is $g=(-1771.2,\,-3956.5)$, norm 4334.9. Each option, checked in Node:

Step 4 of the run from (2.2, −0.95), η = 0.03; loss before the step 1.104

| rule | gradient used | norm | direction | step length | loss after |
| --- | --- | --- | --- | --- | --- |
| none | (−1771.2, −3956.5) | 4334.9 | −114.1° | 130.05 | 29755.6 |
| by norm, c = 12 | (−4.90, −10.95) | 12.0 | −114.1° | 0.360 | 1.047 |
| by value, c = 12 | (−12, −12) | 16.97 | −135.0° | 0.509 | 1.712 |
| by norm, c = 1 | (−0.41, −0.91) | 1.0 | −114.1° | 0.030 | 0.084 |

1. Raw: the update is $\eta g=(-53.1,\,-118.7)$, so the parameters jump to $(54.6,\,118.5)$, where $x_{30}=173$ and the loss is 29756. The next several hundred steps are spent walking back.
2. By norm: $\|g\|=4334.9>12$, so $g$ is multiplied by $12/4334.9=0.00277$. The direction is unchanged, the step is $0.03\cdot12=0.36$, and the loss after it is 1.047. This run then needs no further clipping and ends at $L=5.4\times10^{-4}$.
3. By value: both components exceed 12, so both become $-12$. The vector rotates by 21° toward the diagonal and is longer than $c$, because element-wise clipping bounds each component, not the norm ($\le c\sqrt{n}$ for $n$ parameters).
4. A smaller $c$ is safer on this step, but it is also applied on the plateau, where it slows every run (the figure at $c=1$).

## Choosing the threshold

Pascanu et al. suggested setting $c$ from the average gradient norm over a long stretch of training. In practice: train for a few hundred steps, log the norm returned by `clip_grad_norm_`, and set $c$ a little above its typical range. Then clipping is an emergency brake that fires on a small fraction of steps.

If clipping fires on most steps, the method has changed. With $\|g\|>c$ always, the update is $-\eta c\,g/\|g\|$: **normalized gradient descent** with step length $\eta c$. That can train, but $\eta$ and $c$ are now one knob, and the step no longer shrinks as the gradient does near a minimum. With $c=1$, an LLM run is in this regime whenever its gradient norm is above 1, which is common early in training; as the norm falls below 1, clipping switches itself off.

> **Clipping matters with Adam too:** Adam divides by $\sqrt{v}$, so one spike cannot make a huge step: with $\beta=(0.9,0.999)$, steady gradients of 1 and one spike of 1000, the spike step is only $3.19\times$ normal. The damage comes afterwards. The spike adds $0.001\cdot1000^2=1000$ to $v$, and $v$ forgets slowly. Ten steps later updates are still $1.14\times$ normal (the spike is still in $m$); after 100 steps they are $0.033\times$ normal, and after 1000 steps $0.052\times$ (checked numerically). One unclipped spike freezes that parameter for about a thousand steps. Clipping before the optimizer keeps the spike out of both moments.

## Adaptive gradient clipping

A single global $c$ treats a layer with large weights and a layer with tiny weights the same. **Adaptive gradient clipping** (AGC; Brock, De, Smith and Simonyan, 2021) clips each unit's gradient relative to the size of its weights, so it bounds the *relative* change per step. It was what let their normalizer-free ResNets (NFNets) train at large batch sizes without batch normalization.

$$
G_i\;\leftarrow\;\begin{cases}\lambda\,\dfrac{\max(\|W_i\|_F,\epsilon)}{\|G_i\|_F}\,G_i & \text{if } \dfrac{\|G_i\|_F}{\max(\|W_i\|_F,\epsilon)}>\lambda\\[1.2ex] G_i & \text{otherwise}\end{cases}
$$

- $W_i, G_i$: the weights and gradient of unit $i$ (one row of a weight matrix, or one output filter of a convolution)
- $\lambda$: the allowed ratio $\|G_i\|/\|W_i\|$, typically around 0.01 to 0.1; larger batches need smaller $\lambda$
- $\epsilon$: $10^{-3}$, so zero-initialized weights can still move

The paper does not clip the final classifier layer. AGC is not in `torch.nn.utils`; it is a few lines over `model.parameters()`, and libraries such as `timm` include it.

## In PyTorch

```python
import torch
from torch.nn.utils import clip_grad_norm_, clip_grad_value_

for step, (x, y) in enumerate(loader):
    optimizer.zero_grad(set_to_none=True)
    loss = loss_fn(model(x), y)
    loss.backward()
    total = clip_grad_norm_(model.parameters(), max_norm=1.0)   # global L2 norm, BEFORE clipping
    optimizer.step()
    log(step, loss=loss.item(), grad_norm=total.item(),
        clipped=bool(total > 1.0))
# clip_grad_norm_ multiplies every .grad by min(1, max_norm / (total + 1e-6)) in place.
# A NaN/inf gradient makes total NaN/inf; error_if_nonfinite=True raises instead.
# clip_grad_value_(model.parameters(), clip_value=0.5) is the element-wise rule.

# Mixed precision (fp16): gradients are multiplied by the loss scale until unscaled.
scaler = torch.amp.GradScaler("cuda")
for x, y in loader:
    optimizer.zero_grad(set_to_none=True)
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        loss = loss_fn(model(x), y)
    scaler.scale(loss).backward()
    scaler.unscale_(optimizer)                # .grad now holds the true gradients
    clip_grad_norm_(model.parameters(), 1.0)  # threshold applies to real values
    scaler.step(optimizer)                    # skipped if any gradient is inf/NaN
    scaler.update()
```

Clip after backward() and before step(); log the returned pre-clip norm

> **Clipping scaled gradients:** With `GradScaler`, `.grad` holds $S\cdot g$ with $S$ around $2^{16}=65536$ until `scaler.unscale_(optimizer)` is called. Clip before that and the threshold acts on numbers 65536 times too large: every step is clipped to a tiny update. Call `unscale_` once per step, before clipping; `scaler.step` knows not to unscale again. With gradient accumulation, clip once, after the last micro-batch's `backward()`.

> **Distributed training:** Under DDP the gradients are already averaged across ranks when `backward()` returns, so every rank computes the same norm and `clip_grad_norm_` works unchanged. When parameters are sharded (FSDP, tensor parallelism), each rank holds only part of $g$: use the wrapper's own method (for example `FSDP.clip_grad_norm_` on the wrapped model), which reduces the squared norms across ranks first.

> **A symptom, not a cure:** Frequent clipping hides a problem rather than solving it: a learning rate that is too high, no warmup, a bad initialization, a missing normalization layer, or bad data. Clipping can also never help a gradient that is too **small**. Vanishing gradients need architecture fixes (gating, residual connections, normalization).

**Strengths**

- Bounds every update at η·c, so a single spike cannot destroy the run
- Norm clipping keeps the gradient's direction exactly
- Free to compute, one line of code, and a no-op on ordinary steps
- The returned norm is one of the most useful training diagnostics

**Limitations**

- Another hyperparameter; too small a c turns training into normalized GD and slows it
- Clipping by value rotates the gradient and does not bound its norm
- Treats the symptom; frequent clipping means the setup needs fixing
- Sharded training needs a distributed norm, and AMP needs unscale_ first

**Question:** The gradient is g = (3, −4, 12) and you clip by global norm with c = 6.5. What is the clipped gradient?

- A. (1.5, −2, 6)
- B. (3, −4, 6.5)
- C. (3, −4, 12)
- D. (0.5, −0.67, 2)

**Answer:** A. (1.5, −2, 6) — $\|g\|=\sqrt{9+16+144}=13>6.5$, so $g$ is multiplied by $6.5/13=0.5$, giving $(1.5,-2,6)$ with norm 6.5. $(3,-4,6.5)$ is clipping by value, which changes the direction.

**Question:** You train with torch.amp.GradScaler and call clip_grad_norm_(model.parameters(), 1.0) right after scaler.scale(loss).backward(), without unscale_. What happens?

- A. Nothing changes; the scaler handles it
- B. The norm is measured on gradients multiplied by the loss scale, so almost every step is clipped to a tiny update
- C. Clipping is skipped automatically
- D. The run immediately produces NaNs

**Answer:** B. The norm is measured on gradients multiplied by the loss scale, so almost every step is clipped to a tiny update — Until `unscale_` runs, `.grad` is $S g$ with $S$ often $2^{16}$. Its norm is far above 1, so every step is rescaled to a norm of 1 in scaled units, which is $1/S$ in real units after `scaler.step` unscales it. Training crawls. Call `scaler.unscale_(optimizer)` before clipping.

**Key points**

- Clip by norm: $g\leftarrow g\min(1,c/\|g\|)$ with the global norm over all parameters; the step is at most $\eta c$ and the direction is kept.
- Clip by value clamps each component, rotating the gradient and not bounding its norm; norm clipping is the default.
- Pascanu et al. (2013): RNN losses have cliffs where the gradient is huge but valid only locally; following it at full length throws the weights away.
- Log the pre-clip norm (`clip_grad_norm_` returns it) and set $c$ just above its usual range; LLMs typically use $c=1.0$.
- Order: `backward()` → (`scaler.unscale_`) → clip → `step()`. With accumulation, clip once per optimizer step.
- AGC (Brock et al., 2021) clips each unit by $\|G_i\|/\|W_i\|$, a relative bound used in NFNets.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
