Gradient clipping

Rescale the gradient when its global norm exceeds a threshold c, so one step can move the weights at most η·c. It keeps the direction, costs almost nothing, and turns rare gradient spikes from cliffs and bad batches into ordinary steps.

11 min read1213 wordscore0 viewsupdated 2026-10-11

Definition

Gradient clipping caps the size of the gradient before the optimizer uses it. Clipping by norm rescales the whole gradient vector gg to length cc whenever ∥g∥>c\|g\|>c and leaves it alone otherwise. Clipping by value clamps every component to [−c,c][-c,c] separately.

A gradient step is only trustworthy close to where the gradient was measured. When ∥g∥\|g\| is suddenly a hundred or a thousand times its usual size, ηg\eta g jumps far outside that region and can undo many steps of progress in one update. Clipping bounds the step at ηc\eta c and changes nothing on ordinary steps. That is why almost every recurrent network and every large language model is trained with it.

Element-wise clipping was already used in recurrent language models (Mikolov, 2012). Pascanu, Mikolov and Bengio (2013), On the difficulty of training recurrent neural networks, explained why it is needed and proposed clipping the norm. Their picture: the loss surface of an RNN has cliffs, narrow walls where a small change in the weights changes the long-run dynamics of the hidden state. The gradient on a wall is huge, but it is only accurate in a tiny neighbourhood. Following it at full length throws the parameters far away. Rescaling keeps the direction and drops the length. Language models adopted the same rule: GPT-3 and LLaMA clip the global norm at 1.0.

The two rules

by norm:   g  ←  g⋅min⁡ ⁣(1, c∥g∥2)by value:   gi  ←  clip⁡(gi, −c, c)\text{by norm: }\; g\;\leftarrow\; g\cdot\min\!\Big(1,\ \frac{c}{\|g\|_2}\Big)\qquad\qquad \text{by value: }\; g_i\;\leftarrow\;\operatorname{clip}(g_i,\,-c,\,c)
gg
the gradient of all parameters, concatenated into one vector
∥g∥2\|g\|_2
the global norm, ∑p∥gp∥2\sqrt{\sum_p\|g_p\|^2} over every parameter tensor pp
cc
the threshold (max_norm / clip_value in PyTorch)

After norm clipping the update is −η gmin⁡(1,c/∥g∥)-\eta\,g\min(1,c/\|g\|), so its length is ηmin⁡(∥g∥,c)≤ηc\eta\min(\|g\|,c)\le\eta c. The rule is a projection of gg onto the ball of radius cc. On a step where ∥g∥≤c\|g\|\le c it is the identity.

The norm is global: one factor for the whole model. Every parameter is scaled by the same amount, so the update keeps the exact direction of the true gradient. Clipping each tensor separately, or each element, changes the relative sizes of the parts and therefore the direction.

At a glance

Where
after loss.backward(), before optimizer.step()
Typical c
1.0 for transformers and LLMs; 0.25 to 5 for LSTMs and GRUs; tune from logged norms otherwise
Cost
one reduction over all gradients and one in-place multiply
Changes
only steps with ∥g∥>c\|g\|>c; the direction is kept, the length becomes cc
Not
a fix for vanishing gradients: it can only shrink
Log
the pre-clip norm that clip_grad_norm_ returns, every step

A cliff in a one-unit RNN

The figure uses a recurrent net with one unit, xt=wtanh⁡(xt−1)+bx_t=w\tanh(x_{t-1})+b, run for T=30T=30 steps from x0=0.5x_0=0.5, with loss L=(x30−0.6)2L=(x_{30}-0.6)^2. For w>1w>1 the map has two attractors, one positive and one negative. Every start in the figure sits on the plateau of the negative one (x30x_{30} between −2.6-2.6 and −3.8-3.8, loss 10.4 to 18.9). Gradient descent drifts toward the region where the negative attractor disappears. Near that boundary, x30x_{30} is extremely sensitive to (w,b)(w,b): that is the cliff.

xₜ = w·tanh(xₜ₋₁) + b, T = 30, L = (x_T − 0.6)²11.522.53−1−0.500.5w → (b ↑); darker = higher lossraw ‖g‖ per step (log)1e-211e21e4c = 12loss per step (log)1e-41e-211e21e4step 60 / 60raw: 3/12 runs thrown off the cliff, worst loss 3.0e4, max ‖g‖ 4.3e3clip by norm c = 12: 0/12 thrown, 0.7% of steps clipped, 12/12 reach L < 0.01now: largest raw ‖g‖ 0.73; longest step raw 0.02, clipped 0.02
||
fig.Twelve gradient-descent runs on the one-unit RNN, 60 steps each at learning rate 0.03. Rose runs use raw gradients, mint runs clip by norm at c = 12, just above the plateau's typical norm of 9 to 12. Three raw runs (starting at w = 2.2, 2.7 and 2.8) land on the wall, see gradient norms of 350 to 4335 and are thrown out of the window; the worst reaches loss 29756. With clipping, none is thrown, only 0.7% of steps are clipped, and all 12 reach L < 0.01. Drop c to 1: 91% of steps are clipped and only 4 of 12 runs converge in 60 steps. Change the learning rate and different runs hit the wall: whether a step lands on a cliff is luck.

Three things to read off the figure. The spikes are rare: across all raw steps the median norm is 0.56 and the 90th percentile 10.2, but the 99th percentile is 293. They are unpredictable: move the learning rate slider and different runs hit the wall. And with a threshold just above the normal range, clipping fires only on those spikes.

Worked example: one step onto the wall

Follow the run that starts at (w,b)=(2.2, −0.95)(w,b)=(2.2,\,-0.95) with η=0.03\eta=0.03. Its first four steps have ∥g∥\|g\| between 7.9 and 10.7. At step 4 it reaches (1.4249, −0.1596)(1.4249,\,-0.1596) with loss 1.104, right on the wall, and the gradient is g=(−1771.2, −3956.5)g=(-1771.2,\,-3956.5), norm 4334.9. Each option, checked in Node:

rulegradient usednormdirectionstep lengthloss after
none(−1771.2, −3956.5)4334.9−114.1°130.0529755.6
by norm, c = 12(−4.90, −10.95)12.0−114.1°0.3601.047
by value, c = 12(−12, −12)16.97−135.0°0.5091.712
by norm, c = 1(−0.41, −0.91)1.0−114.1°0.0300.084
Step 4 of the run from (2.2, −0.95), η = 0.03; loss before the step 1.104
  1. 1Raw: the update is ηg=(−53.1, −118.7)\eta g=(-53.1,\,-118.7), so the parameters jump to (54.6, 118.5)(54.6,\,118.5), where x30=173x_{30}=173 and the loss is 29756. The next several hundred steps are spent walking back.
  2. 2By norm: ∥g∥=4334.9>12\|g\|=4334.9>12, so gg is multiplied by 12/4334.9=0.0027712/4334.9=0.00277. The direction is unchanged, the step is 0.03⋅12=0.360.03\cdot12=0.36, and the loss after it is 1.047. This run then needs no further clipping and ends at L=5.4×10−4L=5.4\times10^{-4}.
  3. 3By value: both components exceed 12, so both become −12-12. The vector rotates by 21° toward the diagonal and is longer than cc, because element-wise clipping bounds each component, not the norm (≤cn\le c\sqrt{n} for nn parameters).
  4. 4A smaller cc is safer on this step, but it is also applied on the plateau, where it slows every run (the figure at c=1c=1).

Choosing the threshold

Pascanu et al. suggested setting cc from the average gradient norm over a long stretch of training. In practice: train for a few hundred steps, log the norm returned by clip_grad_norm_, and set cc a little above its typical range. Then clipping is an emergency brake that fires on a small fraction of steps.

If clipping fires on most steps, the method has changed. With ∥g∥>c\|g\|>c always, the update is −ηc g/∥g∥-\eta c\,g/\|g\|: normalized gradient descent with step length ηc\eta c. That can train, but η\eta and cc are now one knob, and the step no longer shrinks as the gradient does near a minimum. With c=1c=1, an LLM run is in this regime whenever its gradient norm is above 1, which is common early in training; as the norm falls below 1, clipping switches itself off.

Adaptive gradient clipping

A single global cc treats a layer with large weights and a layer with tiny weights the same. Adaptive gradient clipping (AGC; Brock, De, Smith and Simonyan, 2021) clips each unit's gradient relative to the size of its weights, so it bounds the relative change per step. It was what let their normalizer-free ResNets (NFNets) train at large batch sizes without batch normalization.

Gi  ←  {λ max⁡(∥Wi∥F,ϵ)∥Gi∥F Giif ∥Gi∥Fmax⁡(∥Wi∥F,ϵ)>λGiotherwiseG_i\;\leftarrow\;\begin{cases}\lambda\,\dfrac{\max(\|W_i\|_F,\epsilon)}{\|G_i\|_F}\,G_i & \text{if } \dfrac{\|G_i\|_F}{\max(\|W_i\|_F,\epsilon)}>\lambda\\[1.2ex] G_i & \text{otherwise}\end{cases}
Wi,GiW_i, G_i
the weights and gradient of unit ii (one row of a weight matrix, or one output filter of a convolution)
λ\lambda
the allowed ratio ∥Gi∥/∥Wi∥\|G_i\|/\|W_i\|, typically around 0.01 to 0.1; larger batches need smaller λ\lambda
ϵ\epsilon
10−310^{-3}, so zero-initialized weights can still move

The paper does not clip the final classifier layer. AGC is not in torch.nn.utils; it is a few lines over model.parameters(), and libraries such as timm include it.

In PyTorch

python
1import torch
2from torch.nn.utils import clip_grad_norm_, clip_grad_value_
3
4for step, (x, y) in enumerate(loader):
5 optimizer.zero_grad(set_to_none=True)
6 loss = loss_fn(model(x), y)
7 loss.backward()
8 total = clip_grad_norm_(model.parameters(), max_norm=1.0) # global L2 norm, BEFORE clipping
9 optimizer.step()
10 log(step, loss=loss.item(), grad_norm=total.item(),
11 clipped=bool(total > 1.0))
12# clip_grad_norm_ multiplies every .grad by min(1, max_norm / (total + 1e-6)) in place.
13# A NaN/inf gradient makes total NaN/inf; error_if_nonfinite=True raises instead.
14# clip_grad_value_(model.parameters(), clip_value=0.5) is the element-wise rule.
15
16# Mixed precision (fp16): gradients are multiplied by the loss scale until unscaled.
17scaler = torch.amp.GradScaler("cuda")
18for x, y in loader:
19 optimizer.zero_grad(set_to_none=True)
20 with torch.autocast(device_type="cuda", dtype=torch.float16):
21 loss = loss_fn(model(x), y)
22 scaler.scale(loss).backward()
23 scaler.unscale_(optimizer) # .grad now holds the true gradients
24 clip_grad_norm_(model.parameters(), 1.0) # threshold applies to real values
25 scaler.step(optimizer) # skipped if any gradient is inf/NaN
26 scaler.update()
Clip after backward() and before step(); log the returned pre-clip norm

Use when

  • +Bounds every update at η·c, so a single spike cannot destroy the run
  • +Norm clipping keeps the gradient's direction exactly
  • +Free to compute, one line of code, and a no-op on ordinary steps
  • +The returned norm is one of the most useful training diagnostics

Avoid when

  • −Another hyperparameter; too small a c turns training into normalized GD and slows it
  • −Clipping by value rotates the gradient and does not bound its norm
  • −Treats the symptom; frequent clipping means the setup needs fixing
  • −Sharded training needs a distributed norm, and AMP needs unscale_ first

Check yourself

The gradient is g = (3, −4, 12) and you clip by global norm with c = 6.5. What is the clipped gradient?

Check yourself

You train with torch.amp.GradScaler and call clip_grad_norm_(model.parameters(), 1.0) right after scaler.scale(loss).backward(), without unscale_. What happens?

Remember

  • ✓Clip by norm: g←gmin⁡(1,c/∥g∥)g\leftarrow g\min(1,c/\|g\|) with the global norm over all parameters; the step is at most ηc\eta c and the direction is kept.
  • ✓Clip by value clamps each component, rotating the gradient and not bounding its norm; norm clipping is the default.
  • ✓Pascanu et al. (2013): RNN losses have cliffs where the gradient is huge but valid only locally; following it at full length throws the weights away.
  • ✓Log the pre-clip norm (clip_grad_norm_ returns it) and set cc just above its usual range; LLMs typically use c=1.0c=1.0.
  • ✓Order: backward() → (scaler.unscale_) → clip → step(). With accumulation, clip once per optimizer step.
  • ✓AGC (Brock et al., 2021) clips each unit by ∥Gi∥/∥Wi∥\|G_i\|/\|W_i\|, a relative bound used in NFNets.
← back to Deep Learning roadmap