Definition
Gradient clipping caps the size of the gradient before the optimizer uses it. Clipping by norm rescales the whole gradient vector to length whenever and leaves it alone otherwise. Clipping by value clamps every component to separately.
A gradient step is only trustworthy close to where the gradient was measured. When is suddenly a hundred or a thousand times its usual size, jumps far outside that region and can undo many steps of progress in one update. Clipping bounds the step at and changes nothing on ordinary steps. That is why almost every recurrent network and every large language model is trained with it.
Element-wise clipping was already used in recurrent language models (Mikolov, 2012). Pascanu, Mikolov and Bengio (2013), On the difficulty of training recurrent neural networks, explained why it is needed and proposed clipping the norm. Their picture: the loss surface of an RNN has cliffs, narrow walls where a small change in the weights changes the long-run dynamics of the hidden state. The gradient on a wall is huge, but it is only accurate in a tiny neighbourhood. Following it at full length throws the parameters far away. Rescaling keeps the direction and drops the length. Language models adopted the same rule: GPT-3 and LLaMA clip the global norm at 1.0.
The two rules
- the gradient of all parameters, concatenated into one vector
- the global norm, over every parameter tensor
- the threshold (
max_norm/clip_valuein PyTorch)
After norm clipping the update is , so its length is . The rule is a projection of onto the ball of radius . On a step where it is the identity.
The norm is global: one factor for the whole model. Every parameter is scaled by the same amount, so the update keeps the exact direction of the true gradient. Clipping each tensor separately, or each element, changes the relative sizes of the parts and therefore the direction.
At a glance
- Where
- after
loss.backward(), beforeoptimizer.step() - Typical c
- 1.0 for transformers and LLMs; 0.25 to 5 for LSTMs and GRUs; tune from logged norms otherwise
- Cost
- one reduction over all gradients and one in-place multiply
- Changes
- only steps with ; the direction is kept, the length becomes
- Not
- a fix for vanishing gradients: it can only shrink
- Log
- the pre-clip norm that
clip_grad_norm_returns, every step
A cliff in a one-unit RNN
The figure uses a recurrent net with one unit, , run for steps from , with loss . For the map has two attractors, one positive and one negative. Every start in the figure sits on the plateau of the negative one ( between and , loss 10.4 to 18.9). Gradient descent drifts toward the region where the negative attractor disappears. Near that boundary, is extremely sensitive to : that is the cliff.
Three things to read off the figure. The spikes are rare: across all raw steps the median norm is 0.56 and the 90th percentile 10.2, but the 99th percentile is 293. They are unpredictable: move the learning rate slider and different runs hit the wall. And with a threshold just above the normal range, clipping fires only on those spikes.
Worked example: one step onto the wall
Follow the run that starts at with . Its first four steps have between 7.9 and 10.7. At step 4 it reaches with loss 1.104, right on the wall, and the gradient is , norm 4334.9. Each option, checked in Node:
| rule | gradient used | norm | direction | step length | loss after |
|---|---|---|---|---|---|
| none | (−1771.2, −3956.5) | 4334.9 | −114.1° | 130.05 | 29755.6 |
| by norm, c = 12 | (−4.90, −10.95) | 12.0 | −114.1° | 0.360 | 1.047 |
| by value, c = 12 | (−12, −12) | 16.97 | −135.0° | 0.509 | 1.712 |
| by norm, c = 1 | (−0.41, −0.91) | 1.0 | −114.1° | 0.030 | 0.084 |
- 1Raw: the update is , so the parameters jump to , where and the loss is 29756. The next several hundred steps are spent walking back.
- 2By norm: , so is multiplied by . The direction is unchanged, the step is , and the loss after it is 1.047. This run then needs no further clipping and ends at .
- 3By value: both components exceed 12, so both become . The vector rotates by 21° toward the diagonal and is longer than , because element-wise clipping bounds each component, not the norm ( for parameters).
- 4A smaller is safer on this step, but it is also applied on the plateau, where it slows every run (the figure at ).
Choosing the threshold
Pascanu et al. suggested setting from the average gradient norm over a long stretch of training. In practice: train for a few hundred steps, log the norm returned by clip_grad_norm_, and set a little above its typical range. Then clipping is an emergency brake that fires on a small fraction of steps.
If clipping fires on most steps, the method has changed. With always, the update is : normalized gradient descent with step length . That can train, but and are now one knob, and the step no longer shrinks as the gradient does near a minimum. With , an LLM run is in this regime whenever its gradient norm is above 1, which is common early in training; as the norm falls below 1, clipping switches itself off.
Adaptive gradient clipping
A single global treats a layer with large weights and a layer with tiny weights the same. Adaptive gradient clipping (AGC; Brock, De, Smith and Simonyan, 2021) clips each unit's gradient relative to the size of its weights, so it bounds the relative change per step. It was what let their normalizer-free ResNets (NFNets) train at large batch sizes without batch normalization.
- the weights and gradient of unit (one row of a weight matrix, or one output filter of a convolution)
- the allowed ratio , typically around 0.01 to 0.1; larger batches need smaller
- , so zero-initialized weights can still move
The paper does not clip the final classifier layer. AGC is not in torch.nn.utils; it is a few lines over model.parameters(), and libraries such as timm include it.
In PyTorch
1import torch2from torch.nn.utils import clip_grad_norm_, clip_grad_value_34for step, (x, y) in enumerate(loader):5 optimizer.zero_grad(set_to_none=True)6 loss = loss_fn(model(x), y)7 loss.backward()8 total = clip_grad_norm_(model.parameters(), max_norm=1.0) # global L2 norm, BEFORE clipping9 optimizer.step()10 log(step, loss=loss.item(), grad_norm=total.item(),11 clipped=bool(total > 1.0))12# clip_grad_norm_ multiplies every .grad by min(1, max_norm / (total + 1e-6)) in place.13# A NaN/inf gradient makes total NaN/inf; error_if_nonfinite=True raises instead.14# clip_grad_value_(model.parameters(), clip_value=0.5) is the element-wise rule.1516# Mixed precision (fp16): gradients are multiplied by the loss scale until unscaled.17scaler = torch.amp.GradScaler("cuda")18for x, y in loader:19 optimizer.zero_grad(set_to_none=True)20 with torch.autocast(device_type="cuda", dtype=torch.float16):21 loss = loss_fn(model(x), y)22 scaler.scale(loss).backward()23 scaler.unscale_(optimizer) # .grad now holds the true gradients24 clip_grad_norm_(model.parameters(), 1.0) # threshold applies to real values25 scaler.step(optimizer) # skipped if any gradient is inf/NaN26 scaler.update()Use when
- +Bounds every update at η·c, so a single spike cannot destroy the run
- +Norm clipping keeps the gradient's direction exactly
- +Free to compute, one line of code, and a no-op on ordinary steps
- +The returned norm is one of the most useful training diagnostics
Avoid when
- −Another hyperparameter; too small a c turns training into normalized GD and slows it
- −Clipping by value rotates the gradient and does not bound its norm
- −Treats the symptom; frequent clipping means the setup needs fixing
- −Sharded training needs a distributed norm, and AMP needs unscale_ first
Check yourself
The gradient is g = (3, −4, 12) and you clip by global norm with c = 6.5. What is the clipped gradient?
Check yourself
You train with torch.amp.GradScaler and call clip_grad_norm_(model.parameters(), 1.0) right after scaler.scale(loss).backward(), without unscale_. What happens?
Remember
- ✓Clip by norm: with the global norm over all parameters; the step is at most and the direction is kept.
- ✓Clip by value clamps each component, rotating the gradient and not bounding its norm; norm clipping is the default.
- ✓Pascanu et al. (2013): RNN losses have cliffs where the gradient is huge but valid only locally; following it at full length throws the weights away.
- ✓Log the pre-clip norm (
clip_grad_norm_returns it) and set just above its usual range; LLMs typically use . - ✓Order:
backward()→ (scaler.unscale_) → clip →step(). With accumulation, clip once per optimizer step. - ✓AGC (Brock et al., 2021) clips each unit by , a relative bound used in NFNets.