Weight decay

Shrink every weight by the factor 1 − ηλ each step. For plain SGD that is L2 regularization; for Adam it is not, hence AdamW. Before normalization layers it fixes an equilibrium weight norm and with it the effective learning rate.

11 min read900 wordscore0 viewsupdated 2026-10-11

Definition

Weight decay is the update w←(1−ηλ) ww\leftarrow(1-\eta\lambda)\,w applied to the weights at every optimizer step, alongside the step from the loss gradient. Each step removes the same fraction ηλ\eta\lambda of every decayed weight, so weights the loss does not keep pushing up shrink geometrically toward zero.

Weight decay is the regularizer that almost every deep network is trained with. It is usually introduced as "L2 regularization", and for plain SGD the two are the same update. They stop being the same as soon as the optimizer rescales gradients, and in modern networks with normalization layers the decay's main effect is not to make the function simpler at all but to control the effective learning rate. This article treats weight decay as an optimizer operation; the penalty view is in L2 regularization.

The name comes from Hanson and Pratt (1988), who added a decay term to back-propagation to prune networks, and Krogh and Hertz (1991) analysed why it improves generalization. Loshchilov and Hutter (2017; ICLR 2019, "Decoupled Weight Decay Regularization") pointed out that adaptive optimizers had been implementing L2 under the name weight decay and proposed SGDW and AdamW. Van Laarhoven (2017) and Zhang, Wang, Xu and Grosse (2019, "Three Mechanisms of Weight Decay Regularization") showed that for layers followed by BatchNorm, weight decay acts by raising the effective learning rate.

The update

wt+1=(1−ηλ) wt−η ut,ut={∇L(wt)SGDm^t/(v^t+ϵ)Adam→AdamWw_{t+1}=(1-\eta\lambda)\,w_t-\eta\,u_t,\qquad u_t=\begin{cases}\nabla L(w_t)&\text{SGD}\\ \hat m_t/(\sqrt{\hat v_t}+\epsilon)&\text{Adam}\to\text{AdamW}\end{cases}
η\eta
learning rate; PyTorch multiplies the decay by it, so a schedule on η also schedules the decay
λ\lambda
weight decay coefficient (weight_decay)
1−ηλ1-\eta\lambda
per-step shrink factor; with η=0.1, λ=10−4\eta=0.1,\ \lambda=10^{-4} it is 0.999990.99999
utu_t
the optimizer's own step direction, computed from the loss gradient only

Equal to L2 only for SGD. With ut=∇Lu_t=\nabla L, the update is w−η(∇L+λw)w-\eta(\nabla L+\lambda w), exactly gradient descent on L+λ2∥w∥2L+\tfrac{\lambda}{2}\|w\|^2. With an adaptive optimizer, adding λw\lambda w to the gradient means it gets divided by v^\sqrt{\hat v} along with everything else: a weight with large gradients is barely decayed, a quiet one is decayed hard. Applying the shrink outside the optimizer, as above, is decoupled weight decay: every weight loses ηλ\eta\lambda of itself per step whatever its gradient history. See AdamW for the derivation and a figure. With momentum the coupled version also passes λw\lambda w through the velocity, which amplifies it by about 1/(1−μ)1/(1-\mu) in steady state; that is a rescaling of λ\lambda, not a different regularizer.

At a glance

Update
w←(1−ηλ)ww\leftarrow(1-\eta\lambda)w each step, before or with the optimizer step
Time scale
left alone, a weight shrinks by e−1e^{-1} in 1/(ηλ)1/(\eta\lambda) steps
SGD
identical to L2: torch.optim.SGD(weight_decay=λ)
Adam
torch.optim.Adam(weight_decay=λ) is L2 (coupled); torch.optim.AdamW(weight_decay=λ) is decoupled, default 0.01
Do not decay
biases, BatchNorm/LayerNorm/RMSNorm gains and shifts; often embeddings and positional parameters
With normalization
norm settles at ∥w∥∗≈(ηG2/2λ)1/4\|w\|^*\approx(\eta G^2/2\lambda)^{1/4}; the direction then turns by ≈2ηλ\approx\sqrt{2\eta\lambda} rad per step

Weight decay meets normalization: an equilibrium norm

Most weights in a modern network feed a BatchNorm, LayerNorm or RMSNorm. Then the loss depends only on the direction w/∥w∥w/\|w\|, and scale invariance forces two facts: the gradient is orthogonal to ww, and its size falls like 1/∥w∥1/\|w\|. Write ∥∇L(w)∥=G/∥w∥\|\nabla L(w)\|=G/\|w\|, with GG the gradient size at unit norm. Because each gradient step is perpendicular to ww, it can only lengthen ww (Pythagoras), while decay only shortens it:

∥wt+1∥2=(1−ηλ)2∥wt∥2+η2G2∥wt∥2⇒∥w∥∗4=ηG2λ(2−ηλ)≈ηG22λ\|w_{t+1}\|^2=(1-\eta\lambda)^2\|w_t\|^2+\frac{\eta^2G^2}{\|w_t\|^2}\quad\Rightarrow\quad\|w\|^{*4}=\frac{\eta G^2}{\lambda(2-\eta\lambda)}\approx\frac{\eta G^2}{2\lambda}
GG
gradient norm the layer would have at ∥w∥=1\|w\|=1 (mini-batch noise included)
∥w∥∗\|w\|^*
the norm at which shrink and growth balance; stable, reached from any starting norm

The direction moves by about η∥∇L∥/∥w∥=ηG/∥w∥2\eta\|\nabla L\|/\|w\|=\eta G/\|w\|^2 radians per step, so η/∥w∥2\eta/\|w\|^2 is the effective learning rate of the layer. Without decay the norm grows forever, roughly as t1/4t^{1/4}, and the effective learning rate decays by itself; training quietly anneals. With decay, plugging in ∥w∥∗\|w\|^* gives an angular step of ηλ(2−ηλ)≈2ηλ\sqrt{\eta\lambda(2-\eta\lambda)}\approx\sqrt{2\eta\lambda}, independent of GG and of the initialization. In this regime η\eta and λ\lambda matter mostly through their product (Li and Arora, 2020; Wan et al., 2021, who give 2ηλ/(1+μ)\sqrt{2\eta\lambda/(1+\mu)} for momentum μ\mu).

w (last 400 steps)rings at ‖w‖ = 1, 2, 3 · ┄ equilibrium‖w‖ vs step · loss depends only on w/‖w‖, gradient ⊥ w012340100020003000with decay λ = 0.0100λ = 0‖w‖ 1.464 (λ = 0: 2.769) · equilibrium ‖w‖* = 1.496 · mean of last 500 steps 1.486effective lr η/‖w‖² = 0.0466 (λ = 0: 0.0130)RMS angle step, last 500: 0.0444 rad · predicted √(ηλ(2 − ηλ)) = 0.0447
t = 3000· per-step shrink factor 1 − ηλ = 0.99900
fig.A 2-D weight vector in front of a normalization layer, trained by SGD on pure gradient noise (G = 1): every gradient is perpendicular to w with size 1/‖w‖. At η = 0.1, λ = 0.01 from ‖w‖ = 0.5, the norm rises to the predicted equilibrium 1.496 (dashed green; mean of the last 500 steps 1.486) within about 500 steps, while without decay (grey) it keeps growing to 2.769 at step 3000. At the end the effective learning rate η/‖w‖² is 0.047 with decay versus 0.013 without, and the measured RMS angular step is 0.0444 rad against the predicted 0.0447. Start from ‖w‖ = 3 instead: decay pulls it down to the same equilibrium, while the undecayed run keeps growing, to 3.437 at step 3000. Try η = 0.4, λ = 0.01 and then η = 0.1, λ = 0.04: same ηλ, so the same angular step (0.089 rad), though the equilibrium norms differ (2.116 vs 1.058).

Worked example: three starting norms

‖w‖ beforeafter decay ×0.999gradient step η/‖w‖‖w‖ afternet change
0.50.49950.20000.5381grows
1.496 (= ‖w‖*)1.49450.06681.4960unchanged
32.99700.03332.9972shrinks
One step with η=0.1\eta=0.1, λ=0.01\lambda=0.01, G=1G=1 and a unit noise draw: shrink, then a perpendicular step of length ηG/∥w∥\eta G/\|w\| (computed in Node)

Small weights get big perpendicular kicks and grow; large weights get small kicks and decay wins. Near the equilibrium the squared norm relaxes with time constant about 1/(4ηλ)=2501/(4\eta\lambda)=250 steps here, which is why the figure settles by step 500 from either side.

Choosing λ and what to decay

modeloptimizerλ
AlexNet (Krizhevsky et al., 2012)SGD, momentum 0.95e−4
ResNet on ImageNet (He et al., 2016)SGD, momentum 0.91e−4
BERT (Devlin et al., 2019)Adam with decoupled decay0.01
DeiT (Touvron et al., 2021)AdamW0.05
GPT-3 (Brown et al., 2020)AdamW, β₂ = 0.950.1
Weight decay in well-known recipes (as reported in the papers)

The SGD values look tiny and the AdamW values large because the per-step shrink is ηλ\eta\lambda: 0.1×10−4=10−50.1\times10^{-4}=10^{-5} for ResNet, 6×10−4×0.1=6×10−56\times10^{-4}\times0.1=6\times10^{-5} for a GPT-style model at peak learning rate. Compare the products, not the raw λ\lambda. Tune λ\lambda on a log grid together with η\eta, and re-tune when the schedule length changes, since the total shrink over training is about λ∑tηt\lambda\sum_t\eta_t.

Decay matrices and kernels, not biases or normalization parameters. A bias or a norm gain has no scale invariance to exploit and few degrees of freedom; pulling a LayerNorm gain toward 0 simply shrinks the layer's output. The common rule is "decay parameters with 2 or more dimensions".

python
1import torch, torch.nn as nn
2
3def param_groups(model, wd):
4 decay, no_decay = [], []
5 norm_types = (nn.BatchNorm1d, nn.BatchNorm2d, nn.LayerNorm, nn.GroupNorm, nn.RMSNorm)
6 for module in model.modules():
7 for name, p in module.named_parameters(recurse=False):
8 if not p.requires_grad:
9 continue
10 if isinstance(module, (*norm_types, nn.Embedding)) or name == "bias":
11 no_decay.append(p)
12 else:
13 decay.append(p)
14 return [{"params": decay, "weight_decay": wd},
15 {"params": no_decay, "weight_decay": 0.0}]
16
17model = nn.Sequential(nn.Linear(512, 512), nn.LayerNorm(512), nn.GELU(), nn.Linear(512, 10))
18
19# Decoupled (AdamW-style) decay: the standard choice for transformers
20opt = torch.optim.AdamW(param_groups(model, wd=0.1), lr=3e-4, betas=(0.9, 0.95))
21
22# SGDW by hand: shrink first, then a momentum step on the loss gradient only
23sgd = torch.optim.SGD(param_groups(model, wd=0.0), lr=0.1, momentum=0.9)
24def sgdw_step(lr=0.1, wd=5e-4):
25 with torch.no_grad():
26 for p in param_groups(model, wd)[0]["params"]:
27 p.mul_(1 - lr * wd)
28 sgd.step()
29
30# Watch weight norms: before a norm layer they should level off, not keep growing
31for name, p in model.named_parameters():
32 if p.ndim >= 2:
33 print(name, p.norm().item())
Parameter groups, decoupled SGD by hand, and monitoring the equilibrium norm

Use when

  • +One multiply per weight per step, no extra memory
  • +Decoupled form regularizes every weight equally, regardless of the optimizer's rescaling
  • +Keeps weight norms bounded, which stabilizes effective learning rates in normalized networks
  • +Well-understood defaults exist for most architectures

Avoid when

  • −Tied to the learning rate in PyTorch (ηλ\eta\lambda), so changing η\eta or the schedule changes the regularization
  • −Means different things in different optimizers and codebases
  • −Applied to biases or norm parameters it usually hurts, so it needs parameter groups
  • −For normalized layers its effect is on the dynamics, not the function, which makes its benefit harder to reason about

Check yourself

A weight matrix feeds a LayerNorm and is trained with SGD, η = 0.1, λ = 0.01. Roughly how many radians does its direction turn per step once its norm has equilibrated?

Check yourself

For which optimizer is torch's weight_decay=λ exactly the same as adding (λ/2)‖w‖² to the loss and also the same as decoupled decay?

Remember

  • ✓Weight decay: w←(1−ηλ)ww\leftarrow(1-\eta\lambda)w each step; it removes a fixed fraction of every decayed weight.
  • ✓It equals L2 regularization for plain SGD only; with Adam use the decoupled form, AdamW.
  • ✓Exclude biases and normalization parameters with parameter groups; decay weight matrices and kernels.
  • ✓Before normalization layers the norm settles at ∥w∥∗≈(ηG2/2λ)1/4\|w\|^*\approx(\eta G^2/2\lambda)^{1/4}, and the direction turns by ≈2ηλ\approx\sqrt{2\eta\lambda} per step.
  • ✓Typical values: 10−410^{-4} to 5×10−45\times10^{-4} with SGD, 0.010.01 to 0.10.1 with AdamW; compare recipes by the product ηλ\eta\lambda.
← back to Deep Learning roadmap