Definition
Weight decay is the update applied to the weights at every optimizer step, alongside the step from the loss gradient. Each step removes the same fraction of every decayed weight, so weights the loss does not keep pushing up shrink geometrically toward zero.
Weight decay is the regularizer that almost every deep network is trained with. It is usually introduced as "L2 regularization", and for plain SGD the two are the same update. They stop being the same as soon as the optimizer rescales gradients, and in modern networks with normalization layers the decay's main effect is not to make the function simpler at all but to control the effective learning rate. This article treats weight decay as an optimizer operation; the penalty view is in L2 regularization.
The name comes from Hanson and Pratt (1988), who added a decay term to back-propagation to prune networks, and Krogh and Hertz (1991) analysed why it improves generalization. Loshchilov and Hutter (2017; ICLR 2019, "Decoupled Weight Decay Regularization") pointed out that adaptive optimizers had been implementing L2 under the name weight decay and proposed SGDW and AdamW. Van Laarhoven (2017) and Zhang, Wang, Xu and Grosse (2019, "Three Mechanisms of Weight Decay Regularization") showed that for layers followed by BatchNorm, weight decay acts by raising the effective learning rate.
The update
- learning rate; PyTorch multiplies the decay by it, so a schedule on η also schedules the decay
- weight decay coefficient (
weight_decay) - per-step shrink factor; with it is
- the optimizer's own step direction, computed from the loss gradient only
Equal to L2 only for SGD. With , the update is , exactly gradient descent on . With an adaptive optimizer, adding to the gradient means it gets divided by along with everything else: a weight with large gradients is barely decayed, a quiet one is decayed hard. Applying the shrink outside the optimizer, as above, is decoupled weight decay: every weight loses of itself per step whatever its gradient history. See AdamW for the derivation and a figure. With momentum the coupled version also passes through the velocity, which amplifies it by about in steady state; that is a rescaling of , not a different regularizer.
At a glance
- Update
- each step, before or with the optimizer step
- Time scale
- left alone, a weight shrinks by in steps
- SGD
- identical to L2:
torch.optim.SGD(weight_decay=λ) - Adam
torch.optim.Adam(weight_decay=λ)is L2 (coupled);torch.optim.AdamW(weight_decay=λ)is decoupled, default 0.01- Do not decay
- biases, BatchNorm/LayerNorm/RMSNorm gains and shifts; often embeddings and positional parameters
- With normalization
- norm settles at ; the direction then turns by rad per step
Weight decay meets normalization: an equilibrium norm
Most weights in a modern network feed a BatchNorm, LayerNorm or RMSNorm. Then the loss depends only on the direction , and scale invariance forces two facts: the gradient is orthogonal to , and its size falls like . Write , with the gradient size at unit norm. Because each gradient step is perpendicular to , it can only lengthen (Pythagoras), while decay only shortens it:
- gradient norm the layer would have at (mini-batch noise included)
- the norm at which shrink and growth balance; stable, reached from any starting norm
The direction moves by about radians per step, so is the effective learning rate of the layer. Without decay the norm grows forever, roughly as , and the effective learning rate decays by itself; training quietly anneals. With decay, plugging in gives an angular step of , independent of and of the initialization. In this regime and matter mostly through their product (Li and Arora, 2020; Wan et al., 2021, who give for momentum ).
Worked example: three starting norms
| ‖w‖ before | after decay ×0.999 | gradient step η/‖w‖ | ‖w‖ after | net change |
|---|---|---|---|---|
| 0.5 | 0.4995 | 0.2000 | 0.5381 | grows |
| 1.496 (= ‖w‖*) | 1.4945 | 0.0668 | 1.4960 | unchanged |
| 3 | 2.9970 | 0.0333 | 2.9972 | shrinks |
Small weights get big perpendicular kicks and grow; large weights get small kicks and decay wins. Near the equilibrium the squared norm relaxes with time constant about steps here, which is why the figure settles by step 500 from either side.
Choosing λ and what to decay
| model | optimizer | λ |
|---|---|---|
| AlexNet (Krizhevsky et al., 2012) | SGD, momentum 0.9 | 5e−4 |
| ResNet on ImageNet (He et al., 2016) | SGD, momentum 0.9 | 1e−4 |
| BERT (Devlin et al., 2019) | Adam with decoupled decay | 0.01 |
| DeiT (Touvron et al., 2021) | AdamW | 0.05 |
| GPT-3 (Brown et al., 2020) | AdamW, β₂ = 0.95 | 0.1 |
The SGD values look tiny and the AdamW values large because the per-step shrink is : for ResNet, for a GPT-style model at peak learning rate. Compare the products, not the raw . Tune on a log grid together with , and re-tune when the schedule length changes, since the total shrink over training is about .
Decay matrices and kernels, not biases or normalization parameters. A bias or a norm gain has no scale invariance to exploit and few degrees of freedom; pulling a LayerNorm gain toward 0 simply shrinks the layer's output. The common rule is "decay parameters with 2 or more dimensions".
1import torch, torch.nn as nn23def param_groups(model, wd):4 decay, no_decay = [], []5 norm_types = (nn.BatchNorm1d, nn.BatchNorm2d, nn.LayerNorm, nn.GroupNorm, nn.RMSNorm)6 for module in model.modules():7 for name, p in module.named_parameters(recurse=False):8 if not p.requires_grad:9 continue10 if isinstance(module, (*norm_types, nn.Embedding)) or name == "bias":11 no_decay.append(p)12 else:13 decay.append(p)14 return [{"params": decay, "weight_decay": wd},15 {"params": no_decay, "weight_decay": 0.0}]1617model = nn.Sequential(nn.Linear(512, 512), nn.LayerNorm(512), nn.GELU(), nn.Linear(512, 10))1819# Decoupled (AdamW-style) decay: the standard choice for transformers20opt = torch.optim.AdamW(param_groups(model, wd=0.1), lr=3e-4, betas=(0.9, 0.95))2122# SGDW by hand: shrink first, then a momentum step on the loss gradient only23sgd = torch.optim.SGD(param_groups(model, wd=0.0), lr=0.1, momentum=0.9)24def sgdw_step(lr=0.1, wd=5e-4):25 with torch.no_grad():26 for p in param_groups(model, wd)[0]["params"]:27 p.mul_(1 - lr * wd)28 sgd.step()2930# Watch weight norms: before a norm layer they should level off, not keep growing31for name, p in model.named_parameters():32 if p.ndim >= 2:33 print(name, p.norm().item())Use when
- +One multiply per weight per step, no extra memory
- +Decoupled form regularizes every weight equally, regardless of the optimizer's rescaling
- +Keeps weight norms bounded, which stabilizes effective learning rates in normalized networks
- +Well-understood defaults exist for most architectures
Avoid when
- −Tied to the learning rate in PyTorch (), so changing or the schedule changes the regularization
- −Means different things in different optimizers and codebases
- −Applied to biases or norm parameters it usually hurts, so it needs parameter groups
- −For normalized layers its effect is on the dynamics, not the function, which makes its benefit harder to reason about
Check yourself
A weight matrix feeds a LayerNorm and is trained with SGD, η = 0.1, λ = 0.01. Roughly how many radians does its direction turn per step once its norm has equilibrated?
Check yourself
For which optimizer is torch's weight_decay=λ exactly the same as adding (λ/2)‖w‖² to the loss and also the same as decoupled decay?
Remember
- ✓Weight decay: each step; it removes a fixed fraction of every decayed weight.
- ✓It equals L2 regularization for plain SGD only; with Adam use the decoupled form, AdamW.
- ✓Exclude biases and normalization parameters with parameter groups; decay weight matrices and kernels.
- ✓Before normalization layers the norm settles at , and the direction turns by per step.
- ✓Typical values: to with SGD, to with AdamW; compare recipes by the product .