# Weight decay

> Shrink every weight by the factor 1 − ηλ each step. For plain SGD that is L2 regularization; for Adam it is not, hence AdamW. Before normalization layers it fixes an equilibrium weight norm and with it the effective learning rate.

Source: https://www.mlfoundry.in/learn/weight-decay  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**Weight decay:** is the update $w\leftarrow(1-\eta\lambda)\,w$ applied to the weights at every optimizer step, alongside the step from the loss gradient. Each step removes the same **fraction** $\eta\lambda$ of every decayed weight, so weights the loss does not keep pushing up shrink geometrically toward zero.

Weight decay is the regularizer that almost every deep network is trained with. It is usually introduced as "L2 regularization", and for plain SGD the two are the same update. They stop being the same as soon as the optimizer rescales gradients, and in modern networks with normalization layers the decay's main effect is not to make the function simpler at all but to control the effective learning rate. This article treats weight decay as an optimizer operation; the penalty view is in **L2 regularization**.

The name comes from Hanson and Pratt (1988), who added a decay term to back-propagation to prune networks, and Krogh and Hertz (1991) analysed why it improves generalization. Loshchilov and Hutter (2017; ICLR 2019, "Decoupled Weight Decay Regularization") pointed out that adaptive optimizers had been implementing L2 under the name weight decay and proposed **SGDW** and **AdamW**. Van Laarhoven (2017) and Zhang, Wang, Xu and Grosse (2019, "Three Mechanisms of Weight Decay Regularization") showed that for layers followed by BatchNorm, weight decay acts by raising the effective learning rate.

## The update

$$
w_{t+1}=(1-\eta\lambda)\,w_t-\eta\,u_t,\qquad u_t=\begin{cases}\nabla L(w_t)&\text{SGD}\\ \hat m_t/(\sqrt{\hat v_t}+\epsilon)&\text{Adam}\to\text{AdamW}\end{cases}
$$

- $\eta$: learning rate; PyTorch multiplies the decay by it, so a schedule on η also schedules the decay
- $\lambda$: weight decay coefficient (`weight_decay`)
- $1-\eta\lambda$: per-step shrink factor; with $\eta=0.1,\ \lambda=10^{-4}$ it is $0.99999$
- $u_t$: the optimizer's own step direction, computed from the loss gradient only

**Equal to L2 only for SGD.** With $u_t=\nabla L$, the update is $w-\eta(\nabla L+\lambda w)$, exactly gradient descent on $L+\tfrac{\lambda}{2}\|w\|^2$. With an adaptive optimizer, adding $\lambda w$ to the gradient means it gets divided by $\sqrt{\hat v}$ along with everything else: a weight with large gradients is barely decayed, a quiet one is decayed hard. Applying the shrink outside the optimizer, as above, is **decoupled** weight decay: every weight loses $\eta\lambda$ of itself per step whatever its gradient history. See **AdamW** for the derivation and a figure. With momentum the coupled version also passes $\lambda w$ through the velocity, which amplifies it by about $1/(1-\mu)$ in steady state; that is a rescaling of $\lambda$, not a different regularizer.

**At a glance**

- **Update:** $w\leftarrow(1-\eta\lambda)w$ each step, before or with the optimizer step
- **Time scale:** left alone, a weight shrinks by $e^{-1}$ in $1/(\eta\lambda)$ steps
- **SGD:** identical to L2: `torch.optim.SGD(weight_decay=λ)`
- **Adam:** `torch.optim.Adam(weight_decay=λ)` is L2 (coupled); `torch.optim.AdamW(weight_decay=λ)` is decoupled, default 0.01
- **Do not decay:** biases, BatchNorm/LayerNorm/RMSNorm gains and shifts; often embeddings and positional parameters
- **With normalization:** norm settles at $\|w\|^*\approx(\eta G^2/2\lambda)^{1/4}$; the direction then turns by $\approx\sqrt{2\eta\lambda}$ rad per step

## Weight decay meets normalization: an equilibrium norm

Most weights in a modern network feed a BatchNorm, LayerNorm or RMSNorm. Then the loss depends only on the direction $w/\|w\|$, and scale invariance forces two facts: the gradient is orthogonal to $w$, and its size falls like $1/\|w\|$. Write $\|\nabla L(w)\|=G/\|w\|$, with $G$ the gradient size at unit norm. Because each gradient step is perpendicular to $w$, it can only **lengthen** $w$ (Pythagoras), while decay only shortens it:

$$
\|w_{t+1}\|^2=(1-\eta\lambda)^2\|w_t\|^2+\frac{\eta^2G^2}{\|w_t\|^2}\quad\Rightarrow\quad\|w\|^{*4}=\frac{\eta G^2}{\lambda(2-\eta\lambda)}\approx\frac{\eta G^2}{2\lambda}
$$

- $G$: gradient norm the layer would have at $\|w\|=1$ (mini-batch noise included)
- $\|w\|^*$: the norm at which shrink and growth balance; stable, reached from any starting norm

The direction moves by about $\eta\|\nabla L\|/\|w\|=\eta G/\|w\|^2$ radians per step, so $\eta/\|w\|^2$ is the **effective learning rate** of the layer. Without decay the norm grows forever, roughly as $t^{1/4}$, and the effective learning rate decays by itself; training quietly anneals. With decay, plugging in $\|w\|^*$ gives an angular step of $\sqrt{\eta\lambda(2-\eta\lambda)}\approx\sqrt{2\eta\lambda}$, independent of $G$ and of the initialization. In this regime $\eta$ and $\lambda$ matter mostly through their **product** (Li and Arora, 2020; Wan et al., 2021, who give $\sqrt{2\eta\lambda/(1+\mu)}$ for momentum $\mu$).

> Figure: A 2-D weight vector in front of a normalization layer, trained by SGD on pure gradient noise (G = 1): every gradient is perpendicular to w with size 1/‖w‖. At η = 0.1, λ = 0.01 from ‖w‖ = 0.5, the norm rises to the predicted equilibrium 1.496 (dashed green; mean of the last 500 steps 1.486) within about 500 steps, while without decay (grey) it keeps growing to 2.769 at step 3000. At the end the effective learning rate η/‖w‖² is 0.047 with decay versus 0.013 without, and the measured RMS angular step is 0.0444 rad against the predicted 0.0447. Start from ‖w‖ = 3 instead: decay pulls it down to the same equilibrium, while the undecayed run keeps growing, to 3.437 at step 3000. Try η = 0.4, λ = 0.01 and then η = 0.1, λ = 0.04: same ηλ, so the same angular step (0.089 rad), though the equilibrium norms differ (2.116 vs 1.058).

## Worked example: three starting norms

One step with $\eta=0.1$, $\lambda=0.01$, $G=1$ and a unit noise draw: shrink, then a perpendicular step of length $\eta G/\|w\|$ (computed in Node)

| ‖w‖ before | after decay ×0.999 | gradient step η/‖w‖ | ‖w‖ after | net change |
| --- | --- | --- | --- | --- |
| 0.5 | 0.4995 | 0.2000 | 0.5381 | grows |
| 1.496 (= ‖w‖*) | 1.4945 | 0.0668 | 1.4960 | unchanged |
| 3 | 2.9970 | 0.0333 | 2.9972 | shrinks |

Small weights get big perpendicular kicks and grow; large weights get small kicks and decay wins. Near the equilibrium the squared norm relaxes with time constant about $1/(4\eta\lambda)=250$ steps here, which is why the figure settles by step 500 from either side.

## Choosing λ and what to decay

Weight decay in well-known recipes (as reported in the papers)

| model | optimizer | λ |
| --- | --- | --- |
| AlexNet (Krizhevsky et al., 2012) | SGD, momentum 0.9 | 5e−4 |
| ResNet on ImageNet (He et al., 2016) | SGD, momentum 0.9 | 1e−4 |
| BERT (Devlin et al., 2019) | Adam with decoupled decay | 0.01 |
| DeiT (Touvron et al., 2021) | AdamW | 0.05 |
| GPT-3 (Brown et al., 2020) | AdamW, β₂ = 0.95 | 0.1 |

The SGD values look tiny and the AdamW values large because the per-step shrink is $\eta\lambda$: $0.1\times10^{-4}=10^{-5}$ for ResNet, $6\times10^{-4}\times0.1=6\times10^{-5}$ for a GPT-style model at peak learning rate. Compare the products, not the raw $\lambda$. Tune $\lambda$ on a log grid together with $\eta$, and re-tune when the schedule length changes, since the total shrink over training is about $\lambda\sum_t\eta_t$.

Decay **matrices and kernels**, not biases or normalization parameters. A bias or a norm gain has no scale invariance to exploit and few degrees of freedom; pulling a LayerNorm gain toward 0 simply shrinks the layer's output. The common rule is "decay parameters with 2 or more dimensions".

```python
import torch, torch.nn as nn

def param_groups(model, wd):
    decay, no_decay = [], []
    norm_types = (nn.BatchNorm1d, nn.BatchNorm2d, nn.LayerNorm, nn.GroupNorm, nn.RMSNorm)
    for module in model.modules():
        for name, p in module.named_parameters(recurse=False):
            if not p.requires_grad:
                continue
            if isinstance(module, (*norm_types, nn.Embedding)) or name == "bias":
                no_decay.append(p)
            else:
                decay.append(p)
    return [{"params": decay, "weight_decay": wd},
            {"params": no_decay, "weight_decay": 0.0}]

model = nn.Sequential(nn.Linear(512, 512), nn.LayerNorm(512), nn.GELU(), nn.Linear(512, 10))

# Decoupled (AdamW-style) decay: the standard choice for transformers
opt = torch.optim.AdamW(param_groups(model, wd=0.1), lr=3e-4, betas=(0.9, 0.95))

# SGDW by hand: shrink first, then a momentum step on the loss gradient only
sgd = torch.optim.SGD(param_groups(model, wd=0.0), lr=0.1, momentum=0.9)
def sgdw_step(lr=0.1, wd=5e-4):
    with torch.no_grad():
        for p in param_groups(model, wd)[0]["params"]:
            p.mul_(1 - lr * wd)
    sgd.step()

# Watch weight norms: before a norm layer they should level off, not keep growing
for name, p in model.named_parameters():
    if p.ndim >= 2:
        print(name, p.norm().item())
```

Parameter groups, decoupled SGD by hand, and monitoring the equilibrium norm

> **weight_decay is not one thing:** In `SGD` and `Adam` the argument adds $\lambda w$ to the gradient; in `AdamW` it is a decoupled shrink. In PyTorch the shrink is $\eta\lambda$, in some other codebases it is $\lambda$ per step (already divided by the peak learning rate). A $\lambda$ copied from a paper is only meaningful together with its optimizer and convention.

> **Decay as a learning-rate schedule:** For scale-invariant weights, removing weight decay does not make the network less regularized in the classical sense, it makes the weight norm grow and the effective learning rate fall. Li and Arora (2020) showed that SGD with weight decay and momentum on such networks is equivalent to training with an exponentially **increasing** learning rate schedule and no decay. When you change $\lambda$, think "I changed the effective step size of most layers".

> **Late in training:** At the end of a cosine or linear-decay schedule, $\eta\to0$ and so does the decay $\eta\lambda$, so weights stop shrinking. The equilibrium argument applies while $\eta$ changes slowly compared with the relaxation time $1/(4\eta\lambda)$; with very small $\lambda$ the norm never reaches equilibrium within the run, which is one reason recipes with long schedules use larger $\lambda$.

**Strengths**

- One multiply per weight per step, no extra memory
- Decoupled form regularizes every weight equally, regardless of the optimizer's rescaling
- Keeps weight norms bounded, which stabilizes effective learning rates in normalized networks
- Well-understood defaults exist for most architectures

**Limitations**

- Tied to the learning rate in PyTorch ($\eta\lambda$), so changing $\eta$ or the schedule changes the regularization
- Means different things in different optimizers and codebases
- Applied to biases or norm parameters it usually hurts, so it needs parameter groups
- For normalized layers its effect is on the dynamics, not the function, which makes its benefit harder to reason about

**Question:** A weight matrix feeds a LayerNorm and is trained with SGD, η = 0.1, λ = 0.01. Roughly how many radians does its direction turn per step once its norm has equilibrated?

- A. 0.001
- B. 0.045
- C. 0.1
- D. It depends on the gradient size G

**Answer:** B. 0.045 — At equilibrium the angular step is $\sqrt{\eta\lambda(2-\eta\lambda)}\approx\sqrt{2\cdot0.1\cdot0.01}=0.0447$, independent of $G$: a larger $G$ just makes the equilibrium norm larger, and the two effects cancel. The figure measures 0.0444.

**Question:** For which optimizer is torch's weight_decay=λ exactly the same as adding (λ/2)‖w‖² to the loss and also the same as decoupled decay?

- A. Plain SGD without momentum
- B. Adam
- C. AdamW
- D. All of them

**Answer:** A. Plain SGD without momentum — Plain SGD: $w-\eta(\nabla L+\lambda w)=(1-\eta\lambda)w-\eta\nabla L$, all three coincide. With momentum the coupled form still equals L2 in the loss, but the decay passes through the velocity. Adam's `weight_decay` equals L2 but not decoupled decay; AdamW is decoupled but not L2.

**Key points**

- Weight decay: $w\leftarrow(1-\eta\lambda)w$ each step; it removes a fixed fraction of every decayed weight.
- It equals L2 regularization for plain SGD only; with Adam use the decoupled form, **AdamW**.
- Exclude biases and normalization parameters with parameter groups; decay weight matrices and kernels.
- Before normalization layers the norm settles at $\|w\|^*\approx(\eta G^2/2\lambda)^{1/4}$, and the direction turns by $\approx\sqrt{2\eta\lambda}$ per step.
- Typical values: $10^{-4}$ to $5\times10^{-4}$ with SGD, $0.01$ to $0.1$ with AdamW; compare recipes by the product $\eta\lambda$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
