Definition
AdamW is Adam with decoupled weight decay: each step first shrinks every weight by the factor , then applies the ordinary Adam update computed from the loss gradient alone. The decay never passes through Adam's per-parameter rescaling.
Weight decay is the oldest regularizer in neural networks: pull every weight a little toward zero on every step. For plain SGD it does not matter whether you implement it as an L2 penalty added to the loss or as a direct shrink of the weights; the two give the same update. For Adam they are different algorithms, and for years the popular libraries implemented the wrong one under the name "weight decay".
Ilya Loshchilov and Frank Hutter pointed this out in "Fixing Weight Decay Regularization in Adam" (arXiv, November 2017), published at ICLR 2019 as "Decoupled Weight Decay Regularization". They proposed SGDW and AdamW, showed that decoupling makes the best weight-decay value much less dependent on the learning rate, and that AdamW closed most of the generalization gap between Adam and SGD with momentum on CIFAR-10 and ImageNet32×32. AdamW is now the default optimizer for transformers.
The update rule
- the gradient of the loss only; no regularization term is added to it
- Adam's first and second moment estimates, element-wise
- learning rate (
lr); it multiplies the decay too - weight decay (
weight_decay, PyTorch default ) - the decoupled decay: the same fraction of every weight, whatever its gradient history
Compare Adam with an L2 penalty, which is what torch.optim.Adam(..., weight_decay=λ) does. The penalty's gradient is added to before the moments are formed:
- loss gradient plus the L2 gradient; both go into m and v
- roughly the RMS of this parameter's recent gradients
Adam divides everything in the numerator by , including the L2 term. A weight whose gradients have RMS 10 is decayed 10 times less than intended; one with RMS 0.1 is decayed 10 times more. The regularization strength ends up depending on the gradient scale, which varies by orders of magnitude between layers and changes during training. AdamW takes the decay out of the rescaled part, so every weight loses the same fraction per step.
At a glance
- Paper
- Loshchilov & Hutter, arXiv 2017; ICLR 2019, "Decoupled Weight Decay Regularization"
- PyTorch
torch.optim.AdamW(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01, amsgrad=False)- Decay per step
- , done before the Adam step; with the defaults, of each weight
- State
- two tensors per parameter (, ), as in Adam; decay adds none
- Typical λ
- to ; large language models commonly use with
- Usually not decayed
- biases, LayerNorm/BatchNorm gains and shifts; often embeddings too
decay/step L2 2.96e−2 · W 3.00e−3
w L2 0.269 · W 0.748
decay/step L2 2.98e−3 · W 3.00e−3
w L2 0.755 · W 0.748
decay/step L2 2.97e−4 · W 3.00e−3
w L2 0.949 · W 0.748
Why L2 and weight decay coincide for SGD but not for Adam
For SGD the L2 gradient is just added to the step: . That is a weight shrink of followed by a gradient step, so L2 regularization is weight decay (Loshchilov and Hutter note the equivalence holds once the coefficients are rescaled by the learning rate). With momentum the two differ slightly, because the L2 term also enters the velocity.
With any preconditioner (Adam's is ), the L2 step becomes : the decay is preconditioned too. No single makes it equal to unless is a multiple of the identity. So "Adam + L2" regularizes each parameter in proportion to , and the parameters that the loss pushes on hardest are regularized least.
There is a second, practical benefit. With L2 inside Adam, and interact: the L2 gradient competes with the loss gradient inside the normalization. With decoupled decay, sets how fast weights shrink relative to the step, and Loshchilov and Hutter found the good region of much closer to separable, so the two can be tuned more independently.
Worked example
One step from θ = 2 with g = 0.5, η = 0.01, λ = 0.1 (first step, so m̂ = g and v̂ = g²)
- 1Adam + L2. . The first Adam step is , so . Without any decay the step would also be : on step 1 the L2 term changed nothing, because Adam normalized it away.
- 2AdamW. Shrink first: . Then the Adam step on alone: . The decay removed exactly .
- 3Flip the gradient to : Adam + L2 sees and moves to ; AdamW gives . Again the L2 version is a pure sign step, and AdamW's decay is visible as the missing .
| gradient RMS √v̂ | L2 in Adam: decay per step | AdamW: decay per step | L2: decay factor over 10⁵ steps | AdamW: factor over 10⁵ steps |
|---|---|---|---|---|
| 0.01 | 0.368 | |||
| 1 | 0.368 | 0.368 | ||
| 100 | 0.990 | 0.368 |
With AdamW the decay has one time scale, steps, for every weight: on its own it would shrink any weight by over that horizon. With L2 inside Adam the same hyperparameters wipe out a quiet weight and leave a loud one essentially unregularized.
1import torch23# Decay matrices (Linear / attention / conv weights); skip biases and norm parameters.4decay, no_decay = [], []5for name, p in model.named_parameters():6 if not p.requires_grad:7 continue8 (decay if p.ndim >= 2 else no_decay).append(p)910opt = torch.optim.AdamW(11 [{"params": decay, "weight_decay": 0.1},12 {"params": no_decay, "weight_decay": 0.0}],13 lr=3e-4, betas=(0.9, 0.95), eps=1e-8,14)1516# One step from theta = 2 with grad 0.5, lr 0.01, lambda 0.117for Opt in (torch.optim.Adam, torch.optim.AdamW):18 p = torch.nn.Parameter(torch.tensor([2.0]))19 o = Opt([p], lr=0.01, weight_decay=0.1)20 p.grad = torch.tensor([0.5])21 o.step()22 print(Opt.__name__, round(p.item(), 4)) # Adam 1.99, AdamW 1.988Use when
- +Regularization strength no longer depends on each parameter's gradient scale
- +Learning rate and weight decay are closer to independent, so tuning is easier
- +No extra memory or compute over Adam
- +The default for transformers; most published recipes assume it
Avoid when
- −In PyTorch the decay is , so it is coupled to the learning rate and its schedule
- −Needs care with parameter groups: decaying biases and norm gains is usually harmful
- −Still Adam underneath: twice the parameter memory for and
- −Weight decay values do not transfer between Adam and AdamW, or between codebases with different conventions
Check yourself
Two weights train with Adam + L2 (weight_decay = λ). Weight A's gradients have RMS 10, weight B's have RMS 0.1. How do their effective decay rates compare?
Check yourself
With torch.optim.AdamW(lr=1e-3, weight_decay=0.01), by what fraction does the decay alone shrink each weight per step?
Remember
- ✓AdamW: ; the decay never enters or .
- ✓L2 inside Adam is divided by , so weights with large gradients are barely regularized and quiet weights are over-regularized.
- ✓For plain SGD, L2 and weight decay are the same thing; for adaptive methods they are not.
- ✓PyTorch:
AdamWdefaultweight_decay=0.01, applied as , so it follows the learning-rate schedule. - ✓Exclude biases and normalization parameters from decay; AdamW with λ ≈ 0.1 is the standard transformer recipe.