AdamW

Adam with weight decay applied directly to the weights instead of added to the gradient. L2 inside Adam is divided by √v̂, so weights with large gradients are barely regularized; AdamW shrinks every weight by the same factor 1 − ηλ per step.

11 min read876 wordscore0 viewsupdated 2026-10-11

Definition

AdamW is Adam with decoupled weight decay: each step first shrinks every weight by the factor 1−ηλ1-\eta\lambda, then applies the ordinary Adam update computed from the loss gradient alone. The decay never passes through Adam's per-parameter rescaling.

Weight decay is the oldest regularizer in neural networks: pull every weight a little toward zero on every step. For plain SGD it does not matter whether you implement it as an L2 penalty λ2∥θ∥2\tfrac{\lambda}{2}\|\theta\|^2 added to the loss or as a direct shrink of the weights; the two give the same update. For Adam they are different algorithms, and for years the popular libraries implemented the wrong one under the name "weight decay".

Ilya Loshchilov and Frank Hutter pointed this out in "Fixing Weight Decay Regularization in Adam" (arXiv, November 2017), published at ICLR 2019 as "Decoupled Weight Decay Regularization". They proposed SGDW and AdamW, showed that decoupling makes the best weight-decay value much less dependent on the learning rate, and that AdamW closed most of the generalization gap between Adam and SGD with momentum on CIFAR-10 and ImageNet32×32. AdamW is now the default optimizer for transformers.

The update rule

gt=∇θL(θt−1)mt=β1mt−1+(1−β1) gt,m^t=mt/(1−β1 t)vt=β2vt−1+(1−β2) gt2,v^t=vt/(1−β2 t)θt=θt−1−η(m^tv^t+ϵ+λ θt−1)\begin{aligned}g_t&=\nabla_\theta L(\theta_{t-1})\\ m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, & \hat m_t&=m_t/(1-\beta_1^{\,t})\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2, & \hat v_t&=v_t/(1-\beta_2^{\,t})\\ \theta_t&=\theta_{t-1}-\eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}+\lambda\,\theta_{t-1}\right)\end{aligned}
gtg_t
the gradient of the loss only; no regularization term is added to it
mt, vtm_t,\ v_t
Adam's first and second moment estimates, element-wise
η\eta
learning rate (lr); it multiplies the decay too
λ\lambda
weight decay (weight_decay, PyTorch default 0.010.01)
ηλ θt−1\eta\lambda\,\theta_{t-1}
the decoupled decay: the same fraction of every weight, whatever its gradient history

Compare Adam with an L2 penalty, which is what torch.optim.Adam(..., weight_decay=λ) does. The penalty's gradient λθ\lambda\theta is added to gtg_t before the moments are formed:

g~t=gt+λθt−1,θt=θt−1−η m~^tv~^t+ϵ⇒decay part≈ηλv^t θt−1\tilde g_t=g_t+\lambda\theta_{t-1},\qquad \theta_t=\theta_{t-1}-\eta\,\frac{\hat{\tilde m}_t}{\sqrt{\hat{\tilde v}_t}+\epsilon}\quad\Rightarrow\quad\text{decay part}\approx\frac{\eta\lambda}{\sqrt{\hat v_t}}\,\theta_{t-1}
g~t\tilde g_t
loss gradient plus the L2 gradient; both go into m and v
v^t\sqrt{\hat v_t}
roughly the RMS of this parameter's recent gradients

Adam divides everything in the numerator by v^\sqrt{\hat v}, including the L2 term. A weight whose gradients have RMS 10 is decayed 10 times less than intended; one with RMS 0.1 is decayed 10 times more. The regularization strength ends up depending on the gradient scale, which varies by orders of magnitude between layers and changes during training. AdamW takes the decay out of the rescaled part, so every weight loses the same fraction ηλ\eta\lambda per step.

At a glance

Paper
Loshchilov & Hutter, arXiv 2017; ICLR 2019, "Decoupled Weight Decay Regularization"
PyTorch
torch.optim.AdamW(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01, amsgrad=False)
Decay per step
θ←θ (1−ηλ)\theta\leftarrow\theta\,(1-\eta\lambda), done before the Adam step; with the defaults, 10−510^{-5} of each weight
State
two tensors per parameter (mm, vv), as in Adam; decay adds none
Typical λ
0.010.01 to 0.10.1; large language models commonly use 0.10.1 with β2=0.95\beta_2=0.95
Usually not decayed
biases, LayerNorm/BatchNorm gains and shifts; often embeddings too
w(t), loss ½s(w − 1)² + noise, lr = 0.01, λ = 0.3000.250.50.7510500100015002000stepmean w, steps 1001–200000.510.25|0.76s=0.100.77|0.76s=1.00.97|0.76s=10┄ L2 | ■ AdamW | ━ s/(s+λ)┄ Adam + L2 (dashed) · ━ AdamW (solid, the three coincide) · thin grey: no decayAdamW shrinks every weight by ηλ = 3.00e−3 per step; L2 divides that by √v̂ ≈ s
t = 2000|
● s = 0.10 √v̂ 0.101
decay/step L2 2.96e−2 · W 3.00e−3
w L2 0.269 · W 0.748
● s = 1.0 √v̂ 1.008
decay/step L2 2.98e−3 · W 3.00e−3
w L2 0.755 · W 0.748
● s = 10 √v̂ 10.086
decay/step L2 2.97e−4 · W 3.00e−3
w L2 0.949 · W 0.748
fig.Three weights fit the same noisy target w = 1 with the same noise, but their gradients are scaled by s = 0.1, 1 and 10, as if they sat in layers of different scale. Adam is scale-invariant, so without decay all three follow the same path (thin grey, mean 1.001). With λ = 0.3 and lr = 0.01, AdamW (solid) keeps them identical: all three settle at 0.761. Adam with an L2 penalty (dashed) splits them to 0.252, 0.772 and 0.972, exactly the s/(s + λ) you get when the decay is divided by √v̂ ≈ s. The readout shows the decay applied per step at step 2000: 2.96e−2, 2.98e−3 and 2.97e−4 for L2 versus 3.00e−3 for every AdamW weight. Push the scale spread to ×100 and the L2 weights land at 0.033 and 0.998 while AdamW does not move.

Why L2 and weight decay coincide for SGD but not for Adam

For SGD the L2 gradient is just added to the step: θt=θt−1−η(gt+λθt−1)=(1−ηλ)θt−1−ηgt\theta_t=\theta_{t-1}-\eta(g_t+\lambda\theta_{t-1})=(1-\eta\lambda)\theta_{t-1}-\eta g_t. That is a weight shrink of 1−ηλ1-\eta\lambda followed by a gradient step, so L2 regularization is weight decay (Loshchilov and Hutter note the equivalence holds once the coefficients are rescaled by the learning rate). With momentum the two differ slightly, because the L2 term also enters the velocity.

With any preconditioner PP (Adam's is diag⁡(1/(v^+ϵ))\operatorname{diag}(1/(\sqrt{\hat v}+\epsilon))), the L2 step becomes θ−ηP(g+λθ)\theta-\eta P(g+\lambda\theta): the decay is preconditioned too. No single λ\lambda makes it equal to (1−ηλ′)θ−ηPg(1-\eta\lambda')\theta-\eta Pg unless PP is a multiple of the identity. So "Adam + L2" regularizes each parameter in proportion to 1/v^i1/\sqrt{\hat v_i}, and the parameters that the loss pushes on hardest are regularized least.

There is a second, practical benefit. With L2 inside Adam, η\eta and λ\lambda interact: the L2 gradient competes with the loss gradient inside the normalization. With decoupled decay, λ\lambda sets how fast weights shrink relative to the step, and Loshchilov and Hutter found the good region of (η,λ)(\eta,\lambda) much closer to separable, so the two can be tuned more independently.

Worked example

One step from θ = 2 with g = 0.5, η = 0.01, λ = 0.1 (first step, so m̂ = g and v̂ = g²)

  1. 1Adam + L2. g~=0.5+0.1⋅2=0.7\tilde g=0.5+0.1\cdot2=0.7. The first Adam step is η g~/∣g~∣=0.01\eta\,\tilde g/|\tilde g|=0.01, so θ1=1.990\theta_1=1.990. Without any decay the step would also be 0.010.01: on step 1 the L2 term changed nothing, because Adam normalized it away.
  2. 2AdamW. Shrink first: 2⋅(1−0.01⋅0.1)=1.9982\cdot(1-0.01\cdot0.1)=1.998. Then the Adam step on g=0.5g=0.5 alone: 1.998−0.01=1.9881.998-0.01=1.988. The decay removed exactly ηλθ=0.002\eta\lambda\theta=0.002.
  3. 3Flip the gradient to g=−0.5g=-0.5: Adam + L2 sees g~=−0.3\tilde g=-0.3 and moves to 2.0102.010; AdamW gives 1.998+0.01=2.0081.998+0.01=2.008. Again the L2 version is a pure sign step, and AdamW's decay is visible as the missing 0.0020.002.
gradient RMS √v̂L2 in Adam: decay per stepAdamW: decay per stepL2: decay factor over 10⁵ stepsAdamW: factor over 10⁵ steps
0.0110−310^{-3}10−510^{-5}3.5×10−443.5\times10^{-44}0.368
110−510^{-5}10−510^{-5}0.3680.368
10010−710^{-7}10−510^{-5}0.9900.368
Decay alone, with PyTorch's defaults η=10−3\eta=10^{-3}, λ=0.01\lambda=0.01, for three parameters with different gradient RMS v^\sqrt{\hat v} (computed in Node)

With AdamW the decay has one time scale, 1/(ηλ)=1051/(\eta\lambda)=10^5 steps, for every weight: on its own it would shrink any weight by e−1=0.368e^{-1}=0.368 over that horizon. With L2 inside Adam the same hyperparameters wipe out a quiet weight and leave a loud one essentially unregularized.

python
1import torch
2
3# Decay matrices (Linear / attention / conv weights); skip biases and norm parameters.
4decay, no_decay = [], []
5for name, p in model.named_parameters():
6 if not p.requires_grad:
7 continue
8 (decay if p.ndim >= 2 else no_decay).append(p)
9
10opt = torch.optim.AdamW(
11 [{"params": decay, "weight_decay": 0.1},
12 {"params": no_decay, "weight_decay": 0.0}],
13 lr=3e-4, betas=(0.9, 0.95), eps=1e-8,
14)
15
16# One step from theta = 2 with grad 0.5, lr 0.01, lambda 0.1
17for Opt in (torch.optim.Adam, torch.optim.AdamW):
18 p = torch.nn.Parameter(torch.tensor([2.0]))
19 o = Opt([p], lr=0.01, weight_decay=0.1)
20 p.grad = torch.tensor([0.5])
21 o.step()
22 print(Opt.__name__, round(p.item(), 4)) # Adam 1.99, AdamW 1.988
AdamW in PyTorch, with the usual no-decay parameter group, and a one-step check of the numbers above

Use when

  • +Regularization strength no longer depends on each parameter's gradient scale
  • +Learning rate and weight decay are closer to independent, so tuning is easier
  • +No extra memory or compute over Adam
  • +The default for transformers; most published recipes assume it

Avoid when

  • −In PyTorch the decay is ηλ\eta\lambda, so it is coupled to the learning rate and its schedule
  • −Needs care with parameter groups: decaying biases and norm gains is usually harmful
  • −Still Adam underneath: twice the parameter memory for mm and vv
  • −Weight decay values do not transfer between Adam and AdamW, or between codebases with different conventions

Check yourself

Two weights train with Adam + L2 (weight_decay = λ). Weight A's gradients have RMS 10, weight B's have RMS 0.1. How do their effective decay rates compare?

Check yourself

With torch.optim.AdamW(lr=1e-3, weight_decay=0.01), by what fraction does the decay alone shrink each weight per step?

Remember

  • ✓AdamW: θ←θ−η(m^/(v^+ϵ)+λθ)\theta\leftarrow\theta-\eta\big(\hat m/(\sqrt{\hat v}+\epsilon)+\lambda\theta\big); the decay never enters mm or vv.
  • ✓L2 inside Adam is divided by v^\sqrt{\hat v}, so weights with large gradients are barely regularized and quiet weights are over-regularized.
  • ✓For plain SGD, L2 and weight decay are the same thing; for adaptive methods they are not.
  • ✓PyTorch: AdamW default weight_decay=0.01, applied as θ(1−lr⋅λ)\theta(1-\text{lr}\cdot\lambda), so it follows the learning-rate schedule.
  • ✓Exclude biases and normalization parameters from decay; AdamW with λ ≈ 0.1 is the standard transformer recipe.
← back to Deep Learning roadmap