# AdamW

> Adam with weight decay applied directly to the weights instead of added to the gradient. L2 inside Adam is divided by √v̂, so weights with large gradients are barely regularized; AdamW shrinks every weight by the same factor 1 − ηλ per step.

Source: https://www.mlfoundry.in/learn/adamw  
Section: Deep Learning › Optimization Algorithms  
Level: core · 11 min read · updated 2026-10-11

**AdamW:** is Adam with **decoupled weight decay**: each step first shrinks every weight by the factor $1-\eta\lambda$, then applies the ordinary Adam update computed from the loss gradient alone. The decay never passes through Adam's per-parameter rescaling.

Weight decay is the oldest regularizer in neural networks: pull every weight a little toward zero on every step. For plain SGD it does not matter whether you implement it as an L2 penalty $\tfrac{\lambda}{2}\|\theta\|^2$ added to the loss or as a direct shrink of the weights; the two give the same update. For Adam they are **different algorithms**, and for years the popular libraries implemented the wrong one under the name "weight decay".

Ilya Loshchilov and Frank Hutter pointed this out in "Fixing Weight Decay Regularization in Adam" (arXiv, November 2017), published at ICLR 2019 as "Decoupled Weight Decay Regularization". They proposed **SGDW** and **AdamW**, showed that decoupling makes the best weight-decay value much less dependent on the learning rate, and that AdamW closed most of the generalization gap between Adam and SGD with momentum on CIFAR-10 and ImageNet32×32. AdamW is now the default optimizer for transformers.

## The update rule

$$
\begin{aligned}g_t&=\nabla_\theta L(\theta_{t-1})\\ m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, & \hat m_t&=m_t/(1-\beta_1^{\,t})\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2, & \hat v_t&=v_t/(1-\beta_2^{\,t})\\ \theta_t&=\theta_{t-1}-\eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}+\lambda\,\theta_{t-1}\right)\end{aligned}
$$

- $g_t$: the gradient of the loss only; no regularization term is added to it
- $m_t,\ v_t$: Adam's first and second moment estimates, element-wise
- $\eta$: learning rate (`lr`); it multiplies the decay too
- $\lambda$: weight decay (`weight_decay`, PyTorch default $0.01$)
- $\eta\lambda\,\theta_{t-1}$: the decoupled decay: the same fraction of every weight, whatever its gradient history

Compare Adam with an L2 penalty, which is what `torch.optim.Adam(..., weight_decay=λ)` does. The penalty's gradient $\lambda\theta$ is added to $g_t$ **before** the moments are formed:

$$
\tilde g_t=g_t+\lambda\theta_{t-1},\qquad \theta_t=\theta_{t-1}-\eta\,\frac{\hat{\tilde m}_t}{\sqrt{\hat{\tilde v}_t}+\epsilon}\quad\Rightarrow\quad\text{decay part}\approx\frac{\eta\lambda}{\sqrt{\hat v_t}}\,\theta_{t-1}
$$

- $\tilde g_t$: loss gradient plus the L2 gradient; both go into m and v
- $\sqrt{\hat v_t}$: roughly the RMS of this parameter's recent gradients

Adam divides everything in the numerator by $\sqrt{\hat v}$, including the L2 term. A weight whose gradients have RMS 10 is decayed 10 times less than intended; one with RMS 0.1 is decayed 10 times more. The regularization strength ends up depending on the gradient scale, which varies by orders of magnitude between layers and changes during training. AdamW takes the decay out of the rescaled part, so every weight loses the same fraction $\eta\lambda$ per step.

**At a glance**

- **Paper:** Loshchilov & Hutter, arXiv 2017; ICLR 2019, "Decoupled Weight Decay Regularization"
- **PyTorch:** `torch.optim.AdamW(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01, amsgrad=False)`
- **Decay per step:** $\theta\leftarrow\theta\,(1-\eta\lambda)$, done before the Adam step; with the defaults, $10^{-5}$ of each weight
- **State:** two tensors per parameter ($m$, $v$), as in Adam; decay adds none
- **Typical λ:** $0.01$ to $0.1$; large language models commonly use $0.1$ with $\beta_2=0.95$
- **Usually not decayed:** biases, LayerNorm/BatchNorm gains and shifts; often embeddings too

> Figure: Three weights fit the same noisy target w = 1 with the same noise, but their gradients are scaled by s = 0.1, 1 and 10, as if they sat in layers of different scale. Adam is scale-invariant, so without decay all three follow the same path (thin grey, mean 1.001). With λ = 0.3 and lr = 0.01, AdamW (solid) keeps them identical: all three settle at 0.761. Adam with an L2 penalty (dashed) splits them to 0.252, 0.772 and 0.972, exactly the s/(s + λ) you get when the decay is divided by √v̂ ≈ s. The readout shows the decay applied per step at step 2000: 2.96e−2, 2.98e−3 and 2.97e−4 for L2 versus 3.00e−3 for every AdamW weight. Push the scale spread to ×100 and the L2 weights land at 0.033 and 0.998 while AdamW does not move.

## Why L2 and weight decay coincide for SGD but not for Adam

For SGD the L2 gradient is just added to the step: $\theta_t=\theta_{t-1}-\eta(g_t+\lambda\theta_{t-1})=(1-\eta\lambda)\theta_{t-1}-\eta g_t$. That is a weight shrink of $1-\eta\lambda$ followed by a gradient step, so L2 regularization **is** weight decay (Loshchilov and Hutter note the equivalence holds once the coefficients are rescaled by the learning rate). With momentum the two differ slightly, because the L2 term also enters the velocity.

With any preconditioner $P$ (Adam's is $\operatorname{diag}(1/(\sqrt{\hat v}+\epsilon))$), the L2 step becomes $\theta-\eta P(g+\lambda\theta)$: the decay is preconditioned too. No single $\lambda$ makes it equal to $(1-\eta\lambda')\theta-\eta Pg$ unless $P$ is a multiple of the identity. So "Adam + L2" regularizes each parameter in proportion to $1/\sqrt{\hat v_i}$, and the parameters that the loss pushes on hardest are regularized least.

There is a second, practical benefit. With L2 inside Adam, $\eta$ and $\lambda$ interact: the L2 gradient competes with the loss gradient inside the normalization. With decoupled decay, $\lambda$ sets how fast weights shrink relative to the step, and Loshchilov and Hutter found the good region of $(\eta,\lambda)$ much closer to separable, so the two can be tuned more independently.

## Worked example

**One step from θ = 2 with g = 0.5, η = 0.01, λ = 0.1 (first step, so m̂ = g and v̂ = g²)**

1. **Adam + L2.** $\tilde g=0.5+0.1\cdot2=0.7$. The first Adam step is $\eta\,\tilde g/|\tilde g|=0.01$, so $\theta_1=1.990$. Without any decay the step would also be $0.01$: on step 1 the L2 term changed **nothing**, because Adam normalized it away.
2. **AdamW.** Shrink first: $2\cdot(1-0.01\cdot0.1)=1.998$. Then the Adam step on $g=0.5$ alone: $1.998-0.01=1.988$. The decay removed exactly $\eta\lambda\theta=0.002$.
3. Flip the gradient to $g=-0.5$: Adam + L2 sees $\tilde g=-0.3$ and moves to $2.010$; AdamW gives $1.998+0.01=2.008$. Again the L2 version is a pure sign step, and AdamW's decay is visible as the missing $0.002$.

Decay alone, with PyTorch's defaults $\eta=10^{-3}$, $\lambda=0.01$, for three parameters with different gradient RMS $\sqrt{\hat v}$ (computed in Node)

| gradient RMS √v̂ | L2 in Adam: decay per step | AdamW: decay per step | L2: decay factor over 10⁵ steps | AdamW: factor over 10⁵ steps |
| --- | --- | --- | --- | --- |
| 0.01 | $10^{-3}$ | $10^{-5}$ | $3.5\times10^{-44}$ | 0.368 |
| 1 | $10^{-5}$ | $10^{-5}$ | 0.368 | 0.368 |
| 100 | $10^{-7}$ | $10^{-5}$ | 0.990 | 0.368 |

With AdamW the decay has one time scale, $1/(\eta\lambda)=10^5$ steps, for every weight: on its own it would shrink any weight by $e^{-1}=0.368$ over that horizon. With L2 inside Adam the same hyperparameters wipe out a quiet weight and leave a loud one essentially unregularized.

```python
import torch

# Decay matrices (Linear / attention / conv weights); skip biases and norm parameters.
decay, no_decay = [], []
for name, p in model.named_parameters():
    if not p.requires_grad:
        continue
    (decay if p.ndim >= 2 else no_decay).append(p)

opt = torch.optim.AdamW(
    [{"params": decay, "weight_decay": 0.1},
     {"params": no_decay, "weight_decay": 0.0}],
    lr=3e-4, betas=(0.9, 0.95), eps=1e-8,
)

# One step from theta = 2 with grad 0.5, lr 0.01, lambda 0.1
for Opt in (torch.optim.Adam, torch.optim.AdamW):
    p = torch.nn.Parameter(torch.tensor([2.0]))
    o = Opt([p], lr=0.01, weight_decay=0.1)
    p.grad = torch.tensor([0.5])
    o.step()
    print(Opt.__name__, round(p.item(), 4))   # Adam 1.99, AdamW 1.988
```

AdamW in PyTorch, with the usual no-decay parameter group, and a one-step check of the numbers above

> **weight_decay means different things:** `torch.optim.Adam(weight_decay=λ)` is L2 added to the gradient; `torch.optim.AdamW(weight_decay=λ)` is decoupled decay. The defaults differ too: Adam's is 0, AdamW's is 0.01. In PyTorch the decay is multiplied by `lr`, so a learning-rate schedule also schedules the decay, and halving the learning rate halves the regularization. Some codebases instead divide by the peak learning rate so that $\lambda$ is a per-step fraction; check which convention a paper used before copying its $\lambda$.

> **Do not decay biases and norm parameters:** Weight decay is meant to limit the size of the weight matrices. Biases and the gains of LayerNorm or BatchNorm are few, do not add much capacity, and pulling a norm gain toward 0 shrinks the whole layer's output. BERT excluded `bias` and `LayerNorm.weight`; GPT-style training code decays only parameters with 2 or more dimensions. For weights that feed a normalization layer the loss is scale-invariant, so decay there mainly raises the effective learning rate by keeping the norm small.

> **Where you meet it:** Almost every transformer is trained with AdamW: BERT used Adam with the decoupled "weight decay fix", and GPT-3 and LLaMA report $\beta=(0.9,0.95)$ with weight decay 0.1. Vision transformers and modern ConvNets trained with long schedules also use AdamW, typically with warmup and cosine decay.

**Strengths**

- Regularization strength no longer depends on each parameter's gradient scale
- Learning rate and weight decay are closer to independent, so tuning is easier
- No extra memory or compute over Adam
- The default for transformers; most published recipes assume it

**Limitations**

- In PyTorch the decay is $\eta\lambda$, so it is coupled to the learning rate and its schedule
- Needs care with parameter groups: decaying biases and norm gains is usually harmful
- Still Adam underneath: twice the parameter memory for $m$ and $v$
- Weight decay values do not transfer between Adam and AdamW, or between codebases with different conventions

**Question:** Two weights train with Adam + L2 (weight_decay = λ). Weight A's gradients have RMS 10, weight B's have RMS 0.1. How do their effective decay rates compare?

- A. They are equal
- B. A is decayed 100 times less than B
- C. A is decayed 100 times more than B
- D. A is decayed 10 times less than B

**Answer:** B. A is decayed 100 times less than B — The L2 gradient $\lambda\theta$ is divided by $\sqrt{\hat v}$ along with everything else, so the decay per step is about $\eta\lambda/10$ for A and $\eta\lambda/0.1$ for B, a ratio of 100. AdamW would decay both by $\eta\lambda$.

**Question:** With torch.optim.AdamW(lr=1e-3, weight_decay=0.01), by what fraction does the decay alone shrink each weight per step?

- A. 0.01
- B. 1e-3
- C. 1e-5
- D. It depends on the weight's gradient history

**Answer:** C. 1e-5 — PyTorch multiplies the weights by $1-\text{lr}\cdot\text{weight\_decay}=1-10^{-5}$ before the Adam step. Over $10^5$ steps that alone shrinks a weight by $e^{-1}\approx0.37$. It does not depend on the gradient history: that is the point of decoupling.

**Key points**

- AdamW: $\theta\leftarrow\theta-\eta\big(\hat m/(\sqrt{\hat v}+\epsilon)+\lambda\theta\big)$; the decay never enters $m$ or $v$.
- L2 inside Adam is divided by $\sqrt{\hat v}$, so weights with large gradients are barely regularized and quiet weights are over-regularized.
- For plain SGD, L2 and weight decay are the same thing; for adaptive methods they are not.
- PyTorch: `AdamW` default `weight_decay=0.01`, applied as $\theta(1-\text{lr}\cdot\lambda)$, so it follows the learning-rate schedule.
- Exclude biases and normalization parameters from decay; AdamW with λ ≈ 0.1 is the standard transformer recipe.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
