# Adam

> Momentum plus RMSProp plus bias correction: divide a moving average of the gradient by the root of a moving average of its square. Each step is about η in size whatever the gradient's scale, which makes Adam the default optimizer of deep learning.

Source: https://www.mlfoundry.in/learn/adam  
Section: Deep Learning › Optimization Algorithms  
Level: core · 13 min read · updated 2026-10-11

**Adam:** (adaptive moment estimation) keeps two exponential moving averages per parameter: $m$, of the gradient (the **first moment**), and $v$, of the squared gradient (the **second moment**). It corrects both for their zero start and steps by $\eta\,\hat m/(\sqrt{\hat v}+\epsilon)$. Each parameter gets a step of about $\eta$ in the direction its recent gradients agree on.

Diederik Kingma and Jimmy Ba introduced Adam in *Adam: A Method for Stochastic Optimization* (arXiv, December 2014; ICLR 2015). It combines the two ideas that came before it. From **momentum** it takes a smoothed gradient direction, written as an average rather than a sum. From **RMSProp** (and AdaGrad before it) it takes per-parameter division by the gradient's RMS. Its own contribution is the **bias correction** that makes the early steps well-behaved, plus a clean analysis of the step size. The same paper also introduced **AdaMax**, which replaces the RMS with an infinity norm.

Adam became the default optimizer for prototyping, generative models, reinforcement learning and, in its AdamW form, transformers. Its known weak spots led to follow-up work. Reddi, Kale and Kumar (2018) built problems on which Adam does not converge, and proposed **AMSGrad**. Wilson et al. (2017) found adaptive methods generalizing worse than well-tuned SGD on several benchmarks. Loshchilov and Hutter (2017, ICLR 2019) showed that Adam's L2 regularization is not weight decay, which gave **AdamW**.

## The update rule

$$
\begin{aligned}m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, & \hat m_t&=\frac{m_t}{1-\beta_1^{\,t}}\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2, & \hat v_t&=\frac{v_t}{1-\beta_2^{\,t}}\\ \theta_t&=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\end{aligned}
$$

- $g_t$: minibatch gradient at step $t$ (counting from 1); everything is element-wise
- $m_t,\ v_t$: first and second moment estimates, both initialized to 0 (PyTorch's `exp_avg`, `exp_avg_sq`)
- $\hat m_t,\ \hat v_t$: bias-corrected estimates
- $\eta$: learning rate, default $10^{-3}$
- $\beta_1,\ \beta_2$: decay rates, defaults $0.9$ (about 10 steps of memory) and $0.999$ (about 1000 steps)
- $\epsilon$: default $10^{-8}$, added to $\sqrt{\hat v_t}$ **outside** the square root

**At a glance**

- **Paper:** Kingma & Ba, arXiv:1412.6980 (2014), ICLR 2015
- **Defaults:** $\eta=10^{-3}$, $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$ (paper and PyTorch)
- **State:** $m$ and $v$: two extra tensors, i.e. $2\times$ the parameter memory (SGD with momentum stores one)
- **Step size:** $|\Delta\theta|\lesssim\eta$ per coordinate in typical conditions, at most about $3.16\eta$ at the default betas
- **Invariance:** multiplying the loss by any constant leaves the updates unchanged, as long as $\sqrt{\hat v}\gg\epsilon$
- **PyTorch:** `torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, amsgrad=False)`

## Bias correction, derived

Unrolling the recursion from $v_0=0$ gives a weighted sum of past squared gradients whose weights add up to less than 1:

$$
v_t=(1-\beta_2)\sum_{i=1}^{t}\beta_2^{\,t-i}g_i^2\quad\Longrightarrow\quad \mathbb E[v_t]=\mathbb E[g^2]\,(1-\beta_2)\sum_{i=1}^{t}\beta_2^{\,t-i}+\zeta=\mathbb E[g^2]\,(1-\beta_2^{\,t})+\zeta
$$

- $(1-\beta_2)\sum_{i=1}^t\beta_2^{t-i}$: a geometric series, equal to $1-\beta_2^t$: the total weight the average has collected so far
- $\zeta$: zero if $\mathbb E[g_i^2]$ is stationary, and small if it drifts slowly, because old terms carry little weight

So $v_t$ underestimates $\mathbb E[g^2]$ by exactly the factor $1-\beta_2^t$, and dividing by it removes the bias. The same argument applies to $m_t$ with $\beta_1$. The correction matters most when $\beta_2$ is close to 1. At $t=1$, $1-\beta_2^t=0.001$, so $v_1$ is a thousandth of its target. At $t=1000$ the factor is still only $1-0.999^{1000}=0.63$. Over the same time the $m$ correction ($\beta_1=0.9$) is gone after a few dozen steps.

What happens without it? $m$ and $v$ are biased by different amounts. At $t=1$, $m_1/\sqrt{v_1}=0.1g/\sqrt{0.001g^2}=3.16\,\mathrm{sign}(g)$, so the first step is more than three times the intended size, and it keeps growing for a few steps (up to $6\eta$ in the figure's valley). Kingma and Ba point out this is worst exactly when $\beta_2$ is near 1, which is the setting you want for sparse or noisy gradients.

## The step is about η: a trust region and a sign

The ratio $\hat m/\sqrt{\hat v}$ is dimensionless, a kind of signal-to-noise ratio for each coordinate. Since $\hat v$ estimates $\mathbb E[g^2]=\mathbb E[g]^2+\mathrm{Var}(g)$, we have $|\mathbb E[g]|\le\sqrt{\mathbb E[g^2]}$, so in expectation the ratio is at most 1 in size. When gradients agree from step to step, $|\hat m|\approx\sqrt{\hat v}$ and the update is about $\eta\,\mathrm{sign}(g)$. When they flip sign, $\hat m\approx0$ and Adam barely moves. The paper's bound for a single step is:

$$
|\Delta\theta_t|\;\lesssim\;\eta\cdot\max\!\left(1,\ \frac{1-\beta_1}{\sqrt{1-\beta_2}}\right)\;=\;3.16\,\eta\ \text{ at the defaults}
$$

- $(1-\beta_1)/\sqrt{1-\beta_2}$: reached only in the most extreme sparse case: a long run of zero gradients, then a single nonzero one
- $\eta$: the bound in the common case. Kingma and Ba read $\eta$ as a **trust region**: you can choose it from how far parameters may sensibly move

This is Adam's real selling point. The step size is set in **parameter units** by $\eta$. It is not set by the gradient's scale, which varies by orders of magnitude between layers, losses and stages of training. With $\beta_1=0$ and $\beta_2\to0$ the update is exactly $\eta\,\mathrm{sign}(g)$, i.e. signSGD. Balles and Hennig (2018) analyse Adam as a sign method whose step is damped by the variance. The cost of this behaviour is that the step does **not** shrink as the gradient shrinks near a minimum. It only shrinks as the gradients start to disagree, which is why Adam is normally paired with a learning-rate schedule.

> Figure: Adam (blue) and plain gradient descent at the same learning rate (dashed) on S · ½(x² + 25y²) from (−4, 1.6), 200 steps at lr 0.05 and default betas. Adam's first step is exactly 0.05 in each coordinate, and the lower panel shows |Δθ|/η never exceeding 1.00. It reaches 0.11 at step 100 and 3.4e-6 at step 200. A well-tuned GD (9.8e-9) is faster on this smooth problem: Adam's advantage is robustness, not speed. Move the loss scale S anywhere from 1e-4 to 1e4 and Adam's trajectory does not change at all, while GD freezes (S = 1e-2: loss 6.8 after 200 steps) or diverges (S ≥ 10). Only below S = 1e-8, where √v̂ drops toward ε = 1e-8, does Adam slow down (0.0086 at S = 1e-8, 38.4 at 1e-12). Turn bias correction off: the first step becomes 3.16η and the step grows to 6.0η by step 6.

## Where ε goes, and when it matters

The ε is there to avoid dividing by zero, but it also sets a floor. When $\sqrt{\hat v}\ll\epsilon$ the update becomes $\eta\,\hat m/\epsilon$: plain momentum SGD with learning rate $\eta/\epsilon$, and the scale-invariance is gone. For a model whose gradients are near $10^{-8}$ that is the regime it trains in (in the figure, $S=10^{-10}$ stalls at a loss of 7.0). Some recipes raise ε deliberately. RoBERTa used $\epsilon=10^{-6}$, and TensorFlow's documentation suggests 1.0 or 0.1 for Inception on ImageNet. That trades adaptivity for stability and makes Adam more SGD-like.

Implementations put ε in different places. PyTorch follows the paper's Algorithm 1 and adds ε to the **bias-corrected** $\sqrt{\hat v_t}$. TensorFlow/Keras uses the paper's "efficient" form $\eta_t=\eta\sqrt{1-\beta_2^t}/(1-\beta_1^t)$ and adds its $\hat\epsilon$ (default $10^{-7}$) to the **uncorrected** $\sqrt{v_t}$. The two are equivalent only if $\hat\epsilon=\epsilon\sqrt{1-\beta_2^t}$, so a given ε is effectively larger early in training in Keras. If you port a recipe, port ε together with where it goes.

## Worked example

The figure's valley at S = 1, start $(-4,\,1.6)$, gradient $(-4,\,40)$, $\eta=0.05$, default betas (computed in Node)

| step t | $\hat m_t$ | $\sqrt{\hat v_t}$ | $\Delta\theta_t/\eta$ | $\theta_t$ | loss |
| --- | --- | --- | --- | --- | --- |
| 1 | (−4.000, 40.00) | (4.000, 40.00) | (1.000, −1.000) | (−3.950, 1.550) | 37.83 |
| 2 | (−3.974, 39.34) | (3.975, 39.38) | (0.9997, −0.9991) | (−3.900, 1.500) | 35.73 |
| 3 | (−3.947, 38.66) | (3.950, 38.76) | (0.9991, −0.9974) | (−3.850, 1.450) | 33.70 |

**Four facts in four lines**

1. **Bias correction.** Constant $g=2$: $m_1=0.2$, $v_1=0.004$, so $\hat m_1=0.2/0.1=2$, $\hat v_1=0.004/0.001=4$, and the step is $\eta\cdot2/2=\eta$. Uncorrected: $0.2/\sqrt{0.004}=3.16$, then $4.25$ and $4.95$ on steps 2 and 3.
2. **Scale-free.** The table's x gradient is 4 and its y gradient is 40, yet both coordinates moved by exactly $\eta=0.05$. GD at the same $\eta$ would have moved y ten times further than x.
3. **Noise shrinks the step.** A gradient alternating $+1,-1,+1,\dots$ gives $\hat v\to1$ and $|\hat m|\to(1-\beta_1)/(1+\beta_1)=0.053$, so after many steps Adam moves only $0.053\,\eta$ per step.
4. **The worst case.** After 10,000 steps of exactly zero gradient, a single gradient of any size produces a step of $3.16\eta$, the bound $(1-\beta_1)/\sqrt{1-\beta_2}$. After only 1000 zero steps it is $2.52\eta$, because $\hat v$'s correction has not yet faded.

## Memory: twice the parameters

Adam stores $m$ and $v$ for every parameter. In fp32 that is 8 bytes per parameter on top of the weights: about 56 GB of optimizer state for a 7-billion-parameter model, before activations, gradients or a mixed-precision master copy. That is why large-scale training shards optimizer state across GPUs (ZeRO, FSDP), and why memory-saving variants exist. **8-bit Adam** (Dettmers et al., 2021) quantizes the states block-wise. **Adafactor** (Shazeer and Stern, 2018) stores a factored, row-and-column version of $v$. SGD with momentum needs half the state, and plain SGD needs none.

## L2 regularization is not weight decay in Adam

`torch.optim.Adam(..., weight_decay=λ)` adds $\lambda\theta$ to the gradient before the moments are computed. That decay term is then divided by $\sqrt{\hat v}$ like everything else. A weight with large, frequent gradients is barely decayed, and a weight with tiny gradients is decayed strongly. That is not what weight decay is meant to do, and it is not equivalent to the decoupled decay $\theta\leftarrow\theta-\eta\lambda\theta$ that SGD's L2 amounts to. Loshchilov and Hutter's fix is to apply the decay directly to the weights, outside the adaptive step. That is **AdamW** (see its article), and it is what `torch.optim.AdamW` implements.

```python
import torch

opt = torch.optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999), eps=1e-8)
# Weight decay: use AdamW (decoupled), not Adam(weight_decay=...) (coupled L2).
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

# The figure's valley: f = 0.5 * (x^2 + 25 y^2)
p = torch.tensor([-4.0, 1.6], requires_grad=True)
opt = torch.optim.Adam([p], lr=0.05)
for t in range(200):
    opt.zero_grad()
    loss = 0.5 * (p[0] ** 2 + 25 * p[1] ** 2)
    loss.backward()
    opt.step()
    if t == 0:
        print(p.tolist())        # about [-3.95, 1.55]: exactly lr per coordinate
print(opt.state[p].keys())       # dict_keys(['step', 'exp_avg', 'exp_avg_sq'])

# The update, written out (element-wise), as in torch.optim.Adam
@torch.no_grad()
def adam_step(p, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8):
    m.mul_(b1).add_(g, alpha=1 - b1)
    v.mul_(b2).addcmul_(g, g, value=1 - b2)
    m_hat = m / (1 - b1 ** t)
    v_hat = v / (1 - b2 ** t)
    p.sub_(lr * m_hat / (v_hat.sqrt() + eps))     # eps outside the sqrt
```

Adam in PyTorch, reproducing the table, and the rule written out

> **Adam(weight_decay) is coupled L2:** Passing `weight_decay` to `torch.optim.Adam` gives L2 regularization, which the adaptive denominator distorts. It does not give the uniform shrinkage you probably expect. For transformers and most modern recipes use `torch.optim.AdamW`, and retune $\lambda$: AdamW's typical values ($0.01$ to $0.1$) are far larger than useful L2 coefficients for Adam.

> **Adam is a sign method with a trust region:** Ignore the noise and Adam's step is $\eta\,\mathrm{sign}(g)$: magnitude information is thrown away, direction is kept per coordinate, and $\eta$ is the radius of the step. This explains why one learning rate works across layers, why rescaling the loss does not change it, and why the learning rate has to be decayed (the step will not shrink by itself near a minimum).

> **Convergence caveats and β₂:** Reddi et al. (2018) showed that because $\hat v$ can **shrink**, the effective step can grow again after rare large gradients have faded, and Adam can then fail to converge on some convex problems. AMSGrad keeps the running maximum of $\hat v$ (`amsgrad=True`). In practice the usual remedy is a lower $\beta_2$ for stability on large models: GPT-3 used $\beta_2=0.95$ and RoBERTa used $0.98$, both with learning-rate warm-up.

**Strengths**

- Step of about η per coordinate whatever the gradient scale: one learning rate works across layers and losses
- Bias correction makes the first steps well-sized
- Momentum smooths noisy gradients; steps shrink automatically where gradients disagree
- Strong defaults: $\eta=10^{-3}$ or $3\times10^{-4}$ and $\beta=(0.9,0.999)$ often work without tuning

**Limitations**

- Two extra state tensors: optimizer memory is $2\times$ the parameters
- Built-in weight_decay is coupled L2, not true weight decay (use AdamW)
- The step does not shrink near a minimum, and $\epsilon$ caps adaptivity for tiny gradients: it needs a schedule
- Can fail to converge in contrived cases, and sometimes generalizes worse than tuned SGD on vision tasks

**Question:** You multiply your loss by 1000 (e.g. sum instead of mean over a batch of 1000) and keep Adam's settings. What happens to the updates?

- A. They become 1000× larger
- B. They become about 31.6× larger
- C. Essentially nothing: m̂ and √v̂ both scale by 1000
- D. Training diverges

**Answer:** C. Essentially nothing: m̂ and √v̂ both scale by 1000 — $\hat m\to1000\hat m$ and $\sqrt{\hat v}\to1000\sqrt{\hat v}$, so the ratio is unchanged. The only exception is the $\epsilon$ term, which becomes even more negligible. SGD's update, by contrast, would be 1000× larger, and the learning rate would have to be retuned.

**Question:** Without bias correction, how large is Adam's very first step (β₁ = 0.9, β₂ = 0.999, ε negligible)?

- A. η
- B. 0.1 η
- C. about 3.16 η
- D. 0, because m₀ = 0

**Answer:** C. about 3.16 η — $m_1=0.1g$ and $v_1=0.001g^2$, so $m_1/\sqrt{v_1}=0.1/\sqrt{0.001}=3.16$. With correction both are rescaled to $g$ and $g^2$ and the step is exactly $\eta$.

**Key points**

- $m$ = EMA of $g$, $v$ = EMA of $g^2$, each divided by $1-\beta^t$; step $-\eta\,\hat m/(\sqrt{\hat v}+\epsilon)$.
- Bias correction comes from $\mathbb E[v_t]=\mathbb E[g^2](1-\beta_2^t)$; without it the first step is $3.16\eta$.
- The step is about $\eta$ per coordinate (sign-like), at most about $3.16\eta$ at the defaults, and independent of the loss's scale while $\sqrt{\hat v}\gg\epsilon$.
- Defaults: $\eta=10^{-3}$, $\beta=(0.9,0.999)$, $\epsilon=10^{-8}$ outside the square root (PyTorch); Keras's $\hat\epsilon=10^{-7}$ sits elsewhere.
- Memory: two state tensors, twice the parameters.
- Adam(weight_decay) is coupled L2; use AdamW for decoupled weight decay.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
