Adam

Momentum plus RMSProp plus bias correction: divide a moving average of the gradient by the root of a moving average of its square. Each step is about η in size whatever the gradient's scale, which makes Adam the default optimizer of deep learning.

13 min read1287 wordscore0 viewsupdated 2026-10-11

Definition

Adam (adaptive moment estimation) keeps two exponential moving averages per parameter: mm, of the gradient (the first moment), and vv, of the squared gradient (the second moment). It corrects both for their zero start and steps by η m^/(v^+ϵ)\eta\,\hat m/(\sqrt{\hat v}+\epsilon). Each parameter gets a step of about η\eta in the direction its recent gradients agree on.

Diederik Kingma and Jimmy Ba introduced Adam in Adam: A Method for Stochastic Optimization (arXiv, December 2014; ICLR 2015). It combines the two ideas that came before it. From momentum it takes a smoothed gradient direction, written as an average rather than a sum. From RMSProp (and AdaGrad before it) it takes per-parameter division by the gradient's RMS. Its own contribution is the bias correction that makes the early steps well-behaved, plus a clean analysis of the step size. The same paper also introduced AdaMax, which replaces the RMS with an infinity norm.

Adam became the default optimizer for prototyping, generative models, reinforcement learning and, in its AdamW form, transformers. Its known weak spots led to follow-up work. Reddi, Kale and Kumar (2018) built problems on which Adam does not converge, and proposed AMSGrad. Wilson et al. (2017) found adaptive methods generalizing worse than well-tuned SGD on several benchmarks. Loshchilov and Hutter (2017, ICLR 2019) showed that Adam's L2 regularization is not weight decay, which gave AdamW.

The update rule

mt=β1mt−1+(1−β1) gt,m^t=mt1−β1 tvt=β2vt−1+(1−β2) gt2,v^t=vt1−β2 tθt=θt−1−η m^tv^t+ϵ\begin{aligned}m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, & \hat m_t&=\frac{m_t}{1-\beta_1^{\,t}}\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2, & \hat v_t&=\frac{v_t}{1-\beta_2^{\,t}}\\ \theta_t&=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\end{aligned}
gtg_t
minibatch gradient at step tt (counting from 1); everything is element-wise
mt, vtm_t,\ v_t
first and second moment estimates, both initialized to 0 (PyTorch's exp_avg, exp_avg_sq)
m^t, v^t\hat m_t,\ \hat v_t
bias-corrected estimates
η\eta
learning rate, default 10−310^{-3}
β1, β2\beta_1,\ \beta_2
decay rates, defaults 0.90.9 (about 10 steps of memory) and 0.9990.999 (about 1000 steps)
ϵ\epsilon
default 10−810^{-8}, added to v^t\sqrt{\hat v_t} outside the square root

At a glance

Paper
Kingma & Ba, arXiv:1412.6980 (2014), ICLR 2015
Defaults
η=10−3\eta=10^{-3}, β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8} (paper and PyTorch)
State
mm and vv: two extra tensors, i.e. 2×2\times the parameter memory (SGD with momentum stores one)
Step size
∣Δθ∣≲η|\Delta\theta|\lesssim\eta per coordinate in typical conditions, at most about 3.16η3.16\eta at the default betas
Invariance
multiplying the loss by any constant leaves the updates unchanged, as long as v^≫ϵ\sqrt{\hat v}\gg\epsilon
PyTorch
torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, amsgrad=False)

Bias correction, derived

Unrolling the recursion from v0=0v_0=0 gives a weighted sum of past squared gradients whose weights add up to less than 1:

vt=(1−β2)∑i=1tβ2 t−igi2⟹E[vt]=E[g2] (1−β2)∑i=1tβ2 t−i+ζ=E[g2] (1−β2 t)+ζv_t=(1-\beta_2)\sum_{i=1}^{t}\beta_2^{\,t-i}g_i^2\quad\Longrightarrow\quad \mathbb E[v_t]=\mathbb E[g^2]\,(1-\beta_2)\sum_{i=1}^{t}\beta_2^{\,t-i}+\zeta=\mathbb E[g^2]\,(1-\beta_2^{\,t})+\zeta
(1−β2)∑i=1tβ2t−i(1-\beta_2)\sum_{i=1}^t\beta_2^{t-i}
a geometric series, equal to 1−β2t1-\beta_2^t: the total weight the average has collected so far
ζ\zeta
zero if E[gi2]\mathbb E[g_i^2] is stationary, and small if it drifts slowly, because old terms carry little weight

So vtv_t underestimates E[g2]\mathbb E[g^2] by exactly the factor 1−β2t1-\beta_2^t, and dividing by it removes the bias. The same argument applies to mtm_t with β1\beta_1. The correction matters most when β2\beta_2 is close to 1. At t=1t=1, 1−β2t=0.0011-\beta_2^t=0.001, so v1v_1 is a thousandth of its target. At t=1000t=1000 the factor is still only 1−0.9991000=0.631-0.999^{1000}=0.63. Over the same time the mm correction (β1=0.9\beta_1=0.9) is gone after a few dozen steps.

What happens without it? mm and vv are biased by different amounts. At t=1t=1, m1/v1=0.1g/0.001g2=3.16 sign(g)m_1/\sqrt{v_1}=0.1g/\sqrt{0.001g^2}=3.16\,\mathrm{sign}(g), so the first step is more than three times the intended size, and it keeps growing for a few steps (up to 6η6\eta in the figure's valley). Kingma and Ba point out this is worst exactly when β2\beta_2 is near 1, which is the setting you want for sparse or noisy gradients.

The step is about η: a trust region and a sign

The ratio m^/v^\hat m/\sqrt{\hat v} is dimensionless, a kind of signal-to-noise ratio for each coordinate. Since v^\hat v estimates E[g2]=E[g]2+Var(g)\mathbb E[g^2]=\mathbb E[g]^2+\mathrm{Var}(g), we have ∣E[g]∣≤E[g2]|\mathbb E[g]|\le\sqrt{\mathbb E[g^2]}, so in expectation the ratio is at most 1 in size. When gradients agree from step to step, ∣m^∣≈v^|\hat m|\approx\sqrt{\hat v} and the update is about η sign(g)\eta\,\mathrm{sign}(g). When they flip sign, m^≈0\hat m\approx0 and Adam barely moves. The paper's bound for a single step is:

∣Δθt∣  ≲  η⋅max⁡ ⁣(1, 1−β11−β2)  =  3.16 η  at the defaults|\Delta\theta_t|\;\lesssim\;\eta\cdot\max\!\left(1,\ \frac{1-\beta_1}{\sqrt{1-\beta_2}}\right)\;=\;3.16\,\eta\ \text{ at the defaults}
(1−β1)/1−β2(1-\beta_1)/\sqrt{1-\beta_2}
reached only in the most extreme sparse case: a long run of zero gradients, then a single nonzero one
η\eta
the bound in the common case. Kingma and Ba read η\eta as a trust region: you can choose it from how far parameters may sensibly move

This is Adam's real selling point. The step size is set in parameter units by η\eta. It is not set by the gradient's scale, which varies by orders of magnitude between layers, losses and stages of training. With β1=0\beta_1=0 and β2→0\beta_2\to0 the update is exactly η sign(g)\eta\,\mathrm{sign}(g), i.e. signSGD. Balles and Hennig (2018) analyse Adam as a sign method whose step is damped by the variance. The cost of this behaviour is that the step does not shrink as the gradient shrinks near a minimum. It only shrinks as the gradients start to disagree, which is why Adam is normally paired with a learning-rate schedule.

f = S · ½(x² + 25y²), S = 1startminloss ÷ S (log) vs step1e-81e-41|Δθ|/η per coordinate0136x y0100200
t = 200
● Adam loss ÷ S 3.4e-6
┄ GD, same η loss ÷ S 9.8e-9
m̂ = (−0.008, −4.1e-4) · √v̂ = (1.502, 9.288)
Δθ/η = (0.005, 4.4e-5) · max so far 1.00
ε = 1e-8 sits next to √v̂: once gradients shrink to about ε (S ≤ 1e-8) Adam stops being scale-free. Worst-case bound on |Δθ|/η for these betas: 3.16. GD is stable only for η < 2/(25·S) = 0.080.
fig.Adam (blue) and plain gradient descent at the same learning rate (dashed) on S · ½(x² + 25y²) from (−4, 1.6), 200 steps at lr 0.05 and default betas. Adam's first step is exactly 0.05 in each coordinate, and the lower panel shows |Δθ|/η never exceeding 1.00. It reaches 0.11 at step 100 and 3.4e-6 at step 200. A well-tuned GD (9.8e-9) is faster on this smooth problem: Adam's advantage is robustness, not speed. Move the loss scale S anywhere from 1e-4 to 1e4 and Adam's trajectory does not change at all, while GD freezes (S = 1e-2: loss 6.8 after 200 steps) or diverges (S ≥ 10). Only below S = 1e-8, where √v̂ drops toward ε = 1e-8, does Adam slow down (0.0086 at S = 1e-8, 38.4 at 1e-12). Turn bias correction off: the first step becomes 3.16η and the step grows to 6.0η by step 6.

Where ε goes, and when it matters

The ε is there to avoid dividing by zero, but it also sets a floor. When v^≪ϵ\sqrt{\hat v}\ll\epsilon the update becomes η m^/ϵ\eta\,\hat m/\epsilon: plain momentum SGD with learning rate η/ϵ\eta/\epsilon, and the scale-invariance is gone. For a model whose gradients are near 10−810^{-8} that is the regime it trains in (in the figure, S=10−10S=10^{-10} stalls at a loss of 7.0). Some recipes raise ε deliberately. RoBERTa used ϵ=10−6\epsilon=10^{-6}, and TensorFlow's documentation suggests 1.0 or 0.1 for Inception on ImageNet. That trades adaptivity for stability and makes Adam more SGD-like.

Implementations put ε in different places. PyTorch follows the paper's Algorithm 1 and adds ε to the bias-corrected v^t\sqrt{\hat v_t}. TensorFlow/Keras uses the paper's "efficient" form ηt=η1−β2t/(1−β1t)\eta_t=\eta\sqrt{1-\beta_2^t}/(1-\beta_1^t) and adds its ϵ^\hat\epsilon (default 10−710^{-7}) to the uncorrected vt\sqrt{v_t}. The two are equivalent only if ϵ^=ϵ1−β2t\hat\epsilon=\epsilon\sqrt{1-\beta_2^t}, so a given ε is effectively larger early in training in Keras. If you port a recipe, port ε together with where it goes.

Worked example

step tm^t\hat m_tv^t\sqrt{\hat v_t}Δθt/η\Delta\theta_t/\etaθt\theta_tloss
1(−4.000, 40.00)(4.000, 40.00)(1.000, −1.000)(−3.950, 1.550)37.83
2(−3.974, 39.34)(3.975, 39.38)(0.9997, −0.9991)(−3.900, 1.500)35.73
3(−3.947, 38.66)(3.950, 38.76)(0.9991, −0.9974)(−3.850, 1.450)33.70
The figure's valley at S = 1, start (−4, 1.6)(-4,\,1.6), gradient (−4, 40)(-4,\,40), η=0.05\eta=0.05, default betas (computed in Node)

Four facts in four lines

  1. 1Bias correction. Constant g=2g=2: m1=0.2m_1=0.2, v1=0.004v_1=0.004, so m^1=0.2/0.1=2\hat m_1=0.2/0.1=2, v^1=0.004/0.001=4\hat v_1=0.004/0.001=4, and the step is η⋅2/2=η\eta\cdot2/2=\eta. Uncorrected: 0.2/0.004=3.160.2/\sqrt{0.004}=3.16, then 4.254.25 and 4.954.95 on steps 2 and 3.
  2. 2Scale-free. The table's x gradient is 4 and its y gradient is 40, yet both coordinates moved by exactly η=0.05\eta=0.05. GD at the same η\eta would have moved y ten times further than x.
  3. 3Noise shrinks the step. A gradient alternating +1,−1,+1,…+1,-1,+1,\dots gives v^→1\hat v\to1 and ∣m^∣→(1−β1)/(1+β1)=0.053|\hat m|\to(1-\beta_1)/(1+\beta_1)=0.053, so after many steps Adam moves only 0.053 η0.053\,\eta per step.
  4. 4The worst case. After 10,000 steps of exactly zero gradient, a single gradient of any size produces a step of 3.16η3.16\eta, the bound (1−β1)/1−β2(1-\beta_1)/\sqrt{1-\beta_2}. After only 1000 zero steps it is 2.52η2.52\eta, because v^\hat v's correction has not yet faded.

Memory: twice the parameters

Adam stores mm and vv for every parameter. In fp32 that is 8 bytes per parameter on top of the weights: about 56 GB of optimizer state for a 7-billion-parameter model, before activations, gradients or a mixed-precision master copy. That is why large-scale training shards optimizer state across GPUs (ZeRO, FSDP), and why memory-saving variants exist. 8-bit Adam (Dettmers et al., 2021) quantizes the states block-wise. Adafactor (Shazeer and Stern, 2018) stores a factored, row-and-column version of vv. SGD with momentum needs half the state, and plain SGD needs none.

L2 regularization is not weight decay in Adam

torch.optim.Adam(..., weight_decay=λ) adds λθ\lambda\theta to the gradient before the moments are computed. That decay term is then divided by v^\sqrt{\hat v} like everything else. A weight with large, frequent gradients is barely decayed, and a weight with tiny gradients is decayed strongly. That is not what weight decay is meant to do, and it is not equivalent to the decoupled decay θ←θ−ηλθ\theta\leftarrow\theta-\eta\lambda\theta that SGD's L2 amounts to. Loshchilov and Hutter's fix is to apply the decay directly to the weights, outside the adaptive step. That is AdamW (see its article), and it is what torch.optim.AdamW implements.

python
1import torch
2
3opt = torch.optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999), eps=1e-8)
4# Weight decay: use AdamW (decoupled), not Adam(weight_decay=...) (coupled L2).
5opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
6
7# The figure's valley: f = 0.5 * (x^2 + 25 y^2)
8p = torch.tensor([-4.0, 1.6], requires_grad=True)
9opt = torch.optim.Adam([p], lr=0.05)
10for t in range(200):
11 opt.zero_grad()
12 loss = 0.5 * (p[0] ** 2 + 25 * p[1] ** 2)
13 loss.backward()
14 opt.step()
15 if t == 0:
16 print(p.tolist()) # about [-3.95, 1.55]: exactly lr per coordinate
17print(opt.state[p].keys()) # dict_keys(['step', 'exp_avg', 'exp_avg_sq'])
18
19# The update, written out (element-wise), as in torch.optim.Adam
20@torch.no_grad()
21def adam_step(p, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8):
22 m.mul_(b1).add_(g, alpha=1 - b1)
23 v.mul_(b2).addcmul_(g, g, value=1 - b2)
24 m_hat = m / (1 - b1 ** t)
25 v_hat = v / (1 - b2 ** t)
26 p.sub_(lr * m_hat / (v_hat.sqrt() + eps)) # eps outside the sqrt
Adam in PyTorch, reproducing the table, and the rule written out

Use when

  • +Step of about η per coordinate whatever the gradient scale: one learning rate works across layers and losses
  • +Bias correction makes the first steps well-sized
  • +Momentum smooths noisy gradients; steps shrink automatically where gradients disagree
  • +Strong defaults: η=10−3\eta=10^{-3} or 3×10−43\times10^{-4} and β=(0.9,0.999)\beta=(0.9,0.999) often work without tuning

Avoid when

  • −Two extra state tensors: optimizer memory is 2×2\times the parameters
  • −Built-in weight_decay is coupled L2, not true weight decay (use AdamW)
  • −The step does not shrink near a minimum, and ϵ\epsilon caps adaptivity for tiny gradients: it needs a schedule
  • −Can fail to converge in contrived cases, and sometimes generalizes worse than tuned SGD on vision tasks

Check yourself

You multiply your loss by 1000 (e.g. sum instead of mean over a batch of 1000) and keep Adam's settings. What happens to the updates?

Check yourself

Without bias correction, how large is Adam's very first step (β₁ = 0.9, β₂ = 0.999, ε negligible)?

Remember

  • ✓mm = EMA of gg, vv = EMA of g2g^2, each divided by 1−βt1-\beta^t; step −η m^/(v^+ϵ)-\eta\,\hat m/(\sqrt{\hat v}+\epsilon).
  • ✓Bias correction comes from E[vt]=E[g2](1−β2t)\mathbb E[v_t]=\mathbb E[g^2](1-\beta_2^t); without it the first step is 3.16η3.16\eta.
  • ✓The step is about η\eta per coordinate (sign-like), at most about 3.16η3.16\eta at the defaults, and independent of the loss's scale while v^≫ϵ\sqrt{\hat v}\gg\epsilon.
  • ✓Defaults: η=10−3\eta=10^{-3}, β=(0.9,0.999)\beta=(0.9,0.999), ϵ=10−8\epsilon=10^{-8} outside the square root (PyTorch); Keras's ϵ^=10−7\hat\epsilon=10^{-7} sits elsewhere.
  • ✓Memory: two state tensors, twice the parameters.
  • ✓Adam(weight_decay) is coupled L2; use AdamW for decoupled weight decay.
← back to Deep Learning roadmap