Definition
Adam (adaptive moment estimation) keeps two exponential moving averages per parameter: , of the gradient (the first moment), and , of the squared gradient (the second moment). It corrects both for their zero start and steps by . Each parameter gets a step of about in the direction its recent gradients agree on.
Diederik Kingma and Jimmy Ba introduced Adam in Adam: A Method for Stochastic Optimization (arXiv, December 2014; ICLR 2015). It combines the two ideas that came before it. From momentum it takes a smoothed gradient direction, written as an average rather than a sum. From RMSProp (and AdaGrad before it) it takes per-parameter division by the gradient's RMS. Its own contribution is the bias correction that makes the early steps well-behaved, plus a clean analysis of the step size. The same paper also introduced AdaMax, which replaces the RMS with an infinity norm.
Adam became the default optimizer for prototyping, generative models, reinforcement learning and, in its AdamW form, transformers. Its known weak spots led to follow-up work. Reddi, Kale and Kumar (2018) built problems on which Adam does not converge, and proposed AMSGrad. Wilson et al. (2017) found adaptive methods generalizing worse than well-tuned SGD on several benchmarks. Loshchilov and Hutter (2017, ICLR 2019) showed that Adam's L2 regularization is not weight decay, which gave AdamW.
The update rule
- minibatch gradient at step (counting from 1); everything is element-wise
- first and second moment estimates, both initialized to 0 (PyTorch's
exp_avg,exp_avg_sq) - bias-corrected estimates
- learning rate, default
- decay rates, defaults (about 10 steps of memory) and (about 1000 steps)
- default , added to outside the square root
At a glance
- Paper
- Kingma & Ba, arXiv:1412.6980 (2014), ICLR 2015
- Defaults
- , , , (paper and PyTorch)
- State
- and : two extra tensors, i.e. the parameter memory (SGD with momentum stores one)
- Step size
- per coordinate in typical conditions, at most about at the default betas
- Invariance
- multiplying the loss by any constant leaves the updates unchanged, as long as
- PyTorch
torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, amsgrad=False)
Bias correction, derived
Unrolling the recursion from gives a weighted sum of past squared gradients whose weights add up to less than 1:
- a geometric series, equal to : the total weight the average has collected so far
- zero if is stationary, and small if it drifts slowly, because old terms carry little weight
So underestimates by exactly the factor , and dividing by it removes the bias. The same argument applies to with . The correction matters most when is close to 1. At , , so is a thousandth of its target. At the factor is still only . Over the same time the correction () is gone after a few dozen steps.
What happens without it? and are biased by different amounts. At , , so the first step is more than three times the intended size, and it keeps growing for a few steps (up to in the figure's valley). Kingma and Ba point out this is worst exactly when is near 1, which is the setting you want for sparse or noisy gradients.
The step is about η: a trust region and a sign
The ratio is dimensionless, a kind of signal-to-noise ratio for each coordinate. Since estimates , we have , so in expectation the ratio is at most 1 in size. When gradients agree from step to step, and the update is about . When they flip sign, and Adam barely moves. The paper's bound for a single step is:
- reached only in the most extreme sparse case: a long run of zero gradients, then a single nonzero one
- the bound in the common case. Kingma and Ba read as a trust region: you can choose it from how far parameters may sensibly move
This is Adam's real selling point. The step size is set in parameter units by . It is not set by the gradient's scale, which varies by orders of magnitude between layers, losses and stages of training. With and the update is exactly , i.e. signSGD. Balles and Hennig (2018) analyse Adam as a sign method whose step is damped by the variance. The cost of this behaviour is that the step does not shrink as the gradient shrinks near a minimum. It only shrinks as the gradients start to disagree, which is why Adam is normally paired with a learning-rate schedule.
Where ε goes, and when it matters
The ε is there to avoid dividing by zero, but it also sets a floor. When the update becomes : plain momentum SGD with learning rate , and the scale-invariance is gone. For a model whose gradients are near that is the regime it trains in (in the figure, stalls at a loss of 7.0). Some recipes raise ε deliberately. RoBERTa used , and TensorFlow's documentation suggests 1.0 or 0.1 for Inception on ImageNet. That trades adaptivity for stability and makes Adam more SGD-like.
Implementations put ε in different places. PyTorch follows the paper's Algorithm 1 and adds ε to the bias-corrected . TensorFlow/Keras uses the paper's "efficient" form and adds its (default ) to the uncorrected . The two are equivalent only if , so a given ε is effectively larger early in training in Keras. If you port a recipe, port ε together with where it goes.
Worked example
| step t | loss | ||||
|---|---|---|---|---|---|
| 1 | (−4.000, 40.00) | (4.000, 40.00) | (1.000, −1.000) | (−3.950, 1.550) | 37.83 |
| 2 | (−3.974, 39.34) | (3.975, 39.38) | (0.9997, −0.9991) | (−3.900, 1.500) | 35.73 |
| 3 | (−3.947, 38.66) | (3.950, 38.76) | (0.9991, −0.9974) | (−3.850, 1.450) | 33.70 |
Four facts in four lines
- 1Bias correction. Constant : , , so , , and the step is . Uncorrected: , then and on steps 2 and 3.
- 2Scale-free. The table's x gradient is 4 and its y gradient is 40, yet both coordinates moved by exactly . GD at the same would have moved y ten times further than x.
- 3Noise shrinks the step. A gradient alternating gives and , so after many steps Adam moves only per step.
- 4The worst case. After 10,000 steps of exactly zero gradient, a single gradient of any size produces a step of , the bound . After only 1000 zero steps it is , because 's correction has not yet faded.
Memory: twice the parameters
Adam stores and for every parameter. In fp32 that is 8 bytes per parameter on top of the weights: about 56 GB of optimizer state for a 7-billion-parameter model, before activations, gradients or a mixed-precision master copy. That is why large-scale training shards optimizer state across GPUs (ZeRO, FSDP), and why memory-saving variants exist. 8-bit Adam (Dettmers et al., 2021) quantizes the states block-wise. Adafactor (Shazeer and Stern, 2018) stores a factored, row-and-column version of . SGD with momentum needs half the state, and plain SGD needs none.
L2 regularization is not weight decay in Adam
torch.optim.Adam(..., weight_decay=λ) adds to the gradient before the moments are computed. That decay term is then divided by like everything else. A weight with large, frequent gradients is barely decayed, and a weight with tiny gradients is decayed strongly. That is not what weight decay is meant to do, and it is not equivalent to the decoupled decay that SGD's L2 amounts to. Loshchilov and Hutter's fix is to apply the decay directly to the weights, outside the adaptive step. That is AdamW (see its article), and it is what torch.optim.AdamW implements.
1import torch23opt = torch.optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999), eps=1e-8)4# Weight decay: use AdamW (decoupled), not Adam(weight_decay=...) (coupled L2).5opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)67# The figure's valley: f = 0.5 * (x^2 + 25 y^2)8p = torch.tensor([-4.0, 1.6], requires_grad=True)9opt = torch.optim.Adam([p], lr=0.05)10for t in range(200):11 opt.zero_grad()12 loss = 0.5 * (p[0] ** 2 + 25 * p[1] ** 2)13 loss.backward()14 opt.step()15 if t == 0:16 print(p.tolist()) # about [-3.95, 1.55]: exactly lr per coordinate17print(opt.state[p].keys()) # dict_keys(['step', 'exp_avg', 'exp_avg_sq'])1819# The update, written out (element-wise), as in torch.optim.Adam20@torch.no_grad()21def adam_step(p, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8):22 m.mul_(b1).add_(g, alpha=1 - b1)23 v.mul_(b2).addcmul_(g, g, value=1 - b2)24 m_hat = m / (1 - b1 ** t)25 v_hat = v / (1 - b2 ** t)26 p.sub_(lr * m_hat / (v_hat.sqrt() + eps)) # eps outside the sqrtUse when
- +Step of about η per coordinate whatever the gradient scale: one learning rate works across layers and losses
- +Bias correction makes the first steps well-sized
- +Momentum smooths noisy gradients; steps shrink automatically where gradients disagree
- +Strong defaults: or and often work without tuning
Avoid when
- −Two extra state tensors: optimizer memory is the parameters
- −Built-in weight_decay is coupled L2, not true weight decay (use AdamW)
- −The step does not shrink near a minimum, and caps adaptivity for tiny gradients: it needs a schedule
- −Can fail to converge in contrived cases, and sometimes generalizes worse than tuned SGD on vision tasks
Check yourself
You multiply your loss by 1000 (e.g. sum instead of mean over a batch of 1000) and keep Adam's settings. What happens to the updates?
Check yourself
Without bias correction, how large is Adam's very first step (β₁ = 0.9, β₂ = 0.999, ε negligible)?
Remember
- ✓ = EMA of , = EMA of , each divided by ; step .
- ✓Bias correction comes from ; without it the first step is .
- ✓The step is about per coordinate (sign-like), at most about at the defaults, and independent of the loss's scale while .
- ✓Defaults: , , outside the square root (PyTorch); Keras's sits elsewhere.
- ✓Memory: two state tensors, twice the parameters.
- ✓Adam(weight_decay) is coupled L2; use AdamW for decoupled weight decay.