Definition
AMSGrad is a variant of Adam that keeps the element-wise maximum of all second-moment estimates seen so far, , and divides by . The effective step size of each parameter can therefore only shrink.
Adam's convergence proof (Kingma and Ba, 2015) was taken on trust for three years. Sashank Reddi, Satyen Kale and Sanjiv Kumar, in "On the Convergence of Adam and Beyond" (ICLR 2018, one of the conference's best paper awards), showed that it has a gap: the analysis needs the effective step size to be non-increasing, and an exponential moving average of squared gradients does not guarantee that. They built simple convex problems on which Adam converges to the worst possible point, and proposed AMSGrad as a minimal fix.
The failure is not specific to Adam. Any method that divides by an exponential moving average of with a short memory, including RMSProp (Adam with ), AdaDelta and Nadam, is affected. AdaGrad is not: it divides by a running sum, which can only grow.
The update rule
- running maximum of , per parameter; it never decreases
- step size; the theory uses , practice a constant or a schedule
- projection onto the feasible set (the online-learning setting of the paper; absent in deep learning)
- the effective per-parameter step: non-increasing in t, which is what the proof needs
The paper writes the algorithm without bias correction. PyTorch's amsgrad=True keeps the maximum of the uncorrected and then applies the usual corrections: . The cost is one more state tensor per parameter: three instead of Adam's two.
At a glance
- Paper
- Reddi, Kale & Kumar, "On the Convergence of Adam and Beyond", ICLR 2018
- PyTorch
torch.optim.Adam(..., amsgrad=True)ortorch.optim.AdamW(..., amsgrad=True); statemax_exp_avg_sq- Guarantee
- regret for online convex problems, so average regret
- Memory
- 3 tensors per parameter (, , ) vs 2 for Adam
- In practice
- rarely better than tuned Adam/AdamW; not a default anywhere major
The counterexample
Online convex optimization: at each step an adversary reveals a loss , the optimizer picks from and pays . The paper's Theorem 1 uses linear losses with period 3:
- the rare large gradient; the figure uses
- the minimizer of the summed loss, because its slope is positive
- regret ; a good algorithm has
With and , Adam's forgets almost immediately. On the step with gradient , jumps up and the step is normalized to about . On the next two steps, gradient , collapses back toward 1 and each step is again close to , now in the wrong direction. Two wrong-way steps beat one right-way step, so drifts up to , the point with the highest loss. The rare large gradient carries the information, and Adam's normalization throws it away.
AMSGrad remembers the large , so the two small gradients are divided by and produce small steps. The large gradient then wins and goes to . Theorem 2 of the paper extends the construction to any , and Theorem 3 gives a stochastic version, so tuning cannot rule the failure out in general.
Why the proof needs a non-increasing step
The regret analysis of adaptive methods telescopes a sum of terms . Each bracket must be non-negative for the sum to be bounded by its last term. Reddi et al. call the bracket and note that it is always for SGD and AdaGrad, but can be negative for Adam and RMSProp whenever drops faster than grows, which is exactly what happens after a rare large gradient. AMSGrad forces and , so by construction.
The fix is the smallest one that restores the property while keeping Adam's adaptivity: unlike AdaGrad, does not grow with every gradient, only when a gradient larger than anything recent arrives. The paper also proposes AdamNC, which instead lets grow toward 1 over time, so becomes an ever-longer average.
Worked example: the first two periods
| t | g_t | v_t | Adam step | Adam x_t | max v | AMSGrad step | AMSGrad x_t |
|---|---|---|---|---|---|---|---|
| 1 | 3 | 8.100 | −0.1054 | −0.1054 | 8.100 | −0.1054 | −0.1054 |
| 2 | −1 | 1.710 | +0.0541 | −0.0513 | 8.100 | +0.0248 | −0.0806 |
| 3 | −1 | 1.071 | +0.0558 | +0.0045 | 8.100 | +0.0203 | −0.0603 |
| 4 | 3 | 8.207 | −0.0524 | −0.0479 | 8.207 | −0.0524 | −0.1126 |
| 5 | −1 | 1.721 | +0.0341 | −0.0138 | 8.207 | +0.0156 | −0.0970 |
| 6 | −1 | 1.072 | +0.0394 | +0.0256 | 8.207 | +0.0143 | −0.0828 |
- 1Step 1: and the step is toward for both methods.
- 2Step 2: the gradient is . Adam's , so the step is . AMSGrad divides by instead: .
- 3After one period Adam has moved up by ; AMSGrad has moved down by . After two periods: versus . Repeat this a few thousand times and Adam sits at .
- 4At each period costs while costs , so Adam's regret grows by 2 per period: . That is the 0.667 in the figure.
1import math, torch23opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01, amsgrad=True)4# or torch.optim.Adam(model.parameters(), lr=1e-3, amsgrad=True)56def online(amsgrad, C=3.0, beta2=0.1, alpha=0.1, T=100_000):7 x, v, vmax = 0.0, 0.0, 0.08 for t in range(1, T + 1):9 g = C if t % 3 == 1 else -1.0 # beta1 = 0, so m_t = g_t10 v = beta2 * v + (1 - beta2) * g * g11 vmax = max(vmax, v)12 step = alpha / math.sqrt(t) * g / math.sqrt(vmax if amsgrad else v)13 x = min(1.0, max(-1.0, x - step)) # projection onto [-1, 1]14 return x1516print(online(False), online(True)) # ~ 1.0 (worst point) -1.0 (optimum)Use when
- +Restores a convergence guarantee: regret on online convex problems
- +One-line change; available as
amsgrad=Truein Adam and AdamW - +Effective step size is monotone, so no sudden step growth after a rare large gradient
Avoid when
- −Usually no measurable improvement over Adam on real deep-learning tasks
- −One extra state tensor per parameter (50% more optimizer memory than Adam)
- −A single early spike can permanently shrink a parameter's step
- −The theory uses and a convex loss; neither holds in deep learning
Check yourself
In the Theorem 1 problem (gradient 3 once, then −1 twice, repeating), why does Adam with β2 = 0.1 drift to x = +1?
Check yourself
What does AMSGrad store that Adam does not?
Remember
- ✓AMSGrad divides by instead of : each parameter's effective step can only shrink.
- ✓Reddi et al. (2018) showed Adam and RMSProp can converge to the worst point of a simple convex problem, because an EMA of lets the step size grow after a rare large gradient.
- ✓The fix restores the condition that the regret proof needs and gives regret.
- ✓PyTorch:
Adam(amsgrad=True)/AdamW(amsgrad=True); one more state tensor per parameter. - ✓Empirically the gains are small; a large (default 0.999) already avoids most of the problem.