AMSGrad

Adam with one change: divide by the running maximum of v instead of v itself, so the per-parameter step size can never grow. That repairs the convergence proof Reddi et al. showed Adam lacks, but in practice the gains over Adam are usually small.

11 min read879 wordsadvanced0 viewsupdated 2026-10-11

Definition

AMSGrad is a variant of Adam that keeps the element-wise maximum of all second-moment estimates seen so far, v^t=max⁡(v^t−1,vt)\hat v_t=\max(\hat v_{t-1},v_t), and divides by v^t\sqrt{\hat v_t}. The effective step size η/v^t\eta/\sqrt{\hat v_t} of each parameter can therefore only shrink.

Adam's convergence proof (Kingma and Ba, 2015) was taken on trust for three years. Sashank Reddi, Satyen Kale and Sanjiv Kumar, in "On the Convergence of Adam and Beyond" (ICLR 2018, one of the conference's best paper awards), showed that it has a gap: the analysis needs the effective step size to be non-increasing, and an exponential moving average of squared gradients does not guarantee that. They built simple convex problems on which Adam converges to the worst possible point, and proposed AMSGrad as a minimal fix.

The failure is not specific to Adam. Any method that divides by an exponential moving average of g2g^2 with a short memory, including RMSProp (Adam with β1=0\beta_1=0), AdaDelta and Nadam, is affected. AdaGrad is not: it divides by a running sum, which can only grow.

The update rule

mt=β1mt−1+(1−β1) gtvt=β2vt−1+(1−β2) gt2v^t=max⁡(v^t−1, vt)θt=ΠF ⁣(θt−1−αt mtv^t)\begin{aligned}m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2\\ \hat v_t&=\max(\hat v_{t-1},\,v_t)\\ \theta_{t}&=\Pi_{\mathcal F}\!\left(\theta_{t-1}-\alpha_t\,\frac{m_t}{\sqrt{\hat v_t}}\right)\end{aligned}
v^t\hat v_t
running maximum of vv, per parameter; it never decreases
αt\alpha_t
step size; the theory uses α/t\alpha/\sqrt t, practice a constant or a schedule
ΠF\Pi_{\mathcal F}
projection onto the feasible set F\mathcal F (the online-learning setting of the paper; absent in deep learning)
αt/v^t\alpha_t/\sqrt{\hat v_t}
the effective per-parameter step: non-increasing in t, which is what the proof needs

The paper writes the algorithm without bias correction. PyTorch's amsgrad=True keeps the maximum of the uncorrected vv and then applies the usual corrections: θ←θ−η1−β1t mt/(max⁡s≤tvs/1−β2t+ϵ)\theta\leftarrow\theta-\frac{\eta}{1-\beta_1^t}\,m_t\big/\big(\sqrt{\max_{s\le t}v_s}/\sqrt{1-\beta_2^t}+\epsilon\big). The cost is one more state tensor per parameter: three instead of Adam's two.

At a glance

Paper
Reddi, Kale & Kumar, "On the Convergence of Adam and Beyond", ICLR 2018
PyTorch
torch.optim.Adam(..., amsgrad=True) or torch.optim.AdamW(..., amsgrad=True); state max_exp_avg_sq
Guarantee
O(T)O(\sqrt T) regret for online convex problems, so average regret RT/T→0R_T/T\to0
Memory
3 tensors per parameter (mm, vv, max⁡v\max v) vs 2 for Adam
In practice
rarely better than tuned Adam/AdamW; not a default anywhere major

The counterexample

Online convex optimization: at each step tt an adversary reveals a loss ftf_t, the optimizer picks xtx_t from F=[−1,1]\mathcal F=[-1,1] and pays ft(xt)f_t(x_t). The paper's Theorem 1 uses linear losses with period 3:

ft(x)={C xt mod 3=1−xotherwiseC>2,∑one periodft(x)=(C−2) xf_t(x)=\begin{cases}C\,x & t\bmod 3=1\\ -x & \text{otherwise}\end{cases}\qquad C>2,\qquad \sum_{\text{one period}}f_t(x)=(C-2)\,x
CC
the rare large gradient; the figure uses C=3C=3
x∗=−1x^*=-1
the minimizer of the summed loss, because its slope C−2C-2 is positive
RTR_T
regret ∑t≤Tft(xt)−ft(x∗)\sum_{t\le T}f_t(x_t)-f_t(x^*); a good algorithm has RT/T→0R_T/T\to0

With β1=0\beta_1=0 and β2=1/(1+C2)\beta_2=1/(1+C^2), Adam's vtv_t forgets almost immediately. On the step with gradient CC, vv jumps up and the step αtC/vt\alpha_t C/\sqrt{v_t} is normalized to about αt\alpha_t. On the next two steps, gradient −1-1, vv collapses back toward 1 and each step is again close to αt\alpha_t, now in the wrong direction. Two wrong-way steps beat one right-way step, so xx drifts up to +1+1, the point with the highest loss. The rare large gradient carries the information, and Adam's normalization throws it away.

AMSGrad remembers the large vv, so the two small gradients are divided by max⁡v\sqrt{\max v} and produce small steps. The large gradient then wins and xx goes to −1-1. Theorem 2 of the paper extends the construction to any β1<β2\beta_1<\sqrt{\beta_2}, and Theorem 3 gives a stochastic version, so tuning β\beta cannot rule the failure out in general.

iterate x_t on F = [−1, 1] · optimum x* = −1 · per period loss = 1·x−101worst pointoptimumeffective step α_t/√v (Adam: band = min…max per bin) vs α_t/√v̂ (AMSGrad), log scale1e-61e-41e-2111e11e21e31e41e51e6step t (log scale)━ Adam x = 1.000, R_t/t = 0.667 · ━ AMSGrad x = −1.000, R_t/t = 0.000sitting at x = +1 costs 0.667 regret per step forever; at −1 it costs 0
t = 1,000,000β1 = 0 · Adam α_t/√v ∈ [3.5e−5, 9.9e−5] · AMSGrad α_t/√v̂ 3.5e−5
fig.The paper's online problems on [−1, 1], with α_t = α/√t for 10⁶ steps (log-scaled time axis). Theorem 1 preset (3x, −x, −x; β1 = 0, β2 = 0.1, α = 0.1): Adam (orange) reaches x = 0.997 by step 1,000 and stays at the worst point, with average regret 0.667 per step at t = 10⁶. AMSGrad (blue) reaches −1 by step 10⁴ and its average regret falls to 0.019, then 0.002 at 10⁵. Below, Adam's effective step jumps by a factor of about 3 inside every period; AMSGrad's only decreases. Raise β2 to 0.5 and Adam converges too. The §5 preset (1010x once per 101 steps, −10x otherwise; β1 = 0.9, β2 = 0.99, α = 0.5) is the paper's synthetic experiment: Adam ends at +1 for any β2 up to 0.999 and drifts the right way only at 0.9999; AMSGrad reaches −0.984 by 10⁵ steps and −0.997 by 10⁶.

Why the proof needs a non-increasing step

The regret analysis of adaptive methods telescopes a sum of terms ∥xt−x∗∥2(v^t/αt−v^t−1/αt−1)\|x_t-x^*\|^2\big(\sqrt{\hat v_{t}}/\alpha_t-\sqrt{\hat v_{t-1}}/\alpha_{t-1}\big). Each bracket must be non-negative for the sum to be bounded by its last term. Reddi et al. call the bracket Γt\Gamma_t and note that it is always ≥0\ge0 for SGD and AdaGrad, but can be negative for Adam and RMSProp whenever vv drops faster than t\sqrt t grows, which is exactly what happens after a rare large gradient. AMSGrad forces v^t≥v^t−1\hat v_t\ge\hat v_{t-1} and αt≤αt−1\alpha_t\le\alpha_{t-1}, so Γt≥0\Gamma_t\ge0 by construction.

The fix is the smallest one that restores the property while keeping Adam's adaptivity: unlike AdaGrad, max⁡v\max v does not grow with every gradient, only when a gradient larger than anything recent arrives. The paper also proposes AdamNC, which instead lets β2\beta_2 grow toward 1 over time, so vv becomes an ever-longer average.

Worked example: the first two periods

tg_tv_tAdam stepAdam x_tmax vAMSGrad stepAMSGrad x_t
138.100−0.1054−0.10548.100−0.1054−0.1054
2−11.710+0.0541−0.05138.100+0.0248−0.0806
3−11.071+0.0558+0.00458.100+0.0203−0.0603
438.207−0.0524−0.04798.207−0.0524−0.1126
5−11.721+0.0341−0.01388.207+0.0156−0.0970
6−11.072+0.0394+0.02568.207+0.0143−0.0828
Theorem 1 problem with C=3C=3, β1=0\beta_1=0, β2=0.1\beta_2=0.1, αt=0.1/t\alpha_t=0.1/\sqrt t, x0=0x_0=0 (computed with the figure's code)
  1. 1Step 1: v1=0.9⋅32=8.1v_1=0.9\cdot3^2=8.1 and the step is 0.1⋅3/8.1=0.10540.1\cdot3/\sqrt{8.1}=0.1054 toward −1-1 for both methods.
  2. 2Step 2: the gradient is −1-1. Adam's v2=0.1⋅8.1+0.9=1.71v_2=0.1\cdot8.1+0.9=1.71, so the step is 0.12⋅1/1.71=0.0541\tfrac{0.1}{\sqrt2}\cdot1/\sqrt{1.71}=0.0541. AMSGrad divides by 8.1\sqrt{8.1} instead: 0.02480.0248.
  3. 3After one period Adam has moved up by 0.00450.0045; AMSGrad has moved down by 0.06030.0603. After two periods: +0.0256+0.0256 versus −0.0828-0.0828. Repeat this a few thousand times and Adam sits at +1+1.
  4. 4At x=+1x=+1 each period costs (C−2)⋅1=1(C-2)\cdot1=1 while x∗=−1x^*=-1 costs −1-1, so Adam's regret grows by 2 per period: RT/T→2/3R_T/T\to2/3. That is the 0.667 in the figure.
python
1import math, torch
2
3opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01, amsgrad=True)
4# or torch.optim.Adam(model.parameters(), lr=1e-3, amsgrad=True)
5
6def online(amsgrad, C=3.0, beta2=0.1, alpha=0.1, T=100_000):
7 x, v, vmax = 0.0, 0.0, 0.0
8 for t in range(1, T + 1):
9 g = C if t % 3 == 1 else -1.0 # beta1 = 0, so m_t = g_t
10 v = beta2 * v + (1 - beta2) * g * g
11 vmax = max(vmax, v)
12 step = alpha / math.sqrt(t) * g / math.sqrt(vmax if amsgrad else v)
13 x = min(1.0, max(-1.0, x - step)) # projection onto [-1, 1]
14 return x
15
16print(online(False), online(True)) # ~ 1.0 (worst point) -1.0 (optimum)
AMSGrad in PyTorch, and the Theorem 1 problem run by hand

Use when

  • +Restores a convergence guarantee: O(T)O(\sqrt T) regret on online convex problems
  • +One-line change; available as amsgrad=True in Adam and AdamW
  • +Effective step size is monotone, so no sudden step growth after a rare large gradient

Avoid when

  • −Usually no measurable improvement over Adam on real deep-learning tasks
  • −One extra state tensor per parameter (50% more optimizer memory than Adam)
  • −A single early spike can permanently shrink a parameter's step
  • −The theory uses αt=α/t\alpha_t=\alpha/\sqrt t and a convex loss; neither holds in deep learning

Check yourself

In the Theorem 1 problem (gradient 3 once, then −1 twice, repeating), why does Adam with β2 = 0.1 drift to x = +1?

Check yourself

What does AMSGrad store that Adam does not?

Remember

  • ✓AMSGrad divides by max⁡s≤tvs\sqrt{\max_{s\le t}v_s} instead of vt\sqrt{v_t}: each parameter's effective step can only shrink.
  • ✓Reddi et al. (2018) showed Adam and RMSProp can converge to the worst point of a simple convex problem, because an EMA of g2g^2 lets the step size grow after a rare large gradient.
  • ✓The fix restores the condition Γt≥0\Gamma_t\ge0 that the regret proof needs and gives O(T)O(\sqrt T) regret.
  • ✓PyTorch: Adam(amsgrad=True) / AdamW(amsgrad=True); one more state tensor per parameter.
  • ✓Empirically the gains are small; a large β2\beta_2 (default 0.999) already avoids most of the problem.
← back to Deep Learning roadmap