# AMSGrad

> Adam with one change: divide by the running maximum of v instead of v itself, so the per-parameter step size can never grow. That repairs the convergence proof Reddi et al. showed Adam lacks, but in practice the gains over Adam are usually small.

Source: https://www.mlfoundry.in/learn/amsgrad  
Section: Deep Learning › Optimization Algorithms  
Level: advanced · 11 min read · updated 2026-10-11

**AMSGrad:** is a variant of Adam that keeps the element-wise **maximum** of all second-moment estimates seen so far, $\hat v_t=\max(\hat v_{t-1},v_t)$, and divides by $\sqrt{\hat v_t}$. The effective step size $\eta/\sqrt{\hat v_t}$ of each parameter can therefore only shrink.

Adam's convergence proof (Kingma and Ba, 2015) was taken on trust for three years. Sashank Reddi, Satyen Kale and Sanjiv Kumar, in "On the Convergence of Adam and Beyond" (ICLR 2018, one of the conference's best paper awards), showed that it has a gap: the analysis needs the effective step size to be non-increasing, and an exponential moving average of squared gradients does not guarantee that. They built simple **convex** problems on which Adam converges to the worst possible point, and proposed AMSGrad as a minimal fix.

The failure is not specific to Adam. Any method that divides by an exponential moving average of $g^2$ with a short memory, including RMSProp (Adam with $\beta_1=0$), AdaDelta and Nadam, is affected. AdaGrad is not: it divides by a running **sum**, which can only grow.

## The update rule

$$
\begin{aligned}m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2\\ \hat v_t&=\max(\hat v_{t-1},\,v_t)\\ \theta_{t}&=\Pi_{\mathcal F}\!\left(\theta_{t-1}-\alpha_t\,\frac{m_t}{\sqrt{\hat v_t}}\right)\end{aligned}
$$

- $\hat v_t$: running maximum of $v$, per parameter; it never decreases
- $\alpha_t$: step size; the theory uses $\alpha/\sqrt t$, practice a constant or a schedule
- $\Pi_{\mathcal F}$: projection onto the feasible set $\mathcal F$ (the online-learning setting of the paper; absent in deep learning)
- $\alpha_t/\sqrt{\hat v_t}$: the effective per-parameter step: non-increasing in t, which is what the proof needs

The paper writes the algorithm without bias correction. PyTorch's `amsgrad=True` keeps the maximum of the **uncorrected** $v$ and then applies the usual corrections: $\theta\leftarrow\theta-\frac{\eta}{1-\beta_1^t}\,m_t\big/\big(\sqrt{\max_{s\le t}v_s}/\sqrt{1-\beta_2^t}+\epsilon\big)$. The cost is one more state tensor per parameter: three instead of Adam's two.

**At a glance**

- **Paper:** Reddi, Kale & Kumar, "On the Convergence of Adam and Beyond", ICLR 2018
- **PyTorch:** `torch.optim.Adam(..., amsgrad=True)` or `torch.optim.AdamW(..., amsgrad=True)`; state `max_exp_avg_sq`
- **Guarantee:** $O(\sqrt T)$ regret for online convex problems, so average regret $R_T/T\to0$
- **Memory:** 3 tensors per parameter ($m$, $v$, $\max v$) vs 2 for Adam
- **In practice:** rarely better than tuned Adam/AdamW; not a default anywhere major

## The counterexample

Online convex optimization: at each step $t$ an adversary reveals a loss $f_t$, the optimizer picks $x_t$ from $\mathcal F=[-1,1]$ and pays $f_t(x_t)$. The paper's Theorem 1 uses linear losses with period 3:

$$
f_t(x)=\begin{cases}C\,x & t\bmod 3=1\\ -x & \text{otherwise}\end{cases}\qquad C>2,\qquad \sum_{\text{one period}}f_t(x)=(C-2)\,x
$$

- $C$: the rare large gradient; the figure uses $C=3$
- $x^*=-1$: the minimizer of the summed loss, because its slope $C-2$ is positive
- $R_T$: regret $\sum_{t\le T}f_t(x_t)-f_t(x^*)$; a good algorithm has $R_T/T\to0$

With $\beta_1=0$ and $\beta_2=1/(1+C^2)$, Adam's $v_t$ forgets almost immediately. On the step with gradient $C$, $v$ jumps up and the step $\alpha_t C/\sqrt{v_t}$ is normalized to about $\alpha_t$. On the next two steps, gradient $-1$, $v$ collapses back toward 1 and each step is again close to $\alpha_t$, now in the **wrong** direction. Two wrong-way steps beat one right-way step, so $x$ drifts up to $+1$, the point with the **highest** loss. The rare large gradient carries the information, and Adam's normalization throws it away.

AMSGrad remembers the large $v$, so the two small gradients are divided by $\sqrt{\max v}$ and produce small steps. The large gradient then wins and $x$ goes to $-1$. Theorem 2 of the paper extends the construction to **any** $\beta_1<\sqrt{\beta_2}$, and Theorem 3 gives a stochastic version, so tuning $\beta$ cannot rule the failure out in general.

> Figure: The paper's online problems on [−1, 1], with α_t = α/√t for 10⁶ steps (log-scaled time axis). Theorem 1 preset (3x, −x, −x; β1 = 0, β2 = 0.1, α = 0.1): Adam (orange) reaches x = 0.997 by step 1,000 and stays at the worst point, with average regret 0.667 per step at t = 10⁶. AMSGrad (blue) reaches −1 by step 10⁴ and its average regret falls to 0.019, then 0.002 at 10⁵. Below, Adam's effective step jumps by a factor of about 3 inside every period; AMSGrad's only decreases. Raise β2 to 0.5 and Adam converges too. The §5 preset (1010x once per 101 steps, −10x otherwise; β1 = 0.9, β2 = 0.99, α = 0.5) is the paper's synthetic experiment: Adam ends at +1 for any β2 up to 0.999 and drifts the right way only at 0.9999; AMSGrad reaches −0.984 by 10⁵ steps and −0.997 by 10⁶.

## Why the proof needs a non-increasing step

The regret analysis of adaptive methods telescopes a sum of terms $\|x_t-x^*\|^2\big(\sqrt{\hat v_{t}}/\alpha_t-\sqrt{\hat v_{t-1}}/\alpha_{t-1}\big)$. Each bracket must be non-negative for the sum to be bounded by its last term. Reddi et al. call the bracket $\Gamma_t$ and note that it is always $\ge0$ for SGD and AdaGrad, but can be negative for Adam and RMSProp whenever $v$ drops faster than $\sqrt t$ grows, which is exactly what happens after a rare large gradient. AMSGrad forces $\hat v_t\ge\hat v_{t-1}$ and $\alpha_t\le\alpha_{t-1}$, so $\Gamma_t\ge0$ by construction.

The fix is the smallest one that restores the property while keeping Adam's adaptivity: unlike AdaGrad, $\max v$ does not grow with every gradient, only when a gradient larger than anything recent arrives. The paper also proposes **AdamNC**, which instead lets $\beta_2$ grow toward 1 over time, so $v$ becomes an ever-longer average.

## Worked example: the first two periods

Theorem 1 problem with $C=3$, $\beta_1=0$, $\beta_2=0.1$, $\alpha_t=0.1/\sqrt t$, $x_0=0$ (computed with the figure's code)

| t | g_t | v_t | Adam step | Adam x_t | max v | AMSGrad step | AMSGrad x_t |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 3 | 8.100 | −0.1054 | −0.1054 | 8.100 | −0.1054 | −0.1054 |
| 2 | −1 | 1.710 | +0.0541 | −0.0513 | 8.100 | +0.0248 | −0.0806 |
| 3 | −1 | 1.071 | +0.0558 | +0.0045 | 8.100 | +0.0203 | −0.0603 |
| 4 | 3 | 8.207 | −0.0524 | −0.0479 | 8.207 | −0.0524 | −0.1126 |
| 5 | −1 | 1.721 | +0.0341 | −0.0138 | 8.207 | +0.0156 | −0.0970 |
| 6 | −1 | 1.072 | +0.0394 | +0.0256 | 8.207 | +0.0143 | −0.0828 |

1. Step 1: $v_1=0.9\cdot3^2=8.1$ and the step is $0.1\cdot3/\sqrt{8.1}=0.1054$ toward $-1$ for both methods.
2. Step 2: the gradient is $-1$. Adam's $v_2=0.1\cdot8.1+0.9=1.71$, so the step is $\tfrac{0.1}{\sqrt2}\cdot1/\sqrt{1.71}=0.0541$. AMSGrad divides by $\sqrt{8.1}$ instead: $0.0248$.
3. After one period Adam has moved **up** by $0.0045$; AMSGrad has moved down by $0.0603$. After two periods: $+0.0256$ versus $-0.0828$. Repeat this a few thousand times and Adam sits at $+1$.
4. At $x=+1$ each period costs $(C-2)\cdot1=1$ while $x^*=-1$ costs $-1$, so Adam's regret grows by 2 per period: $R_T/T\to2/3$. That is the 0.667 in the figure.

```python
import math, torch

opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01, amsgrad=True)
# or torch.optim.Adam(model.parameters(), lr=1e-3, amsgrad=True)

def online(amsgrad, C=3.0, beta2=0.1, alpha=0.1, T=100_000):
    x, v, vmax = 0.0, 0.0, 0.0
    for t in range(1, T + 1):
        g = C if t % 3 == 1 else -1.0          # beta1 = 0, so m_t = g_t
        v = beta2 * v + (1 - beta2) * g * g
        vmax = max(vmax, v)
        step = alpha / math.sqrt(t) * g / math.sqrt(vmax if amsgrad else v)
        x = min(1.0, max(-1.0, x - step))      # projection onto [-1, 1]
    return x

print(online(False), online(True))   # ~ 1.0 (worst point)   -1.0 (optimum)
```

AMSGrad in PyTorch, and the Theorem 1 problem run by hand

> **One spike caps the step forever:** Because $\max v$ never decreases, a single huge gradient (a bad batch, a loss spike, the first steps of an un-warmed-up model) permanently shrinks that parameter's step size. Plain Adam recovers after about $1/(1-\beta_2)$ steps; AMSGrad does not. Gradient clipping or warmup matter more, not less, when `amsgrad=True`.

> **Small gains in practice:** Reddi et al. reported modest improvements on logistic regression and small networks (MNIST, CIFAR-10). Later practitioners mostly did not see them: the fast.ai team (Gugger & Howard, 2018) found no benefit over Adam in their experiments, and AMSGrad is not the default in any major training recipe. Zhang et al. (NeurIPS 2022) showed that Adam itself converges when $\beta_2$ is large enough for the problem, which matches the figure: $\beta_2=0.999$ is the default for good reason.

> **What the counterexample really says:** It says Adam's guarantee cannot hold for **every** problem at fixed $(\beta_1,\beta_2)$, not that Adam usually fails. The trouble needs rare, large, informative gradients and a $\beta_2$ whose memory is shorter than the gap between them. Heavy-tailed gradient noise (for example rare tokens in language models) is the realistic version of this setting.

**Strengths**

- Restores a convergence guarantee: $O(\sqrt T)$ regret on online convex problems
- One-line change; available as `amsgrad=True` in Adam and AdamW
- Effective step size is monotone, so no sudden step growth after a rare large gradient

**Limitations**

- Usually no measurable improvement over Adam on real deep-learning tasks
- One extra state tensor per parameter (50% more optimizer memory than Adam)
- A single early spike can permanently shrink a parameter's step
- The theory uses $\alpha_t=\alpha/\sqrt t$ and a convex loss; neither holds in deep learning

**Question:** In the Theorem 1 problem (gradient 3 once, then −1 twice, repeating), why does Adam with β2 = 0.1 drift to x = +1?

- A. Its v forgets the large gradient, so the two small gradients produce steps almost as large as the big one
- B. The momentum term overshoots the optimum
- C. The bias correction is wrong for β1 = 0
- D. The projection onto [−1, 1] pushes it to the boundary

**Answer:** A. Its v forgets the large gradient, so the two small gradients produce steps almost as large as the big one — With such a short memory, $v$ drops from 8.1 to about 1.7 and then 1.07 after the large gradient, so each $-1$ step is nearly as big as the $+3$ step (0.054 and 0.056 versus 0.105 in the first period). Two wrong-way steps outweigh one right-way step. AMSGrad keeps dividing by $\sqrt{8.1}$, and the large gradient wins.

**Question:** What does AMSGrad store that Adam does not?

- A. The previous gradient
- B. The element-wise maximum of v over all steps so far
- C. The maximum of m
- D. A per-parameter learning-rate schedule

**Answer:** B. The element-wise maximum of v over all steps so far — It keeps $\max_{s\le t}v_s$ for every parameter (`max_exp_avg_sq` in PyTorch) and divides by its square root, so the effective step size can only decrease.

**Key points**

- AMSGrad divides by $\sqrt{\max_{s\le t}v_s}$ instead of $\sqrt{v_t}$: each parameter's effective step can only shrink.
- Reddi et al. (2018) showed Adam and RMSProp can converge to the worst point of a simple convex problem, because an EMA of $g^2$ lets the step size grow after a rare large gradient.
- The fix restores the condition $\Gamma_t\ge0$ that the regret proof needs and gives $O(\sqrt T)$ regret.
- PyTorch: `Adam(amsgrad=True)` / `AdamW(amsgrad=True)`; one more state tensor per parameter.
- Empirically the gains are small; a large $\beta_2$ (default 0.999) already avoids most of the problem.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
