# Nadam

> Adam with Nesterov momentum: the update applies next step's momentum now, so the fresh gradient gets about twice Adam's weight. PyTorch's NAdam also warms the momentum up slowly (momentum_decay), which changes its behaviour over the first thousands of steps.

Source: https://www.mlfoundry.in/learn/nadam  
Section: Deep Learning › Optimization Algorithms  
Level: advanced · 11 min read · updated 2026-10-11

**Nadam:** (Nesterov-accelerated adaptive moment estimation) is Adam with its classical momentum replaced by **Nesterov momentum**. The numerator of the update uses a look-ahead mix, $\mu_{t+1}\hat m_t+(1-\mu_t)\hat g_t$, instead of $\hat m_t$; the denominator $\sqrt{\hat v_t}+\epsilon$ is unchanged.

Adam combines two ideas: momentum for the direction and RMSProp for the scale. By 2015 it was known that for SGD, **Nesterov's accelerated gradient** (Nesterov, 1983; brought to deep learning by Sutskever, Martens, Dahl and Hinton, 2013) is usually a slightly better momentum than the classical heavy ball: it corrects course one step earlier when the velocity points the wrong way. Timothy Dozat's "Incorporating Nesterov Momentum into Adam" (ICLR 2016 workshop track) asked the obvious question, whether the same swap helps Adam, and showed how to do it without computing a gradient at a shifted point.

The result, Nadam, is a drop-in replacement for Adam with the same memory and cost. It is available as `torch.optim.NAdam` and `keras.optimizers.Nadam`. Dozat's experiments showed it converging faster than Adam and the other optimizers he compared; in later practice the improvement is real but modest, and AdamW remains the more common default.

## From Nesterov momentum to Nadam

Nesterov's method evaluates the gradient at the point the momentum is about to carry you to: $g_t=\nabla f(\theta_{t-1}-\eta\mu m_{t-1})$, then $m_t=\mu m_{t-1}+g_t$ and $\theta_t=\theta_{t-1}-\eta m_t$. Dozat's rewrite keeps the gradient at the current point and instead applies the **next** step's momentum early:

$$
g_t=\nabla f(\theta_{t-1}),\qquad m_t=\mu\,m_{t-1}+g_t,\qquad \theta_t=\theta_{t-1}-\eta\,\big(\mu\,m_t+g_t\big)
$$

- $\mu\,m_t$: the momentum step that classical momentum would take on the next iteration, taken now
- $g_t$: the current gradient, applied once on its own and once inside m_t

Classical momentum steps by $m_t$. The rewrite steps by $\mu m_t+g_t$, which looks one step ahead along the velocity. Doing the same inside Adam, with Adam's averaging and bias corrections, gives the Nadam numerator below. This is the form PyTorch implements:

$$
\begin{aligned}\mu_t&=\beta_1\big(1-\tfrac12\,0.96^{\,t\psi}\big)\\ m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2\\ \hat m_t&=\frac{\mu_{t+1}\,m_t}{1-\prod_{i=1}^{t+1}\mu_i}+\frac{(1-\mu_t)\,g_t}{1-\prod_{i=1}^{t}\mu_i},\qquad \hat v_t=\frac{v_t}{1-\beta_2^{\,t}}\\ \theta_t&=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\end{aligned}
$$

- $\mu_t$: momentum coefficient at step $t$; it warms up from $\beta_1/2$ toward $\beta_1$
- $\psi$: `momentum_decay`, PyTorch default $0.004=1/250$ (Dozat's schedule)
- $\prod\mu_i$: bias correction for a time-varying momentum coefficient
- $\eta$: learning rate; PyTorch's default for NAdam is $2\times10^{-3}$, twice Adam's

With a constant $\mu_t=\beta_1$ and $t$ large the numerator is simply $\beta_1\hat m_t+(1-\beta_1)g_t$, against Adam's $\hat m_t$. Expanding $m_t$ shows what changes: the fresh gradient $g_t$ gets weight $(1-\beta_1)(1+\beta_1)=0.19$ instead of Adam's $0.10$ (for $\beta_1=0.9$), and every older gradient is discounted by one extra factor $\beta_1$. Nadam reacts to a change in the gradient roughly twice as fast, while keeping almost the same smoothing.

**At a glance**

- **Paper:** Dozat, "Incorporating Nesterov Momentum into Adam", ICLR 2016 workshop
- **PyTorch:** `torch.optim.NAdam(params, lr=2e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, momentum_decay=4e-3, decoupled_weight_decay=False)`
- **Weight on g_t:** $0.19$ vs Adam's $0.10$ at $\beta_1=0.9$, constant $\mu$, steady state
- **Momentum schedule:** $\mu_1=0.450$, $\mu_{1000}=0.518$, $\mu_{4245}=0.675$, $\mu_{10^4}=0.812$, $\mu_{5\cdot10^4}=0.900$ (defaults)
- **State:** $m$, $v$ and the scalar product $\prod\mu_i$; same memory as Adam
- **Weight decay:** `weight_decay` is L2 by default; `decoupled_weight_decay=True` makes it AdamW-style ("NAdamW")

> Figure: The overview's valley f = ½(x² + 25y²) from (−4, 1.6), 120 full-batch steps. At lr = 0.1, β1 = 0.9: Adam (blue) first gets below loss 0.01 at step 66 and ends at 1.4e−4 after 5 crossings of the valley floor. NAdam with a constant μ = β1 (green) gets there at step 63 and ends at 1.9e−5: the look-ahead puts 0.190 of the update on the fresh gradient instead of 0.100. PyTorch's default NAdam (orange, ψ = 0.004) is slower at first (step 73) but ends at 1.1e−5 with only 3 crossings, because over these 120 steps its μ_t is only 0.450 to 0.459 (the shaded part of the schedule plot): it is effectively running with half the momentum. Raise lr to 0.3 and the gap grows: Adam needs 56 steps to reach 0.01, constant-μ NAdam 40 and PyTorch's NAdam 26. Move ψ to 1 and the orange schedule climbs to 0.897 within the 120 steps.

## The momentum schedule is not a detail

Dozat followed Sutskever et al. (2013) in warming the momentum up: $\mu_t=\beta_1(1-\tfrac12\cdot0.96^{t/250})$. PyTorch exposes the $1/250$ as `momentum_decay`. The warm-up is very slow. With the defaults, $\mu_t$ starts at $0.450$, is still $0.518$ after 1,000 steps and $0.575$ after 2,000, crosses the halfway point $0.675$ at step 4,245, and reaches $0.900$ only after about 50,000 steps.

So for a short fine-tuning run, PyTorch's NAdam is closer to "Adam with $\beta_1\approx0.5$ plus a Nesterov term" than to Adam with $\beta_1=0.9$. That can help (less overshoot, as in the figure) or hurt (more gradient noise passes through), but either way a comparison between `Adam` and `NAdam` at default settings is partly a comparison of momentum values. Two settings remove the schedule: $\psi=0$ gives a constant $\mu=\beta_1/2$, and a very large $\psi$ (say $10^3$) makes $0.96^{t\psi}$ vanish from the first step, giving a constant $\mu=\beta_1$.

## Worked example: the first step

**Start at (−4, 1.6) on the figure's valley, gradient (−4, 40), η = 0.1, β1 = 0.9**

1. **Adam.** $\hat m_1=g_1$ and $\hat v_1=g_1^2$, so the step is exactly $\eta\operatorname{sign}(g)$: $(-4,1.6)\to(-3.9,\,1.5)$.
2. **NAdam, constant μ = 0.9.** $m_1=0.1g$. The momentum term is $\mu\,m_1/(1-\mu^2)=0.9\cdot0.1g/0.19=0.4737g$, the gradient term $(1-\mu)g/(1-\mu)=g$, total $1.4737g$. Dividing by $|g|$, each coordinate moves $0.147$: $(-3.853,\,1.453)$.
3. **NAdam, PyTorch defaults.** $\mu_1=0.45006$, $\mu_2=0.45013$. Gradient term $1\cdot g$, momentum term $0.45013\cdot0.1g/(1-0.45006\cdot0.45013)=0.0565g$, total $1.0565g$: each coordinate moves $0.106$, to $(-3.894,\,1.494)$.
4. Nadam's first step is larger than Adam's, by up to $1+\beta_1(1-\beta_1)/(1-\beta_1^2)=1+\beta_1/(1+\beta_1)\approx1.47$. Combined with the larger default learning rate (2e−3), it makes NAdam somewhat more aggressive early in training.

Weight each optimizer puts on the gradient from $k$ steps ago, in steady state, $\beta_1=0.9$, constant $\mu$ (before dividing by $\sqrt{\hat v}$)

| lag k | Adam: (1 − β1)β1ᵏ | Nadam: k = 0 → (1 − β1)(1 + β1), else (1 − β1)β1ᵏ⁺¹ |
| --- | --- | --- |
| 0 (fresh) | 0.1000 | 0.1900 |
| 1 | 0.0900 | 0.0810 |
| 2 | 0.0810 | 0.0729 |
| 3 | 0.0729 | 0.0656 |

```python
import torch

# Defaults: lr 2e-3, mu_t warms from 0.45 toward 0.9 over tens of thousands of steps
opt = torch.optim.NAdam(model.parameters(), lr=2e-3, betas=(0.9, 0.999), momentum_decay=4e-3)

# "NAdamW": AdamW-style weight decay instead of L2 added to the gradient
opt = torch.optim.NAdam(model.parameters(), lr=1e-3, weight_decay=0.01,
                        decoupled_weight_decay=True)

# Constant mu = beta1 from step 1 (Dozat's update with a fixed momentum)
opt = torch.optim.NAdam(model.parameters(), lr=1e-3, momentum_decay=1e3)

# Inspect the schedule PyTorch uses
beta1, psi = 0.9, 4e-3
for t in (1, 1000, 4245, 10_000, 50_000):
    print(t, round(beta1 * (1 - 0.5 * 0.96 ** (t * psi)), 3))
# 1 0.45   1000 0.518   4245 0.675   10000 0.812   50000 0.9
```

NAdam in PyTorch: defaults, decoupled decay, and turning the momentum schedule off

> **Comparing NAdam to Adam at defaults:** Default NAdam differs from default Adam in three ways at once: Nesterov look-ahead, a learning rate twice as large ($2\times10^{-3}$ vs $10^{-3}$), and an effective momentum near $0.5$ for the first few thousand steps. If you want to measure the Nesterov effect, match the learning rates and set `momentum_decay` large so that $\mu_t=\beta_1$.

> **Why the look-ahead helps:** Momentum's weakness is lag: when the gradient reverses, classical momentum keeps going in the old direction until the new gradients outweigh the stored velocity. By doubling the weight of the newest gradient, Nadam starts braking one step earlier. On a smooth valley that shows up as slightly less overshoot and a faster final approach; with noisy minibatch gradients the gain is smaller, because the newest gradient is also the noisiest.

> **Relatives:** Nadam is one of several "Adam plus one idea" variants: AdamW (decoupled decay), AMSGrad (monotone step size), RAdam (variance-rectified warmup) and AdaMax (infinity-norm $v$). They combine: PyTorch's NAdam supports decoupled decay, and NAdam with decoupled decay is sometimes called NAdamW. The convergence counterexample of Reddi et al. (2018) applies to Nadam as much as to Adam.

**Strengths**

- Faster reaction to gradient changes: the fresh gradient gets $(1-\beta_1)(1+\beta_1)$ instead of $1-\beta_1$
- Same memory and essentially the same cost as Adam
- Drop-in replacement; supports decoupled weight decay in PyTorch
- Often slightly faster convergence on smooth, well-conditioned problems

**Limitations**

- Gains over a tuned Adam/AdamW are usually small
- PyTorch's default schedule keeps $\mu_t$ near $0.5$ for thousands of steps, which confounds comparisons
- Larger first steps and a larger default learning rate can destabilize early training
- Less tested at scale than AdamW; few large-model recipes use it

**Question:** With a constant μ = β1 = 0.9, in steady state, what weight does Nadam's numerator give the current gradient (before dividing by √v̂)?

- A. 0.10
- B. 0.19
- C. 0.90
- D. 1.00

**Answer:** B. 0.19 — The numerator is $\beta_1 m_t+(1-\beta_1)g_t$ with $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$, so $g_t$ appears with weight $\beta_1(1-\beta_1)+(1-\beta_1)=(1-\beta_1)(1+\beta_1)=0.19$. Adam gives it $1-\beta_1=0.10$.

**Question:** You fine-tune for 2,000 steps with torch.optim.NAdam at its defaults. Roughly what momentum coefficient μ_t is it using near the end?

- A. 0.9
- B. 0.99
- C. about 0.58
- D. 0, momentum is off until warmup ends

**Answer:** C. about 0.58 — $\mu_t=0.9(1-\tfrac12\cdot0.96^{0.004t})$; at $t=2000$ that is $0.575$. The schedule only approaches $\beta_1=0.9$ after about 50,000 steps, so short runs effectively use about half the momentum.

**Key points**

- Nadam = Adam with Nesterov momentum: numerator $\mu_{t+1}\hat m_t+(1-\mu_t)\hat g_t$, denominator unchanged.
- Dozat's trick applies the next step's momentum now, so no gradient at a shifted point is needed.
- The fresh gradient's weight rises from $1-\beta_1$ to $(1-\beta_1)(1+\beta_1)$: about twice as responsive.
- `torch.optim.NAdam`: lr 2e−3, `momentum_decay=4e-3` warms $\mu_t$ from 0.45 to 0.9 over roughly $5\times10^4$ steps.
- `decoupled_weight_decay=True` gives AdamW-style decay; gains over tuned AdamW are usually modest.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
