Nadam

Adam with Nesterov momentum: the update applies next step's momentum now, so the fresh gradient gets about twice Adam's weight. PyTorch's NAdam also warms the momentum up slowly (momentum_decay), which changes its behaviour over the first thousands of steps.

11 min read805 wordsadvanced0 viewsupdated 2026-10-11

Definition

Nadam (Nesterov-accelerated adaptive moment estimation) is Adam with its classical momentum replaced by Nesterov momentum. The numerator of the update uses a look-ahead mix, μt+1m^t+(1−μt)g^t\mu_{t+1}\hat m_t+(1-\mu_t)\hat g_t, instead of m^t\hat m_t; the denominator v^t+ϵ\sqrt{\hat v_t}+\epsilon is unchanged.

Adam combines two ideas: momentum for the direction and RMSProp for the scale. By 2015 it was known that for SGD, Nesterov's accelerated gradient (Nesterov, 1983; brought to deep learning by Sutskever, Martens, Dahl and Hinton, 2013) is usually a slightly better momentum than the classical heavy ball: it corrects course one step earlier when the velocity points the wrong way. Timothy Dozat's "Incorporating Nesterov Momentum into Adam" (ICLR 2016 workshop track) asked the obvious question, whether the same swap helps Adam, and showed how to do it without computing a gradient at a shifted point.

The result, Nadam, is a drop-in replacement for Adam with the same memory and cost. It is available as torch.optim.NAdam and keras.optimizers.Nadam. Dozat's experiments showed it converging faster than Adam and the other optimizers he compared; in later practice the improvement is real but modest, and AdamW remains the more common default.

From Nesterov momentum to Nadam

Nesterov's method evaluates the gradient at the point the momentum is about to carry you to: gt=∇f(θt−1−ημmt−1)g_t=\nabla f(\theta_{t-1}-\eta\mu m_{t-1}), then mt=μmt−1+gtm_t=\mu m_{t-1}+g_t and θt=θt−1−ηmt\theta_t=\theta_{t-1}-\eta m_t. Dozat's rewrite keeps the gradient at the current point and instead applies the next step's momentum early:

gt=∇f(θt−1),mt=μ mt−1+gt,θt=θt−1−η (μ mt+gt)g_t=\nabla f(\theta_{t-1}),\qquad m_t=\mu\,m_{t-1}+g_t,\qquad \theta_t=\theta_{t-1}-\eta\,\big(\mu\,m_t+g_t\big)
μ mt\mu\,m_t
the momentum step that classical momentum would take on the next iteration, taken now
gtg_t
the current gradient, applied once on its own and once inside m_t

Classical momentum steps by mtm_t. The rewrite steps by μmt+gt\mu m_t+g_t, which looks one step ahead along the velocity. Doing the same inside Adam, with Adam's averaging and bias corrections, gives the Nadam numerator below. This is the form PyTorch implements:

μt=β1(1−12 0.96 tψ)mt=β1mt−1+(1−β1) gt,vt=β2vt−1+(1−β2) gt2m^t=μt+1 mt1−∏i=1t+1μi+(1−μt) gt1−∏i=1tμi,v^t=vt1−β2 tθt=θt−1−η m^tv^t+ϵ\begin{aligned}\mu_t&=\beta_1\big(1-\tfrac12\,0.96^{\,t\psi}\big)\\ m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2\\ \hat m_t&=\frac{\mu_{t+1}\,m_t}{1-\prod_{i=1}^{t+1}\mu_i}+\frac{(1-\mu_t)\,g_t}{1-\prod_{i=1}^{t}\mu_i},\qquad \hat v_t=\frac{v_t}{1-\beta_2^{\,t}}\\ \theta_t&=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\end{aligned}
μt\mu_t
momentum coefficient at step tt; it warms up from β1/2\beta_1/2 toward β1\beta_1
ψ\psi
momentum_decay, PyTorch default 0.004=1/2500.004=1/250 (Dozat's schedule)
∏μi\prod\mu_i
bias correction for a time-varying momentum coefficient
η\eta
learning rate; PyTorch's default for NAdam is 2×10−32\times10^{-3}, twice Adam's

With a constant μt=β1\mu_t=\beta_1 and tt large the numerator is simply β1m^t+(1−β1)gt\beta_1\hat m_t+(1-\beta_1)g_t, against Adam's m^t\hat m_t. Expanding mtm_t shows what changes: the fresh gradient gtg_t gets weight (1−β1)(1+β1)=0.19(1-\beta_1)(1+\beta_1)=0.19 instead of Adam's 0.100.10 (for β1=0.9\beta_1=0.9), and every older gradient is discounted by one extra factor β1\beta_1. Nadam reacts to a change in the gradient roughly twice as fast, while keeping almost the same smoothing.

At a glance

Paper
Dozat, "Incorporating Nesterov Momentum into Adam", ICLR 2016 workshop
PyTorch
torch.optim.NAdam(params, lr=2e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, momentum_decay=4e-3, decoupled_weight_decay=False)
Weight on g_t
0.190.19 vs Adam's 0.100.10 at β1=0.9\beta_1=0.9, constant μ\mu, steady state
Momentum schedule
μ1=0.450\mu_1=0.450, μ1000=0.518\mu_{1000}=0.518, μ4245=0.675\mu_{4245}=0.675, μ104=0.812\mu_{10^4}=0.812, μ5⋅104=0.900\mu_{5\cdot10^4}=0.900 (defaults)
State
mm, vv and the scalar product ∏μi\prod\mu_i; same memory as Adam
Weight decay
weight_decay is L2 by default; decoupled_weight_decay=True makes it AdamW-style ("NAdamW")
f = ½(x² + 25y²), start (-4, 1.6), lr = 0.100, β1 = 0.90minloss (log) vs step1e-81e-41060120torch μ_t, ψ = 0.00400.900.4511e21e4step (log); shaded = this runt = 120, torch μ_t = 0.459 · weight on fresh g_t: Adam 0.100 · NAdam(ψ) 0.587 · μ=β1 0.190valley crossings: Adam 5 · NAdam (torch ψ) 3 · NAdam, μ = β1 5
t = 120
fig.The overview's valley f = ½(x² + 25y²) from (−4, 1.6), 120 full-batch steps. At lr = 0.1, β1 = 0.9: Adam (blue) first gets below loss 0.01 at step 66 and ends at 1.4e−4 after 5 crossings of the valley floor. NAdam with a constant μ = β1 (green) gets there at step 63 and ends at 1.9e−5: the look-ahead puts 0.190 of the update on the fresh gradient instead of 0.100. PyTorch's default NAdam (orange, ψ = 0.004) is slower at first (step 73) but ends at 1.1e−5 with only 3 crossings, because over these 120 steps its μ_t is only 0.450 to 0.459 (the shaded part of the schedule plot): it is effectively running with half the momentum. Raise lr to 0.3 and the gap grows: Adam needs 56 steps to reach 0.01, constant-μ NAdam 40 and PyTorch's NAdam 26. Move ψ to 1 and the orange schedule climbs to 0.897 within the 120 steps.

The momentum schedule is not a detail

Dozat followed Sutskever et al. (2013) in warming the momentum up: μt=β1(1−12⋅0.96t/250)\mu_t=\beta_1(1-\tfrac12\cdot0.96^{t/250}). PyTorch exposes the 1/2501/250 as momentum_decay. The warm-up is very slow. With the defaults, μt\mu_t starts at 0.4500.450, is still 0.5180.518 after 1,000 steps and 0.5750.575 after 2,000, crosses the halfway point 0.6750.675 at step 4,245, and reaches 0.9000.900 only after about 50,000 steps.

So for a short fine-tuning run, PyTorch's NAdam is closer to "Adam with β1≈0.5\beta_1\approx0.5 plus a Nesterov term" than to Adam with β1=0.9\beta_1=0.9. That can help (less overshoot, as in the figure) or hurt (more gradient noise passes through), but either way a comparison between Adam and NAdam at default settings is partly a comparison of momentum values. Two settings remove the schedule: ψ=0\psi=0 gives a constant μ=β1/2\mu=\beta_1/2, and a very large ψ\psi (say 10310^3) makes 0.96tψ0.96^{t\psi} vanish from the first step, giving a constant μ=β1\mu=\beta_1.

Worked example: the first step

Start at (−4, 1.6) on the figure's valley, gradient (−4, 40), η = 0.1, β1 = 0.9

  1. 1Adam. m^1=g1\hat m_1=g_1 and v^1=g12\hat v_1=g_1^2, so the step is exactly ηsign⁡(g)\eta\operatorname{sign}(g): (−4,1.6)→(−3.9, 1.5)(-4,1.6)\to(-3.9,\,1.5).
  2. 2NAdam, constant μ = 0.9. m1=0.1gm_1=0.1g. The momentum term is μ m1/(1−μ2)=0.9⋅0.1g/0.19=0.4737g\mu\,m_1/(1-\mu^2)=0.9\cdot0.1g/0.19=0.4737g, the gradient term (1−μ)g/(1−μ)=g(1-\mu)g/(1-\mu)=g, total 1.4737g1.4737g. Dividing by ∣g∣|g|, each coordinate moves 0.1470.147: (−3.853, 1.453)(-3.853,\,1.453).
  3. 3NAdam, PyTorch defaults. μ1=0.45006\mu_1=0.45006, μ2=0.45013\mu_2=0.45013. Gradient term 1⋅g1\cdot g, momentum term 0.45013⋅0.1g/(1−0.45006⋅0.45013)=0.0565g0.45013\cdot0.1g/(1-0.45006\cdot0.45013)=0.0565g, total 1.0565g1.0565g: each coordinate moves 0.1060.106, to (−3.894, 1.494)(-3.894,\,1.494).
  4. 4Nadam's first step is larger than Adam's, by up to 1+β1(1−β1)/(1−β12)=1+β1/(1+β1)≈1.471+\beta_1(1-\beta_1)/(1-\beta_1^2)=1+\beta_1/(1+\beta_1)\approx1.47. Combined with the larger default learning rate (2e−3), it makes NAdam somewhat more aggressive early in training.
lag kAdam: (1 − β1)β1ᵏNadam: k = 0 → (1 − β1)(1 + β1), else (1 − β1)β1ᵏ⁺¹
0 (fresh)0.10000.1900
10.09000.0810
20.08100.0729
30.07290.0656
Weight each optimizer puts on the gradient from kk steps ago, in steady state, β1=0.9\beta_1=0.9, constant μ\mu (before dividing by v^\sqrt{\hat v})
python
1import torch
2
3# Defaults: lr 2e-3, mu_t warms from 0.45 toward 0.9 over tens of thousands of steps
4opt = torch.optim.NAdam(model.parameters(), lr=2e-3, betas=(0.9, 0.999), momentum_decay=4e-3)
5
6# "NAdamW": AdamW-style weight decay instead of L2 added to the gradient
7opt = torch.optim.NAdam(model.parameters(), lr=1e-3, weight_decay=0.01,
8 decoupled_weight_decay=True)
9
10# Constant mu = beta1 from step 1 (Dozat's update with a fixed momentum)
11opt = torch.optim.NAdam(model.parameters(), lr=1e-3, momentum_decay=1e3)
12
13# Inspect the schedule PyTorch uses
14beta1, psi = 0.9, 4e-3
15for t in (1, 1000, 4245, 10_000, 50_000):
16 print(t, round(beta1 * (1 - 0.5 * 0.96 ** (t * psi)), 3))
17# 1 0.45 1000 0.518 4245 0.675 10000 0.812 50000 0.9
NAdam in PyTorch: defaults, decoupled decay, and turning the momentum schedule off

Use when

  • +Faster reaction to gradient changes: the fresh gradient gets (1−β1)(1+β1)(1-\beta_1)(1+\beta_1) instead of 1−β11-\beta_1
  • +Same memory and essentially the same cost as Adam
  • +Drop-in replacement; supports decoupled weight decay in PyTorch
  • +Often slightly faster convergence on smooth, well-conditioned problems

Avoid when

  • −Gains over a tuned Adam/AdamW are usually small
  • −PyTorch's default schedule keeps μt\mu_t near 0.50.5 for thousands of steps, which confounds comparisons
  • −Larger first steps and a larger default learning rate can destabilize early training
  • −Less tested at scale than AdamW; few large-model recipes use it

Check yourself

With a constant μ = β1 = 0.9, in steady state, what weight does Nadam's numerator give the current gradient (before dividing by √v̂)?

Check yourself

You fine-tune for 2,000 steps with torch.optim.NAdam at its defaults. Roughly what momentum coefficient μ_t is it using near the end?

Remember

  • ✓Nadam = Adam with Nesterov momentum: numerator μt+1m^t+(1−μt)g^t\mu_{t+1}\hat m_t+(1-\mu_t)\hat g_t, denominator unchanged.
  • ✓Dozat's trick applies the next step's momentum now, so no gradient at a shifted point is needed.
  • ✓The fresh gradient's weight rises from 1−β11-\beta_1 to (1−β1)(1+β1)(1-\beta_1)(1+\beta_1): about twice as responsive.
  • ✓torch.optim.NAdam: lr 2e−3, momentum_decay=4e-3 warms μt\mu_t from 0.45 to 0.9 over roughly 5×1045\times10^4 steps.
  • ✓decoupled_weight_decay=True gives AdamW-style decay; gains over tuned AdamW are usually modest.
← back to Deep Learning roadmap