Definition
Nadam (Nesterov-accelerated adaptive moment estimation) is Adam with its classical momentum replaced by Nesterov momentum. The numerator of the update uses a look-ahead mix, , instead of ; the denominator is unchanged.
Adam combines two ideas: momentum for the direction and RMSProp for the scale. By 2015 it was known that for SGD, Nesterov's accelerated gradient (Nesterov, 1983; brought to deep learning by Sutskever, Martens, Dahl and Hinton, 2013) is usually a slightly better momentum than the classical heavy ball: it corrects course one step earlier when the velocity points the wrong way. Timothy Dozat's "Incorporating Nesterov Momentum into Adam" (ICLR 2016 workshop track) asked the obvious question, whether the same swap helps Adam, and showed how to do it without computing a gradient at a shifted point.
The result, Nadam, is a drop-in replacement for Adam with the same memory and cost. It is available as torch.optim.NAdam and keras.optimizers.Nadam. Dozat's experiments showed it converging faster than Adam and the other optimizers he compared; in later practice the improvement is real but modest, and AdamW remains the more common default.
From Nesterov momentum to Nadam
Nesterov's method evaluates the gradient at the point the momentum is about to carry you to: , then and . Dozat's rewrite keeps the gradient at the current point and instead applies the next step's momentum early:
- the momentum step that classical momentum would take on the next iteration, taken now
- the current gradient, applied once on its own and once inside m_t
Classical momentum steps by . The rewrite steps by , which looks one step ahead along the velocity. Doing the same inside Adam, with Adam's averaging and bias corrections, gives the Nadam numerator below. This is the form PyTorch implements:
- momentum coefficient at step ; it warms up from toward
momentum_decay, PyTorch default (Dozat's schedule)- bias correction for a time-varying momentum coefficient
- learning rate; PyTorch's default for NAdam is , twice Adam's
With a constant and large the numerator is simply , against Adam's . Expanding shows what changes: the fresh gradient gets weight instead of Adam's (for ), and every older gradient is discounted by one extra factor . Nadam reacts to a change in the gradient roughly twice as fast, while keeping almost the same smoothing.
At a glance
- Paper
- Dozat, "Incorporating Nesterov Momentum into Adam", ICLR 2016 workshop
- PyTorch
torch.optim.NAdam(params, lr=2e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, momentum_decay=4e-3, decoupled_weight_decay=False)- Weight on g_t
- vs Adam's at , constant , steady state
- Momentum schedule
- , , , , (defaults)
- State
- , and the scalar product ; same memory as Adam
- Weight decay
weight_decayis L2 by default;decoupled_weight_decay=Truemakes it AdamW-style ("NAdamW")
The momentum schedule is not a detail
Dozat followed Sutskever et al. (2013) in warming the momentum up: . PyTorch exposes the as momentum_decay. The warm-up is very slow. With the defaults, starts at , is still after 1,000 steps and after 2,000, crosses the halfway point at step 4,245, and reaches only after about 50,000 steps.
So for a short fine-tuning run, PyTorch's NAdam is closer to "Adam with plus a Nesterov term" than to Adam with . That can help (less overshoot, as in the figure) or hurt (more gradient noise passes through), but either way a comparison between Adam and NAdam at default settings is partly a comparison of momentum values. Two settings remove the schedule: gives a constant , and a very large (say ) makes vanish from the first step, giving a constant .
Worked example: the first step
Start at (−4, 1.6) on the figure's valley, gradient (−4, 40), η = 0.1, β1 = 0.9
- 1Adam. and , so the step is exactly : .
- 2NAdam, constant μ = 0.9. . The momentum term is , the gradient term , total . Dividing by , each coordinate moves : .
- 3NAdam, PyTorch defaults. , . Gradient term , momentum term , total : each coordinate moves , to .
- 4Nadam's first step is larger than Adam's, by up to . Combined with the larger default learning rate (2e−3), it makes NAdam somewhat more aggressive early in training.
| lag k | Adam: (1 − β1)β1ᵏ | Nadam: k = 0 → (1 − β1)(1 + β1), else (1 − β1)β1ᵏ⁺¹ |
|---|---|---|
| 0 (fresh) | 0.1000 | 0.1900 |
| 1 | 0.0900 | 0.0810 |
| 2 | 0.0810 | 0.0729 |
| 3 | 0.0729 | 0.0656 |
1import torch23# Defaults: lr 2e-3, mu_t warms from 0.45 toward 0.9 over tens of thousands of steps4opt = torch.optim.NAdam(model.parameters(), lr=2e-3, betas=(0.9, 0.999), momentum_decay=4e-3)56# "NAdamW": AdamW-style weight decay instead of L2 added to the gradient7opt = torch.optim.NAdam(model.parameters(), lr=1e-3, weight_decay=0.01,8 decoupled_weight_decay=True)910# Constant mu = beta1 from step 1 (Dozat's update with a fixed momentum)11opt = torch.optim.NAdam(model.parameters(), lr=1e-3, momentum_decay=1e3)1213# Inspect the schedule PyTorch uses14beta1, psi = 0.9, 4e-315for t in (1, 1000, 4245, 10_000, 50_000):16 print(t, round(beta1 * (1 - 0.5 * 0.96 ** (t * psi)), 3))17# 1 0.45 1000 0.518 4245 0.675 10000 0.812 50000 0.9Use when
- +Faster reaction to gradient changes: the fresh gradient gets instead of
- +Same memory and essentially the same cost as Adam
- +Drop-in replacement; supports decoupled weight decay in PyTorch
- +Often slightly faster convergence on smooth, well-conditioned problems
Avoid when
- −Gains over a tuned Adam/AdamW are usually small
- −PyTorch's default schedule keeps near for thousands of steps, which confounds comparisons
- −Larger first steps and a larger default learning rate can destabilize early training
- −Less tested at scale than AdamW; few large-model recipes use it
Check yourself
With a constant μ = β1 = 0.9, in steady state, what weight does Nadam's numerator give the current gradient (before dividing by √v̂)?
Check yourself
You fine-tune for 2,000 steps with torch.optim.NAdam at its defaults. Roughly what momentum coefficient μ_t is it using near the end?
Remember
- ✓Nadam = Adam with Nesterov momentum: numerator , denominator unchanged.
- ✓Dozat's trick applies the next step's momentum now, so no gradient at a shifted point is needed.
- ✓The fresh gradient's weight rises from to : about twice as responsive.
- ✓
torch.optim.NAdam: lr 2e−3,momentum_decay=4e-3warms from 0.45 to 0.9 over roughly steps. - ✓
decoupled_weight_decay=Truegives AdamW-style decay; gains over tuned AdamW are usually modest.