Momentum

Keep a velocity, an exponentially weighted sum of past gradients, and step along it. Oscillating components cancel, consistent ones build up to η/(1−β) per step, so ravines are crossed quickly instead of zigzagged.

11 min read1204 wordscore0 viewsupdated 2026-10-11

Definition

Momentum (the heavy-ball method) replaces the gradient step θ←θ−ηg\theta\leftarrow\theta-\eta g with a step along a velocity vv that accumulates past gradients: v←βv+gv\leftarrow\beta v+g, then θ←θ−ηv\theta\leftarrow\theta-\eta v. With β=0\beta=0 it is plain gradient descent; with β\beta near 1 the update remembers many past gradients.

Plain gradient descent has no memory. Every step uses only the current gradient, so in a long narrow valley it bounces from wall to wall while creeping along the floor. The learning rate cannot fix this: the steep walls cap it, and the flat floor then gets a tiny step. Momentum gives the iterate inertia. Gradient components that keep pointing the same way add up step after step; components that flip sign every few steps cancel out.

The method is Boris Polyak's, from "Some methods of speeding up the convergence of iteration methods" (USSR Computational Mathematics and Mathematical Physics, 1964). The name comes from his physical picture: a heavy ball rolling on the loss surface under gravity and friction keeps moving through flat stretches and is not thrown around by every bump. Rumelhart, Hinton and Williams included a momentum term in their 1986 backpropagation paper, and Sutskever, Martens, Dahl and Hinton ("On the importance of initialization and momentum in deep learning", ICML 2013) showed that well-tuned momentum, with β\beta raised gradually toward 0.99, trains deep and recurrent networks that plain SGD could not.

The update rule

vt+1=β vt+gt,θt+1=θt−η vt+1,gt=∇θL(θt)v_{t+1}=\beta\,v_t+g_t,\qquad \theta_{t+1}=\theta_t-\eta\,v_{t+1},\qquad g_t=\nabla_\theta L(\theta_t)
vtv_t
velocity, in gradient units; v0=0v_0=0 (PyTorch sets v1=g0v_1=g_0 on the first step)
β\beta
momentum coefficient in [0,1)[0,1); 0.90.9 is the usual default
η\eta
learning rate
gtg_t
the (mini-batch) gradient at the current parameters

Unrolling the recursion shows what the velocity is: an exponentially weighted sum of every gradient so far, with weight βk\beta^k on the gradient from kk steps ago.

vt+1=∑k=0tβk gt−k⇒if gt≡g:vt→g1−β,Δθ→−η1−β gv_{t+1}=\sum_{k=0}^{t}\beta^{k}\,g_{t-k}\qquad\Rightarrow\qquad \text{if } g_t\equiv g:\quad v_t\to\frac{g}{1-\beta},\quad \Delta\theta\to-\frac{\eta}{1-\beta}\,g
βk\beta^k
weight of a gradient kk steps old; it falls to 1/e1/e after about 1/(1−β)1/(1-\beta) steps (10 steps for β=0.9\beta=0.9)
η/(1−β)\eta/(1-\beta)
the effective step in a direction where the gradient is steady: 10η10\eta at β=0.9\beta=0.9, 100η100\eta at β=0.99\beta=0.99

Written with weights (1−β)βk(1-\beta)\beta^k instead of βk\beta^k, the same sum is an exponential moving average (EMA) of the gradients. That is the form Adam uses for its first moment. The two forms differ only by the constant factor 1−β1-\beta, which can be folded into the learning rate.

At a glance

State
one extra tensor per parameter (the velocity)
Hyperparameters
η\eta, β\beta (default 0.90.9), optional dampening dd
Effective step
η(1−d)/(1−β)\eta(1-d)/(1-\beta) along a constant gradient
Stability on a quadratic
ηλmax⁡(1−d)<2(1+β)\eta\lambda_{\max}(1-d)<2(1+\beta), versus ηλmax⁡<2\eta\lambda_{\max}<2 for plain GD
Best rate on a quadratic
(κ−1)/(κ+1)(\sqrt\kappa-1)/(\sqrt\kappa+1) per step, versus (κ−1)/(κ+1)(\kappa-1)/(\kappa+1) for plain GD
PyTorch
torch.optim.SGD(params, lr, momentum=0.9, dampening=0, nesterov=False)
f = ½(x² + 25y²): 25× steeper across than alongstartminloss (log) vs step1e-61e-311e3velocity v (gradient units)-50050v_y acrossv_x along060120
t = 120|
● momentum loss 3.8e-5● plain GD loss 0.063v = (0.03, 0.09), step length 0.002η now 0.0200steady step η(1−d)/(1−β) = 0.200stable for η < 2(1+β)/(25(1−d)) = 0.152
At a constant η the two conventions give exactly the same trajectory.
fig.Momentum (orange) and plain gradient descent (grey) with the same learning rate on f = ½(x² + 25y²). At the defaults η = 0.02, β = 0.9, GD has only reached loss 0.063 after 120 steps while momentum is at 3.8e-5. The velocity panel shows why: the across-valley component v_y swings between +56 and −44 and averages out, while the along-valley component v_x grows from −4 to −18.3 within 9 steps. Drag β to 0.99 and the ball overshoots for the whole run (loss 7.0); push η past the printed limit 2(1+β)/25 = 0.152 and it diverges. 'Polyak tuning' sets η = 0.111, β = 0.444, which crosses loss 1e-3 at step 22. Turn on the η ÷ 10 drop to see the two conventions part ways.

Why it damps ravines

Take a quadratic valley f=12(x2+κy2)f=\tfrac12(x^2+\kappa y^2), so the curvature is 1 along xx and κ\kappa across. Gradient descent is stable only if η<2/κ\eta<2/\kappa, and then the xx coordinate shrinks by 1−η1-\eta per step: with κ=25\kappa=25 and η=0.02\eta=0.02, a factor 0.98, which takes about 300 steps to shrink by e−6e^{-6}.

Momentum treats the two directions differently without being told which is which. Along xx the gradient keeps its sign, so successive gradients add and the step grows toward η/(1−β)\eta/(1-\beta) times the gradient: ten times larger at β=0.9\beta=0.9. Across yy, the iterate overshoots the floor and the gradient flips sign every few steps, so the contributions to vyv_y largely cancel. The ball keeps its speed along the valley and loses it across.

On a quadratic each eigendirection with curvature λ\lambda obeys the linear recurrence θt+1=(1+β−ηλ)θt−β θt−1\theta_{t+1}=(1+\beta-\eta\lambda)\theta_t-\beta\,\theta_{t-1}. It is stable exactly when 0<ηλ<2(1+β)0<\eta\lambda<2(1+\beta), so momentum also tolerates a larger learning rate: up to 0.152 instead of 0.08 for κ=25\kappa=25 and β=0.9\beta=0.9. Polyak showed that the choice β=(κ−1κ+1)2\beta=\big(\tfrac{\sqrt\kappa-1}{\sqrt\kappa+1}\big)^2, η=4(λmax⁡+λmin⁡)2\eta=\tfrac{4}{(\sqrt{\lambda_{\max}}+\sqrt{\lambda_{\min}})^2} makes every direction contract by κ−1κ+1\tfrac{\sqrt\kappa-1}{\sqrt\kappa+1} per step. For κ=25\kappa=25 that is 0.6670.667, against 0.9230.923 for the best plain-GD step: the number of steps needed scales with κ\sqrt\kappa instead of κ\kappa.

ρGD=κ−1κ+1vs.ρHB=κ−1κ+1,κ=λmax⁡λmin⁡\rho_{\text{GD}}=\frac{\kappa-1}{\kappa+1}\qquad\text{vs.}\qquad \rho_{\text{HB}}=\frac{\sqrt\kappa-1}{\sqrt\kappa+1},\qquad \kappa=\frac{\lambda_{\max}}{\lambda_{\min}}
ρ\rho
worst-case contraction of the error per step, with optimal tuning, on a quadratic
κ\kappa
condition number; at κ=104\kappa=10^4, ρGD=0.9998\rho_{\text{GD}}=0.9998 but ρHB=0.980\rho_{\text{HB}}=0.980

Worked example: the first five steps

The figure's valley f=12(x2+25y2)f=\tfrac12(x^2+25y^2), start θ0=(−4, 1.6)\theta_0=(-4,\,1.6), f=40f=40, gradient (−4, 40)(-4,\,40), η=0.02\eta=0.02, β=0.9\beta=0.9, PyTorch convention. Values computed by the figure's code.

tmomentum θvelocity vmomentum lossGD θGD loss
1(−3.920, 0.800)(−4.00, 40.0)15.68(−3.920, 0.800)15.68
2(−3.770, −0.320)(−7.52, 56.0)8.38(−3.842, 0.400)9.38
3(−3.559, −1.168)(−10.54, 42.4)23.39(−3.765, 0.200)7.59
4(−3.298, −1.347)(−13.04, 8.96)28.13(−3.690, 0.100)6.93
5(−2.997, −0.835)(−15.04, −25.6)13.20(−3.616, 0.050)6.57
Momentum vs plain GD, same learning rate
  1. 1Step 1 is identical for both: the buffer starts as v1=g0=(−4,40)v_1=g_0=(-4,40), so Δθ=−0.02 (−4,40)=(0.08,−0.8)\Delta\theta=-0.02\,(-4,40)=(0.08,-0.8).
  2. 2Step 2: g1=(−3.92, 20)g_1=(-3.92,\,20), so v2=0.9 (−4,40)+(−3.92,20)=(−7.52, 56)v_2=0.9\,(-4,40)+(-3.92,20)=(-7.52,\,56). The stored yy-velocity carries the iterate past the floor to y=−0.32y=-0.32, while GD halves yy to 0.40.4.
  3. 3Steps 3 to 5: the yy-velocity is still positive, so yy keeps falling to −1.35-1.35 and the loss rises to 28. The new negative gradients brake it, and by step 5 vyv_y has flipped to −25.6-25.6. That is the swing the velocity panel shows.
  4. 4Meanwhile vxv_x grows every step, −4→−15.0-4\to-15.0, and momentum has already moved xx by 1.01.0 against GD's 0.380.38. After 10 steps the losses are 1.00 vs 5.34; after 40 steps 0.024 vs 1.59.

Two conventions, and dampening

Papers usually write the classical form vt+1=βvt−ηgtv_{t+1}=\beta v_t-\eta g_t, θt+1=θt+vt+1\theta_{t+1}=\theta_t+v_{t+1}, where the velocity is in parameter units and already contains the learning rate. PyTorch (and most frameworks) store vv in gradient units and multiply by η\eta at the end. With a constant η\eta the two produce exactly the same iterates: substitute vcls=−ηvv^{\text{cls}}=-\eta v.

v←βv−ηtg,    θ←θ+v⏟classical (Sutskever et al. 2013)b←βb+(1−d) g,    θ←θ−ηt b⏟PyTorch SGD\underbrace{v\leftarrow\beta v-\eta_t g,\;\; \theta\leftarrow\theta+v}_{\text{classical (Sutskever et al. 2013)}}\qquad\qquad \underbrace{b\leftarrow\beta b+(1-d)\,g,\;\; \theta\leftarrow\theta-\eta_t\,b}_{\text{PyTorch SGD}}
ηt\eta_t
the learning rate at step t, changed by a scheduler
bb
PyTorch's momentum_buffer, initialised to the first gradient
dd
dampening, default 0; it scales the incoming gradient, not the stored velocity

They differ once the learning rate changes. In PyTorch's form the whole step is multiplied by the new ηt\eta_t, so a 10× drop takes effect immediately. In the classical form the velocity was built with the old learning rate and keeps carrying it, decaying by β\beta per step. In the figure with η ÷ 10 after step 30 (at η=0.02\eta=0.02), PyTorch's step length goes from 0.119 at step 30 to 0.023 at step 31; the classical form's is still 0.116 at step 31 and 0.091 at step 35.

Dampening multiplies the incoming gradient by 1−d1-d. With d=βd=\beta the buffer is a true EMA and the effective step is η\eta rather than η/(1−β)\eta/(1-\beta), so the learning rate must be about 1/(1−β)1/(1-\beta) times larger for the same behaviour. In the figure, dampening 0.90.9 at η=0.02\eta=0.02 ends at loss 0.017; at η=0.2\eta=0.2 it ends at 3.0×10−33.0\times10^{-3}. It is not an exact match for undamped η=0.02\eta=0.02 (3.8×10−53.8\times10^{-5}) because PyTorch skips the dampening on the very first step, so that first step is 10× larger. Dampening is rarely used; it must be 0 when nesterov=True.

python
1import torch
2
3theta = torch.tensor([-4.0, 1.6], requires_grad=True)
4opt = torch.optim.SGD([theta], lr=0.02, momentum=0.9) # dampening=0, nesterov=False
5
6def f(t):
7 return 0.5 * (t[0] ** 2 + 25 * t[1] ** 2)
8
9for step in range(5):
10 opt.zero_grad()
11 f(theta).backward()
12 opt.step() # buf = 0.9 * buf + grad (buf = grad on step 1); theta -= lr * buf
13
14print(theta.detach()) # ≈ (-2.997, -0.835)
15print(opt.state[theta]["momentum_buffer"]) # ≈ (-15.04, -25.62)
16
17# Typical use on a real model, with weight decay and a schedule:
18# opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
19# sched = torch.optim.lr_scheduler.MultiStepLR(opt, milestones=[30, 60], gamma=0.1)
20#
21# From scratch, for each parameter p with gradient g:
22# buf = g.clone() if buf is None else beta * buf + (1 - dampening) * g
23# p -= lr * buf
torch.optim.SGD with momentum on the figure's valley (same five steps as the table)

Use when

  • +Faster along consistent directions: up to 1/(1−β)1/(1-\beta) times the plain step
  • +Damps oscillation across ravines without knowing their direction
  • +Tolerates larger learning rates on quadratics: 2(1+β)/λmax⁡2(1+\beta)/\lambda_{\max} instead of 2/λmax⁡2/\lambda_{\max}
  • +Averages out mini-batch noise in the direction of travel
  • +One extra tensor of state, half of Adam's

Avoid when

  • −One more hyperparameter, coupled to the learning rate through η/(1−β)\eta/(1-\beta)
  • −Overshoots and oscillates when β is too large for the curvature
  • −Stale velocity after abrupt changes in the loss or the learning rate
  • −No per-parameter scaling: badly scaled parameters still need adaptive methods

Check yourself

The gradient points the same way with the same size for many steps. With β = 0.9 and learning rate η, how far does momentum (PyTorch convention, no dampening) eventually move per step?

Check yourself

A scheduler cuts the learning rate by 10× at step 30. In which form of momentum does the step size shrink 10× at once?

Remember

  • ✓v←βv+gv\leftarrow\beta v+g, θ←θ−ηv\theta\leftarrow\theta-\eta v: the velocity is an exponentially weighted sum of past gradients.
  • ✓Along a steady gradient the step tends to η/(1−β)\eta/(1-\beta): 10× the plain step at β=0.9\beta=0.9.
  • ✓Oscillating gradient components cancel in the velocity, consistent ones add: that is the ravine damping.
  • ✓On a quadratic, optimally tuned momentum contracts by (κ−1)/(κ+1)(\sqrt\kappa-1)/(\sqrt\kappa+1) per step, versus (κ−1)/(κ+1)(\kappa-1)/(\kappa+1) for GD.
  • ✓PyTorch stores v in gradient units and multiplies by the current η, so learning-rate changes act at once; the classical form keeps the old η in the velocity.
  • ✓Too much momentum underdamps: the error can shrink no faster than √β per step.
← back to Deep Learning roadmap