Definition
Momentum (the heavy-ball method) replaces the gradient step with a step along a velocity that accumulates past gradients: , then . With it is plain gradient descent; with near 1 the update remembers many past gradients.
Plain gradient descent has no memory. Every step uses only the current gradient, so in a long narrow valley it bounces from wall to wall while creeping along the floor. The learning rate cannot fix this: the steep walls cap it, and the flat floor then gets a tiny step. Momentum gives the iterate inertia. Gradient components that keep pointing the same way add up step after step; components that flip sign every few steps cancel out.
The method is Boris Polyak's, from "Some methods of speeding up the convergence of iteration methods" (USSR Computational Mathematics and Mathematical Physics, 1964). The name comes from his physical picture: a heavy ball rolling on the loss surface under gravity and friction keeps moving through flat stretches and is not thrown around by every bump. Rumelhart, Hinton and Williams included a momentum term in their 1986 backpropagation paper, and Sutskever, Martens, Dahl and Hinton ("On the importance of initialization and momentum in deep learning", ICML 2013) showed that well-tuned momentum, with raised gradually toward 0.99, trains deep and recurrent networks that plain SGD could not.
The update rule
- velocity, in gradient units; (PyTorch sets on the first step)
- momentum coefficient in ; is the usual default
- learning rate
- the (mini-batch) gradient at the current parameters
Unrolling the recursion shows what the velocity is: an exponentially weighted sum of every gradient so far, with weight on the gradient from steps ago.
- weight of a gradient steps old; it falls to after about steps (10 steps for )
- the effective step in a direction where the gradient is steady: at , at
Written with weights instead of , the same sum is an exponential moving average (EMA) of the gradients. That is the form Adam uses for its first moment. The two forms differ only by the constant factor , which can be folded into the learning rate.
At a glance
- State
- one extra tensor per parameter (the velocity)
- Hyperparameters
- , (default ), optional dampening
- Effective step
- along a constant gradient
- Stability on a quadratic
- , versus for plain GD
- Best rate on a quadratic
- per step, versus for plain GD
- PyTorch
torch.optim.SGD(params, lr, momentum=0.9, dampening=0, nesterov=False)
Why it damps ravines
Take a quadratic valley , so the curvature is 1 along and across. Gradient descent is stable only if , and then the coordinate shrinks by per step: with and , a factor 0.98, which takes about 300 steps to shrink by .
Momentum treats the two directions differently without being told which is which. Along the gradient keeps its sign, so successive gradients add and the step grows toward times the gradient: ten times larger at . Across , the iterate overshoots the floor and the gradient flips sign every few steps, so the contributions to largely cancel. The ball keeps its speed along the valley and loses it across.
On a quadratic each eigendirection with curvature obeys the linear recurrence . It is stable exactly when , so momentum also tolerates a larger learning rate: up to 0.152 instead of 0.08 for and . Polyak showed that the choice , makes every direction contract by per step. For that is , against for the best plain-GD step: the number of steps needed scales with instead of .
- worst-case contraction of the error per step, with optimal tuning, on a quadratic
- condition number; at , but
Worked example: the first five steps
The figure's valley , start , , gradient , , , PyTorch convention. Values computed by the figure's code.
| t | momentum θ | velocity v | momentum loss | GD θ | GD loss |
|---|---|---|---|---|---|
| 1 | (−3.920, 0.800) | (−4.00, 40.0) | 15.68 | (−3.920, 0.800) | 15.68 |
| 2 | (−3.770, −0.320) | (−7.52, 56.0) | 8.38 | (−3.842, 0.400) | 9.38 |
| 3 | (−3.559, −1.168) | (−10.54, 42.4) | 23.39 | (−3.765, 0.200) | 7.59 |
| 4 | (−3.298, −1.347) | (−13.04, 8.96) | 28.13 | (−3.690, 0.100) | 6.93 |
| 5 | (−2.997, −0.835) | (−15.04, −25.6) | 13.20 | (−3.616, 0.050) | 6.57 |
- 1Step 1 is identical for both: the buffer starts as , so .
- 2Step 2: , so . The stored -velocity carries the iterate past the floor to , while GD halves to .
- 3Steps 3 to 5: the -velocity is still positive, so keeps falling to and the loss rises to 28. The new negative gradients brake it, and by step 5 has flipped to . That is the swing the velocity panel shows.
- 4Meanwhile grows every step, , and momentum has already moved by against GD's . After 10 steps the losses are 1.00 vs 5.34; after 40 steps 0.024 vs 1.59.
Two conventions, and dampening
Papers usually write the classical form , , where the velocity is in parameter units and already contains the learning rate. PyTorch (and most frameworks) store in gradient units and multiply by at the end. With a constant the two produce exactly the same iterates: substitute .
- the learning rate at step t, changed by a scheduler
- PyTorch's
momentum_buffer, initialised to the first gradient dampening, default 0; it scales the incoming gradient, not the stored velocity
They differ once the learning rate changes. In PyTorch's form the whole step is multiplied by the new , so a 10× drop takes effect immediately. In the classical form the velocity was built with the old learning rate and keeps carrying it, decaying by per step. In the figure with η ÷ 10 after step 30 (at ), PyTorch's step length goes from 0.119 at step 30 to 0.023 at step 31; the classical form's is still 0.116 at step 31 and 0.091 at step 35.
Dampening multiplies the incoming gradient by . With the buffer is a true EMA and the effective step is rather than , so the learning rate must be about times larger for the same behaviour. In the figure, dampening at ends at loss 0.017; at it ends at . It is not an exact match for undamped () because PyTorch skips the dampening on the very first step, so that first step is 10× larger. Dampening is rarely used; it must be 0 when nesterov=True.
1import torch23theta = torch.tensor([-4.0, 1.6], requires_grad=True)4opt = torch.optim.SGD([theta], lr=0.02, momentum=0.9) # dampening=0, nesterov=False56def f(t):7 return 0.5 * (t[0] ** 2 + 25 * t[1] ** 2)89for step in range(5):10 opt.zero_grad()11 f(theta).backward()12 opt.step() # buf = 0.9 * buf + grad (buf = grad on step 1); theta -= lr * buf1314print(theta.detach()) # ≈ (-2.997, -0.835)15print(opt.state[theta]["momentum_buffer"]) # ≈ (-15.04, -25.62)1617# Typical use on a real model, with weight decay and a schedule:18# opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)19# sched = torch.optim.lr_scheduler.MultiStepLR(opt, milestones=[30, 60], gamma=0.1)20#21# From scratch, for each parameter p with gradient g:22# buf = g.clone() if buf is None else beta * buf + (1 - dampening) * g23# p -= lr * bufUse when
- +Faster along consistent directions: up to times the plain step
- +Damps oscillation across ravines without knowing their direction
- +Tolerates larger learning rates on quadratics: instead of
- +Averages out mini-batch noise in the direction of travel
- +One extra tensor of state, half of Adam's
Avoid when
- −One more hyperparameter, coupled to the learning rate through
- −Overshoots and oscillates when β is too large for the curvature
- −Stale velocity after abrupt changes in the loss or the learning rate
- −No per-parameter scaling: badly scaled parameters still need adaptive methods
Check yourself
The gradient points the same way with the same size for many steps. With β = 0.9 and learning rate η, how far does momentum (PyTorch convention, no dampening) eventually move per step?
Check yourself
A scheduler cuts the learning rate by 10× at step 30. In which form of momentum does the step size shrink 10× at once?
Remember
- ✓, : the velocity is an exponentially weighted sum of past gradients.
- ✓Along a steady gradient the step tends to : 10× the plain step at .
- ✓Oscillating gradient components cancel in the velocity, consistent ones add: that is the ravine damping.
- ✓On a quadratic, optimally tuned momentum contracts by per step, versus for GD.
- ✓PyTorch stores v in gradient units and multiplies by the current η, so learning-rate changes act at once; the classical form keeps the old η in the velocity.
- ✓Too much momentum underdamps: the error can shrink no faster than √β per step.