# Momentum

> Keep a velocity, an exponentially weighted sum of past gradients, and step along it. Oscillating components cancel, consistent ones build up to η/(1−β) per step, so ravines are crossed quickly instead of zigzagged.

Source: https://www.mlfoundry.in/learn/momentum  
Section: Deep Learning › Optimization Algorithms  
Level: core · 11 min read · updated 2026-10-11

**Momentum:** (the **heavy-ball method**) replaces the gradient step $\theta\leftarrow\theta-\eta g$ with a step along a **velocity** $v$ that accumulates past gradients: $v\leftarrow\beta v+g$, then $\theta\leftarrow\theta-\eta v$. With $\beta=0$ it is plain gradient descent; with $\beta$ near 1 the update remembers many past gradients.

Plain gradient descent has no memory. Every step uses only the current gradient, so in a long narrow valley it bounces from wall to wall while creeping along the floor. The learning rate cannot fix this: the steep walls cap it, and the flat floor then gets a tiny step. Momentum gives the iterate inertia. Gradient components that keep pointing the same way add up step after step; components that flip sign every few steps cancel out.

The method is Boris Polyak's, from "Some methods of speeding up the convergence of iteration methods" (USSR Computational Mathematics and Mathematical Physics, 1964). The name comes from his physical picture: a heavy ball rolling on the loss surface under gravity and friction keeps moving through flat stretches and is not thrown around by every bump. Rumelhart, Hinton and Williams included a momentum term in their 1986 backpropagation paper, and Sutskever, Martens, Dahl and Hinton ("On the importance of initialization and momentum in deep learning", ICML 2013) showed that well-tuned momentum, with $\beta$ raised gradually toward 0.99, trains deep and recurrent networks that plain SGD could not.

## The update rule

$$
v_{t+1}=\beta\,v_t+g_t,\qquad \theta_{t+1}=\theta_t-\eta\,v_{t+1},\qquad g_t=\nabla_\theta L(\theta_t)
$$

- $v_t$: velocity, in gradient units; $v_0=0$ (PyTorch sets $v_1=g_0$ on the first step)
- $\beta$: momentum coefficient in $[0,1)$; $0.9$ is the usual default
- $\eta$: learning rate
- $g_t$: the (mini-batch) gradient at the current parameters

Unrolling the recursion shows what the velocity is: an exponentially weighted sum of every gradient so far, with weight $\beta^k$ on the gradient from $k$ steps ago.

$$
v_{t+1}=\sum_{k=0}^{t}\beta^{k}\,g_{t-k}\qquad\Rightarrow\qquad \text{if } g_t\equiv g:\quad v_t\to\frac{g}{1-\beta},\quad \Delta\theta\to-\frac{\eta}{1-\beta}\,g
$$

- $\beta^k$: weight of a gradient $k$ steps old; it falls to $1/e$ after about $1/(1-\beta)$ steps (10 steps for $\beta=0.9$)
- $\eta/(1-\beta)$: the **effective step** in a direction where the gradient is steady: $10\eta$ at $\beta=0.9$, $100\eta$ at $\beta=0.99$

Written with weights $(1-\beta)\beta^k$ instead of $\beta^k$, the same sum is an **exponential moving average** (EMA) of the gradients. That is the form Adam uses for its first moment. The two forms differ only by the constant factor $1-\beta$, which can be folded into the learning rate.

**At a glance**

- **State:** one extra tensor per parameter (the velocity)
- **Hyperparameters:** $\eta$, $\beta$ (default $0.9$), optional dampening $d$
- **Effective step:** $\eta(1-d)/(1-\beta)$ along a constant gradient
- **Stability on a quadratic:** $\eta\lambda_{\max}(1-d)<2(1+\beta)$, versus $\eta\lambda_{\max}<2$ for plain GD
- **Best rate on a quadratic:** $(\sqrt\kappa-1)/(\sqrt\kappa+1)$ per step, versus $(\kappa-1)/(\kappa+1)$ for plain GD
- **PyTorch:** `torch.optim.SGD(params, lr, momentum=0.9, dampening=0, nesterov=False)`

> Figure: Momentum (orange) and plain gradient descent (grey) with the same learning rate on f = ½(x² + 25y²). At the defaults η = 0.02, β = 0.9, GD has only reached loss 0.063 after 120 steps while momentum is at 3.8e-5. The velocity panel shows why: the across-valley component v_y swings between +56 and −44 and averages out, while the along-valley component v_x grows from −4 to −18.3 within 9 steps. Drag β to 0.99 and the ball overshoots for the whole run (loss 7.0); push η past the printed limit 2(1+β)/25 = 0.152 and it diverges. 'Polyak tuning' sets η = 0.111, β = 0.444, which crosses loss 1e-3 at step 22. Turn on the η ÷ 10 drop to see the two conventions part ways.

## Why it damps ravines

Take a quadratic valley $f=\tfrac12(x^2+\kappa y^2)$, so the curvature is 1 along $x$ and $\kappa$ across. Gradient descent is stable only if $\eta<2/\kappa$, and then the $x$ coordinate shrinks by $1-\eta$ per step: with $\kappa=25$ and $\eta=0.02$, a factor 0.98, which takes about 300 steps to shrink by $e^{-6}$.

Momentum treats the two directions differently without being told which is which. Along $x$ the gradient keeps its sign, so successive gradients add and the step grows toward $\eta/(1-\beta)$ times the gradient: ten times larger at $\beta=0.9$. Across $y$, the iterate overshoots the floor and the gradient flips sign every few steps, so the contributions to $v_y$ largely cancel. The ball keeps its speed along the valley and loses it across.

On a quadratic each eigendirection with curvature $\lambda$ obeys the linear recurrence $\theta_{t+1}=(1+\beta-\eta\lambda)\theta_t-\beta\,\theta_{t-1}$. It is stable exactly when $0<\eta\lambda<2(1+\beta)$, so momentum also tolerates a larger learning rate: up to 0.152 instead of 0.08 for $\kappa=25$ and $\beta=0.9$. Polyak showed that the choice $\beta=\big(\tfrac{\sqrt\kappa-1}{\sqrt\kappa+1}\big)^2$, $\eta=\tfrac{4}{(\sqrt{\lambda_{\max}}+\sqrt{\lambda_{\min}})^2}$ makes every direction contract by $\tfrac{\sqrt\kappa-1}{\sqrt\kappa+1}$ per step. For $\kappa=25$ that is $0.667$, against $0.923$ for the best plain-GD step: the number of steps needed scales with $\sqrt\kappa$ instead of $\kappa$.

$$
\rho_{\text{GD}}=\frac{\kappa-1}{\kappa+1}\qquad\text{vs.}\qquad \rho_{\text{HB}}=\frac{\sqrt\kappa-1}{\sqrt\kappa+1},\qquad \kappa=\frac{\lambda_{\max}}{\lambda_{\min}}
$$

- $\rho$: worst-case contraction of the error per step, with optimal tuning, on a quadratic
- $\kappa$: condition number; at $\kappa=10^4$, $\rho_{\text{GD}}=0.9998$ but $\rho_{\text{HB}}=0.980$

> **Too much momentum is underdamped:** More momentum is not always better. With $\eta=0.02$ on the figure's valley, $\beta=0.8$ reaches loss $8\times10^{-11}$ in 120 steps, $\beta=0.9$ reaches $3.8\times10^{-5}$, and $\beta=0.95$ only $0.076$, worse than plain GD at $\beta=0$. Above the critical value the recurrence has complex roots of modulus $\sqrt\beta$, so the error spirals inward at a rate no better than $\sqrt\beta$ per step however the learning rate is set: 0.975 for $\beta=0.95$, 0.995 for $\beta=0.99$.

## Worked example: the first five steps

The figure's valley $f=\tfrac12(x^2+25y^2)$, start $\theta_0=(-4,\,1.6)$, $f=40$, gradient $(-4,\,40)$, $\eta=0.02$, $\beta=0.9$, PyTorch convention. Values computed by the figure's code.

Momentum vs plain GD, same learning rate

| t | momentum θ | velocity v | momentum loss | GD θ | GD loss |
| --- | --- | --- | --- | --- | --- |
| 1 | (−3.920, 0.800) | (−4.00, 40.0) | 15.68 | (−3.920, 0.800) | 15.68 |
| 2 | (−3.770, −0.320) | (−7.52, 56.0) | 8.38 | (−3.842, 0.400) | 9.38 |
| 3 | (−3.559, −1.168) | (−10.54, 42.4) | 23.39 | (−3.765, 0.200) | 7.59 |
| 4 | (−3.298, −1.347) | (−13.04, 8.96) | 28.13 | (−3.690, 0.100) | 6.93 |
| 5 | (−2.997, −0.835) | (−15.04, −25.6) | 13.20 | (−3.616, 0.050) | 6.57 |

1. Step 1 is identical for both: the buffer starts as $v_1=g_0=(-4,40)$, so $\Delta\theta=-0.02\,(-4,40)=(0.08,-0.8)$.
2. Step 2: $g_1=(-3.92,\,20)$, so $v_2=0.9\,(-4,40)+(-3.92,20)=(-7.52,\,56)$. The stored $y$-velocity carries the iterate past the floor to $y=-0.32$, while GD halves $y$ to $0.4$.
3. Steps 3 to 5: the $y$-velocity is still positive, so $y$ keeps falling to $-1.35$ and the loss rises to 28. The new negative gradients brake it, and by step 5 $v_y$ has flipped to $-25.6$. That is the swing the velocity panel shows.
4. Meanwhile $v_x$ grows every step, $-4\to-15.0$, and momentum has already moved $x$ by $1.0$ against GD's $0.38$. After 10 steps the losses are 1.00 vs 5.34; after 40 steps 0.024 vs 1.59.

## Two conventions, and dampening

Papers usually write the classical form $v_{t+1}=\beta v_t-\eta g_t$, $\theta_{t+1}=\theta_t+v_{t+1}$, where the velocity is in parameter units and already contains the learning rate. PyTorch (and most frameworks) store $v$ in gradient units and multiply by $\eta$ at the end. With a constant $\eta$ the two produce exactly the same iterates: substitute $v^{\text{cls}}=-\eta v$.

$$
\underbrace{v\leftarrow\beta v-\eta_t g,\;\; \theta\leftarrow\theta+v}_{\text{classical (Sutskever et al. 2013)}}\qquad\qquad \underbrace{b\leftarrow\beta b+(1-d)\,g,\;\; \theta\leftarrow\theta-\eta_t\,b}_{\text{PyTorch SGD}}
$$

- $\eta_t$: the learning rate at step t, changed by a scheduler
- $b$: PyTorch's `momentum_buffer`, initialised to the first gradient
- $d$: `dampening`, default 0; it scales the incoming gradient, not the stored velocity

They differ once the learning rate changes. In PyTorch's form the whole step is multiplied by the new $\eta_t$, so a 10× drop takes effect immediately. In the classical form the velocity was built with the old learning rate and keeps carrying it, decaying by $\beta$ per step. In the figure with η ÷ 10 after step 30 (at $\eta=0.02$), PyTorch's step length goes from 0.119 at step 30 to 0.023 at step 31; the classical form's is still 0.116 at step 31 and 0.091 at step 35.

**Dampening** multiplies the incoming gradient by $1-d$. With $d=\beta$ the buffer is a true EMA and the effective step is $\eta$ rather than $\eta/(1-\beta)$, so the learning rate must be about $1/(1-\beta)$ times larger for the same behaviour. In the figure, dampening $0.9$ at $\eta=0.02$ ends at loss 0.017; at $\eta=0.2$ it ends at $3.0\times10^{-3}$. It is not an exact match for undamped $\eta=0.02$ ($3.8\times10^{-5}$) because PyTorch skips the dampening on the very first step, so that first step is 10× larger. Dampening is rarely used; it must be 0 when `nesterov=True`.

```python
import torch

theta = torch.tensor([-4.0, 1.6], requires_grad=True)
opt = torch.optim.SGD([theta], lr=0.02, momentum=0.9)   # dampening=0, nesterov=False

def f(t):
    return 0.5 * (t[0] ** 2 + 25 * t[1] ** 2)

for step in range(5):
    opt.zero_grad()
    f(theta).backward()
    opt.step()           # buf = 0.9 * buf + grad (buf = grad on step 1); theta -= lr * buf

print(theta.detach())                          # ≈ (-2.997, -0.835)
print(opt.state[theta]["momentum_buffer"])     # ≈ (-15.04, -25.62)

# Typical use on a real model, with weight decay and a schedule:
# opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
# sched = torch.optim.lr_scheduler.MultiStepLR(opt, milestones=[30, 60], gamma=0.1)
#
# From scratch, for each parameter p with gradient g:
#   buf = g.clone() if buf is None else beta * buf + (1 - dampening) * g
#   p  -= lr * buf
```

torch.optim.SGD with momentum on the figure's valley (same five steps as the table)

> **Changing β changes the effective learning rate:** Because the steady-state step is $\eta/(1-\beta)$, going from $\beta=0.9$ to $0.99$ makes the step along consistent directions 10× larger at the same $\eta$. A learning rate tuned for one $\beta$ is wrong for another; when you raise $\beta$, lower $\eta$ roughly in proportion to $1-\beta$. The same applies when porting a recipe between a framework that uses the EMA form and one that uses the sum.

> **Velocity carries stale information:** The velocity remembers about $1/(1-\beta)$ steps of history. After a sharp change (a learning-rate restart, a new data distribution, a loss spike), it keeps pushing in the old direction for that long. That is also why momentum overshoots minima, and why the loss curve of SGD with momentum often rises briefly before it falls.

> **Guarantees beyond quadratics:** The $\sqrt\kappa$ speed-up is proved for quadratics. For general smooth strongly convex functions the heavy-ball method with constant parameters is not guaranteed to converge at that rate; Lessard, Recht and Packard (2016) give a strongly convex function on which Polyak's tuning cycles forever. Nesterov's accelerated gradient, the next article, is the variant with a global guarantee.

**Strengths**

- Faster along consistent directions: up to $1/(1-\beta)$ times the plain step
- Damps oscillation across ravines without knowing their direction
- Tolerates larger learning rates on quadratics: $2(1+\beta)/\lambda_{\max}$ instead of $2/\lambda_{\max}$
- Averages out mini-batch noise in the direction of travel
- One extra tensor of state, half of Adam's

**Limitations**

- One more hyperparameter, coupled to the learning rate through $\eta/(1-\beta)$
- Overshoots and oscillates when β is too large for the curvature
- Stale velocity after abrupt changes in the loss or the learning rate
- No per-parameter scaling: badly scaled parameters still need adaptive methods

**Question:** The gradient points the same way with the same size for many steps. With β = 0.9 and learning rate η, how far does momentum (PyTorch convention, no dampening) eventually move per step?

- A. η·|g|
- B. 0.9·η·|g|
- C. 10·η·|g|
- D. 100·η·|g|

**Answer:** C. 10·η·|g| — The velocity tends to $g\sum_k\beta^k=g/(1-\beta)=10g$, so the step tends to $10\eta|g|$. At $\beta=0.99$ it would be $100\eta|g|$, which is why raising $\beta$ needs a smaller learning rate.

**Question:** A scheduler cuts the learning rate by 10× at step 30. In which form of momentum does the step size shrink 10× at once?

- A. Both forms, they are identical
- B. PyTorch's form, v ← βv + g, θ ← θ − ηv
- C. The classical form, v ← βv − ηg, θ ← θ + v
- D. Neither: momentum ignores the learning rate

**Answer:** B. PyTorch's form, v ← βv + g, θ ← θ − ηv — PyTorch multiplies the stored velocity by the current $\eta$, so the whole step is rescaled immediately. The classical velocity already contains the old $\eta$ and only forgets it at rate $\beta$ per step. At a constant learning rate the two forms give identical iterates.

**Key points**

- $v\leftarrow\beta v+g$, $\theta\leftarrow\theta-\eta v$: the velocity is an exponentially weighted sum of past gradients.
- Along a steady gradient the step tends to $\eta/(1-\beta)$: 10× the plain step at $\beta=0.9$.
- Oscillating gradient components cancel in the velocity, consistent ones add: that is the ravine damping.
- On a quadratic, optimally tuned momentum contracts by $(\sqrt\kappa-1)/(\sqrt\kappa+1)$ per step, versus $(\kappa-1)/(\kappa+1)$ for GD.
- PyTorch stores v in gradient units and multiplies by the current η, so learning-rate changes act at once; the classical form keeps the old η in the velocity.
- Too much momentum underdamps: the error can shrink no faster than √β per step.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
