RMSProp

Divide each gradient by the root of a moving average of its recent squares. Every parameter then steps about η whatever its gradient scale, and unlike AdaGrad's ever-growing sum the average forgets, so the step never decays to zero.

10 min read923 wordscore0 viewsupdated 2026-10-11

Definition

RMSProp (root mean square propagation) is an adaptive optimizer that keeps, for every parameter, an exponential moving average ss of the squared gradient and divides the gradient by s\sqrt s. The step is normalized by the gradient's recent RMS, so steep and flat directions move at comparable speeds.

RMSProp was never published as a paper. Geoffrey Hinton presented it in Lecture 6e of his 2012 Coursera course Neural Networks for Machine Learning (slides with Nitish Srivastava and Kevin Swersky), under the title "rmsprop: Divide the gradient by a running average of its recent magnitude". It is usually cited as Tieleman and Hinton (2012), from the lecture slides. Despite that informal start it became a standard optimizer. Graves (2013) used a centered variant with momentum to train handwriting-generating LSTMs, and DeepMind's DQN (Mnih et al., 2015) trained its Atari agents with RMSProp.

Hinton motivated it from rprop (Riedmiller and Braun, 1993), which uses only the sign of the full-batch gradient, with a separately adapted step size per weight. Rprop breaks with minibatches: nine small positive gradients and one large negative one should roughly cancel, but rprop's signs add up to a large positive move. Dividing every minibatch gradient by a running RMS keeps the sign-like normalization but lets magnitudes average properly. It also fixes AdaGrad (Duchi, Hazan and Singer, 2011), whose sum of all past squared gradients only grows and slowly freezes training.

The update rule

st=α st−1+(1−α) gt2,θt+1=θt−ηst+ϵ gts_t=\alpha\,s_{t-1}+(1-\alpha)\,g_t^2,\qquad \theta_{t+1}=\theta_t-\frac{\eta}{\sqrt{s_t}+\epsilon}\,g_t
gtg_t
gradient at step tt; every operation is element-wise, one ss per parameter
sts_t
moving average of g2g^2, starting at s0=0s_0=0 (PyTorch's square_avg)
α\alpha
decay rate (alpha in PyTorch, default 0.99; Hinton suggested 0.9, Keras's rho defaults to 0.9). It averages over roughly the last 1/(1−α)1/(1-\alpha) steps
η\eta
learning rate: PyTorch's default is 10−210^{-2}, Keras's is 10−310^{-3}
ϵ\epsilon
added outside the square root in PyTorch (default 10−810^{-8})
η/(st+ϵ)\eta/(\sqrt{s_t}+\epsilon)
the per-parameter effective learning rate

Compare AdaGrad, st=st−1+gt2s_t=s_{t-1}+g_t^2. Its accumulator sums every squared gradient since the start, so η/st\eta/\sqrt{s_t} falls roughly like 1/t1/\sqrt t and never recovers. RMSProp's average forgets: after about 1/(1−α)1/(1-\alpha) steps old gradients have decayed by 1/e1/e. If the gradients get smaller, ss shrinks within that window and the effective step grows back.

At a glance

Origin
Hinton, Coursera lecture 6e (2012); unpublished
State per parameter
one tensor ss; two with centered=True or with momentum>0; three with both
Step size
about η\eta per coordinate once ss has warmed up, independent of the gradient's scale
First step
η/1−α\eta/\sqrt{1-\alpha}: 10η10\eta at α=0.99\alpha=0.99. There is no bias correction
Relation to Adam
Adam with β1=0\beta_1=0, minus Adam's bias correction
PyTorch
torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-8, weight_decay=0, momentum=0, centered=False)

Why dividing by the RMS helps

If a parameter's gradient is consistently about GG in size, then s≈G2s\approx G^2 and the step is η g/s≈η sign(g)\eta\,g/\sqrt s\approx\eta\,\mathrm{sign}(g). A layer whose gradients are 10−410^{-4} and a layer whose gradients are 10210^{2} both move by about η\eta per step. In an ill-conditioned valley, the steep direction (large gradients) is slowed down and the flat direction (small gradients) is sped up, which is a crude diagonal preconditioner. It works best when the valley's axes line up with the parameter axes. The overview article's figure shows this per-coordinate method losing its edge on a rotated valley.

The step is not exactly the sign. When gradients are noisy, s\sqrt s is the RMS, which is larger than the mean magnitude, so steps along directions that keep flipping sign are smaller than η\eta. A useful identity is s≈mean(g)2+var(g)s\approx\mathrm{mean}(g)^2+\mathrm{var}(g). Plain RMSProp is therefore slowed down by gradient noise as well as by gradient size.

Two variants: centered and momentum

centered:gˉt=α gˉt−1+(1−α) gt,θt+1=θt−η gtst−gˉt 2+ϵmomentum:bt=μ bt−1+gtst+ϵ,θt+1=θt−η bt\begin{aligned}\text{centered:}\quad&\bar g_t=\alpha\,\bar g_{t-1}+(1-\alpha)\,g_t, & \theta_{t+1}&=\theta_t-\frac{\eta\,g_t}{\sqrt{s_t-\bar g_t^{\,2}}+\epsilon}\\ \text{momentum:}\quad&b_t=\mu\,b_{t-1}+\frac{g_t}{\sqrt{s_t}+\epsilon}, & \theta_{t+1}&=\theta_t-\eta\,b_t\end{aligned}
gˉt\bar g_t
moving average of the gradient itself, so st−gˉt2s_t-\bar g_t^2 estimates its variance
μ\mu
momentum coefficient (momentum in PyTorch, default 0). The buffer accumulates already-normalized steps

Centered RMSProp (Graves, 2013) divides by the gradient's standard deviation instead of its RMS. A direction with a consistent gradient has small variance, so it gets a larger step than plain RMSProp would give it. This is closer to dividing by the noise level, and it costs one more tensor per parameter. Momentum is applied after normalization: the velocity adds up normalized steps, so a consistent direction moves by up to about η/(1−μ)\eta/(1-\mu) per step. EfficientNet (Tan and Le, 2019) was trained with RMSProp using decay 0.9 and momentum 0.9.

f = ½((x−a)² + 25(y−b)²), minimum jumps at t = 100startmin 1min 2loss (log) vs step1e-61e-31η/√s: x solid, y ┄1e-41e-210100200
t = 200
● AdaGrad loss 36.0 · η/√s = (7.4e-4, 1.1e-4)
● RMSProp loss 2.8e-5 · η/√s = (0.071, 0.016)
RMSProp averages g² over roughly the last 1/(1−α) = 100 steps; AdaGrad sums every g² since step 1, so its effective rate can only fall.
fig.RMSProp (violet) and AdaGrad (orange) at the same learning rate 0.05 on the valley ½((x−a)² + 25(y−b)²), starting at (−4, 1.6). At t = 100 the minimum jumps from (0, 0) to (3, −1), like a shift in the data. RMSProp's first step is 0.5 per coordinate (10η, because nothing corrects s's zero start), and AdaGrad's is 0.05. By t = 100 RMSProp's loss is 1.8e-5 and AdaGrad's is still 12.6. The lower panel shows why: AdaGrad's effective rate along x falls from 0.0125 to 0.0015 and keeps falling (0.0007 at t = 200), while RMSProp's stays near 0.07. After the jump RMSProp gets back down from 10.2 to 2.8e-5 by t = 200, and AdaGrad only gets from 58 to 36. Try centering (3.8e-6 at t = 100), momentum 0.9, or α = 0.9.

Worked example: constant gradient vs AdaGrad

One parameter with a constant gradient g=2g=2 and η=0.01\eta=0.01. AdaGrad's step is ηg/t g2=η/t\eta g/\sqrt{t\,g^2}=\eta/\sqrt t. RMSProp's is ηg/(1−αt)g2=η/1−αt\eta g/\sqrt{(1-\alpha^t)g^2}=\eta/\sqrt{1-\alpha^t}, since st=(1−αt)g2s_t=(1-\alpha^t)g^2 from a zero start. Neither depends on gg (checked in Node, ε negligible):

step tAdaGradRMSProp α = 0.99RMSProp α = 0.9
10.01000.10000.0316
20.007070.07090.0229
100.003160.03230.0124
1000.001000.01260.0100
10000.0003160.01000.0100
100000.0001000.01000.0100
Step size per iteration under a constant gradient g = 2, η = 0.01

Three RMSProp steps with α = 0.99 when the gradient suddenly shrinks

  1. 1g1=4g_1=4: s1=0.01⋅16=0.16s_1=0.01\cdot16=0.16, s1=0.4\sqrt{s_1}=0.4, step =η⋅4/0.4=10η=\eta\cdot4/0.4=10\eta. With no bias correction, the first steps are large.
  2. 2g2=4g_2=4: s2=0.99⋅0.16+0.16=0.3184s_2=0.99\cdot0.16+0.16=0.3184, s2=0.564\sqrt{s_2}=0.564, step =7.09η=7.09\eta. The step shrinks toward η\eta as ss fills up.
  3. 3g3=0.4g_3=0.4 (ten times smaller): s3=0.3168s_3=0.3168, step =0.4/0.563 η=0.71η=0.4/0.563\,\eta=0.71\eta. The memory of the large gradients damps this step. Within roughly 100 steps ss forgets them and the step returns to about η\eta. AdaGrad's ss would remember them forever.

The table also shows something useful. For a run shorter than 1/(1−α)1/(1-\alpha) steps, αt≈1−(1−α)t\alpha^t\approx1-(1-\alpha)t and st≈(1−α)∑igi2s_t\approx(1-\alpha)\sum_i g_i^2. Over that horizon RMSProp is AdaGrad with learning rate η/1−α\eta/\sqrt{1-\alpha}. The two only part ways once the average starts forgetting.

python
1import torch
2
3opt = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.99, eps=1e-8)
4opt = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.9, momentum=0.9, centered=True)
5
6# First step under a constant gradient of 1: lr / sqrt(1 - alpha) = 10 * lr
7p = torch.zeros(1, requires_grad=True)
8opt = torch.optim.RMSprop([p], lr=0.01) # alpha=0.99
9p.sum().backward(); opt.step()
10print(p.item()) # about -0.1
11
12# The update, written out (all element-wise), as in torch.optim.RMSprop
13@torch.no_grad()
14def rmsprop_step(p, g, s, lr, alpha=0.99, eps=1e-8, momentum=0.0, buf=None, centered=False, g_avg=None):
15 s.mul_(alpha).addcmul_(g, g, value=1 - alpha) # s = alpha*s + (1-alpha)*g^2
16 if centered:
17 g_avg.mul_(alpha).add_(g, alpha=1 - alpha)
18 denom = (s - g_avg * g_avg).sqrt().add_(eps) # std instead of RMS
19 else:
20 denom = s.sqrt().add_(eps) # eps outside the sqrt
21 if momentum > 0:
22 buf.mul_(momentum).addcdiv_(g, denom) # momentum on normalized steps
23 p.sub_(lr * buf)
24 else:
25 p.addcdiv_(g, denom, value=-lr)
RMSProp in PyTorch, its variants, and the rule written out

Use when

  • +Per-parameter step of about η regardless of gradient scale: robust across layers
  • +Forgets old gradients, so the effective rate never decays to zero as in AdaGrad
  • +Adapts within about 1/(1−α) steps when the gradient scale or the target changes
  • +Cheap: one extra tensor per parameter without momentum

Avoid when

  • −No bias correction: the first steps are up to 1/1−α1/\sqrt{1-\alpha} (10×) too large
  • −Unpublished and loosely specified: ε placement, initial s and defaults differ between frameworks
  • −Per-coordinate scaling only: no help when the ill-conditioning is not axis-aligned
  • −Without momentum it does not average out gradient noise in the direction

Check yourself

Training has run for 10,000 steps with a steady gradient. AdaGrad and RMSProp (α = 0.99) started with the same learning rate η. How do their current step sizes compare?

Check yourself

What does centered=True change?

Remember

  • ✓st=αst−1+(1−α)gt2s_t=\alpha s_{t-1}+(1-\alpha)g_t^2, θ←θ−η gt/(st+ϵ)\theta\leftarrow\theta-\eta\,g_t/(\sqrt{s_t}+\epsilon): divide by the recent RMS of each parameter's gradient.
  • ✓It fixes AdaGrad's ever-shrinking step by averaging with decay instead of summing; memory is about 1/(1−α) steps.
  • ✓From Hinton's 2012 Coursera lecture 6e, unpublished. PyTorch defaults: lr 0.01, alpha 0.99, eps 1e-8 outside the sqrt.
  • ✓No bias correction: the first step is η/1−α\eta/\sqrt{1-\alpha}, 10η10\eta at α=0.99\alpha=0.99.
  • ✓Variants: centered (divide by the standard deviation) and momentum on the normalized step. Adam = RMSProp + momentum + bias correction.
← back to Deep Learning roadmap