Definition
RMSProp (root mean square propagation) is an adaptive optimizer that keeps, for every parameter, an exponential moving average of the squared gradient and divides the gradient by . The step is normalized by the gradient's recent RMS, so steep and flat directions move at comparable speeds.
RMSProp was never published as a paper. Geoffrey Hinton presented it in Lecture 6e of his 2012 Coursera course Neural Networks for Machine Learning (slides with Nitish Srivastava and Kevin Swersky), under the title "rmsprop: Divide the gradient by a running average of its recent magnitude". It is usually cited as Tieleman and Hinton (2012), from the lecture slides. Despite that informal start it became a standard optimizer. Graves (2013) used a centered variant with momentum to train handwriting-generating LSTMs, and DeepMind's DQN (Mnih et al., 2015) trained its Atari agents with RMSProp.
Hinton motivated it from rprop (Riedmiller and Braun, 1993), which uses only the sign of the full-batch gradient, with a separately adapted step size per weight. Rprop breaks with minibatches: nine small positive gradients and one large negative one should roughly cancel, but rprop's signs add up to a large positive move. Dividing every minibatch gradient by a running RMS keeps the sign-like normalization but lets magnitudes average properly. It also fixes AdaGrad (Duchi, Hazan and Singer, 2011), whose sum of all past squared gradients only grows and slowly freezes training.
The update rule
- gradient at step ; every operation is element-wise, one per parameter
- moving average of , starting at (PyTorch's
square_avg) - decay rate (
alphain PyTorch, default 0.99; Hinton suggested 0.9, Keras'srhodefaults to 0.9). It averages over roughly the last steps - learning rate: PyTorch's default is , Keras's is
- added outside the square root in PyTorch (default )
- the per-parameter effective learning rate
Compare AdaGrad, . Its accumulator sums every squared gradient since the start, so falls roughly like and never recovers. RMSProp's average forgets: after about steps old gradients have decayed by . If the gradients get smaller, shrinks within that window and the effective step grows back.
At a glance
- Origin
- Hinton, Coursera lecture 6e (2012); unpublished
- State per parameter
- one tensor ; two with
centered=Trueor withmomentum>0; three with both - Step size
- about per coordinate once has warmed up, independent of the gradient's scale
- First step
- : at . There is no bias correction
- Relation to Adam
- Adam with , minus Adam's bias correction
- PyTorch
torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-8, weight_decay=0, momentum=0, centered=False)
Why dividing by the RMS helps
If a parameter's gradient is consistently about in size, then and the step is . A layer whose gradients are and a layer whose gradients are both move by about per step. In an ill-conditioned valley, the steep direction (large gradients) is slowed down and the flat direction (small gradients) is sped up, which is a crude diagonal preconditioner. It works best when the valley's axes line up with the parameter axes. The overview article's figure shows this per-coordinate method losing its edge on a rotated valley.
The step is not exactly the sign. When gradients are noisy, is the RMS, which is larger than the mean magnitude, so steps along directions that keep flipping sign are smaller than . A useful identity is . Plain RMSProp is therefore slowed down by gradient noise as well as by gradient size.
Two variants: centered and momentum
- moving average of the gradient itself, so estimates its variance
- momentum coefficient (
momentumin PyTorch, default 0). The buffer accumulates already-normalized steps
Centered RMSProp (Graves, 2013) divides by the gradient's standard deviation instead of its RMS. A direction with a consistent gradient has small variance, so it gets a larger step than plain RMSProp would give it. This is closer to dividing by the noise level, and it costs one more tensor per parameter. Momentum is applied after normalization: the velocity adds up normalized steps, so a consistent direction moves by up to about per step. EfficientNet (Tan and Le, 2019) was trained with RMSProp using decay 0.9 and momentum 0.9.
Worked example: constant gradient vs AdaGrad
One parameter with a constant gradient and . AdaGrad's step is . RMSProp's is , since from a zero start. Neither depends on (checked in Node, ε negligible):
| step t | AdaGrad | RMSProp α = 0.99 | RMSProp α = 0.9 |
|---|---|---|---|
| 1 | 0.0100 | 0.1000 | 0.0316 |
| 2 | 0.00707 | 0.0709 | 0.0229 |
| 10 | 0.00316 | 0.0323 | 0.0124 |
| 100 | 0.00100 | 0.0126 | 0.0100 |
| 1000 | 0.000316 | 0.0100 | 0.0100 |
| 10000 | 0.000100 | 0.0100 | 0.0100 |
Three RMSProp steps with α = 0.99 when the gradient suddenly shrinks
- 1: , , step . With no bias correction, the first steps are large.
- 2: , , step . The step shrinks toward as fills up.
- 3 (ten times smaller): , step . The memory of the large gradients damps this step. Within roughly 100 steps forgets them and the step returns to about . AdaGrad's would remember them forever.
The table also shows something useful. For a run shorter than steps, and . Over that horizon RMSProp is AdaGrad with learning rate . The two only part ways once the average starts forgetting.
1import torch23opt = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.99, eps=1e-8)4opt = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.9, momentum=0.9, centered=True)56# First step under a constant gradient of 1: lr / sqrt(1 - alpha) = 10 * lr7p = torch.zeros(1, requires_grad=True)8opt = torch.optim.RMSprop([p], lr=0.01) # alpha=0.999p.sum().backward(); opt.step()10print(p.item()) # about -0.11112# The update, written out (all element-wise), as in torch.optim.RMSprop13@torch.no_grad()14def rmsprop_step(p, g, s, lr, alpha=0.99, eps=1e-8, momentum=0.0, buf=None, centered=False, g_avg=None):15 s.mul_(alpha).addcmul_(g, g, value=1 - alpha) # s = alpha*s + (1-alpha)*g^216 if centered:17 g_avg.mul_(alpha).add_(g, alpha=1 - alpha)18 denom = (s - g_avg * g_avg).sqrt().add_(eps) # std instead of RMS19 else:20 denom = s.sqrt().add_(eps) # eps outside the sqrt21 if momentum > 0:22 buf.mul_(momentum).addcdiv_(g, denom) # momentum on normalized steps23 p.sub_(lr * buf)24 else:25 p.addcdiv_(g, denom, value=-lr)Use when
- +Per-parameter step of about η regardless of gradient scale: robust across layers
- +Forgets old gradients, so the effective rate never decays to zero as in AdaGrad
- +Adapts within about 1/(1−α) steps when the gradient scale or the target changes
- +Cheap: one extra tensor per parameter without momentum
Avoid when
- −No bias correction: the first steps are up to (10×) too large
- −Unpublished and loosely specified: ε placement, initial s and defaults differ between frameworks
- −Per-coordinate scaling only: no help when the ill-conditioning is not axis-aligned
- −Without momentum it does not average out gradient noise in the direction
Check yourself
Training has run for 10,000 steps with a steady gradient. AdaGrad and RMSProp (α = 0.99) started with the same learning rate η. How do their current step sizes compare?
Check yourself
What does centered=True change?
Remember
- ✓, : divide by the recent RMS of each parameter's gradient.
- ✓It fixes AdaGrad's ever-shrinking step by averaging with decay instead of summing; memory is about 1/(1−α) steps.
- ✓From Hinton's 2012 Coursera lecture 6e, unpublished. PyTorch defaults: lr 0.01, alpha 0.99, eps 1e-8 outside the sqrt.
- ✓No bias correction: the first step is , at .
- ✓Variants: centered (divide by the standard deviation) and momentum on the normalized step. Adam = RMSProp + momentum + bias correction.