# RMSProp

> Divide each gradient by the root of a moving average of its recent squares. Every parameter then steps about η whatever its gradient scale, and unlike AdaGrad's ever-growing sum the average forgets, so the step never decays to zero.

Source: https://www.mlfoundry.in/learn/rmsprop  
Section: Deep Learning › Optimization Algorithms  
Level: core · 10 min read · updated 2026-10-11

**RMSProp:** (root mean square propagation) is an adaptive optimizer that keeps, for every parameter, an exponential moving average $s$ of the squared gradient and divides the gradient by $\sqrt s$. The step is normalized by the gradient's recent RMS, so steep and flat directions move at comparable speeds.

RMSProp was never published as a paper. Geoffrey Hinton presented it in Lecture 6e of his 2012 Coursera course *Neural Networks for Machine Learning* (slides with Nitish Srivastava and Kevin Swersky), under the title "rmsprop: Divide the gradient by a running average of its recent magnitude". It is usually cited as Tieleman and Hinton (2012), from the lecture slides. Despite that informal start it became a standard optimizer. Graves (2013) used a centered variant with momentum to train handwriting-generating LSTMs, and DeepMind's DQN (Mnih et al., 2015) trained its Atari agents with RMSProp.

Hinton motivated it from **rprop** (Riedmiller and Braun, 1993), which uses only the **sign** of the full-batch gradient, with a separately adapted step size per weight. Rprop breaks with minibatches: nine small positive gradients and one large negative one should roughly cancel, but rprop's signs add up to a large positive move. Dividing every minibatch gradient by a running RMS keeps the sign-like normalization but lets magnitudes average properly. It also fixes **AdaGrad** (Duchi, Hazan and Singer, 2011), whose sum of all past squared gradients only grows and slowly freezes training.

## The update rule

$$
s_t=\alpha\,s_{t-1}+(1-\alpha)\,g_t^2,\qquad \theta_{t+1}=\theta_t-\frac{\eta}{\sqrt{s_t}+\epsilon}\,g_t
$$

- $g_t$: gradient at step $t$; every operation is element-wise, one $s$ per parameter
- $s_t$: moving average of $g^2$, starting at $s_0=0$ (PyTorch's `square_avg`)
- $\alpha$: decay rate (`alpha` in PyTorch, default 0.99; Hinton suggested 0.9, Keras's `rho` defaults to 0.9). It averages over roughly the last $1/(1-\alpha)$ steps
- $\eta$: learning rate: PyTorch's default is $10^{-2}$, Keras's is $10^{-3}$
- $\epsilon$: added **outside** the square root in PyTorch (default $10^{-8}$)
- $\eta/(\sqrt{s_t}+\epsilon)$: the per-parameter **effective learning rate**

Compare AdaGrad, $s_t=s_{t-1}+g_t^2$. Its accumulator sums **every** squared gradient since the start, so $\eta/\sqrt{s_t}$ falls roughly like $1/\sqrt t$ and never recovers. RMSProp's average forgets: after about $1/(1-\alpha)$ steps old gradients have decayed by $1/e$. If the gradients get smaller, $s$ shrinks within that window and the effective step grows back.

**At a glance**

- **Origin:** Hinton, Coursera lecture 6e (2012); unpublished
- **State per parameter:** one tensor $s$; two with `centered=True` or with `momentum>0`; three with both
- **Step size:** about $\eta$ per coordinate once $s$ has warmed up, independent of the gradient's scale
- **First step:** $\eta/\sqrt{1-\alpha}$: $10\eta$ at $\alpha=0.99$. There is no bias correction
- **Relation to Adam:** Adam with $\beta_1=0$, minus Adam's bias correction
- **PyTorch:** `torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-8, weight_decay=0, momentum=0, centered=False)`

## Why dividing by the RMS helps

If a parameter's gradient is consistently about $G$ in size, then $s\approx G^2$ and the step is $\eta\,g/\sqrt s\approx\eta\,\mathrm{sign}(g)$. A layer whose gradients are $10^{-4}$ and a layer whose gradients are $10^{2}$ both move by about $\eta$ per step. In an ill-conditioned valley, the steep direction (large gradients) is slowed down and the flat direction (small gradients) is sped up, which is a crude diagonal preconditioner. It works best when the valley's axes line up with the parameter axes. The overview article's figure shows this per-coordinate method losing its edge on a rotated valley.

The step is not exactly the sign. When gradients are noisy, $\sqrt s$ is the RMS, which is larger than the mean magnitude, so steps along directions that keep flipping sign are smaller than $\eta$. A useful identity is $s\approx\mathrm{mean}(g)^2+\mathrm{var}(g)$. Plain RMSProp is therefore slowed down by gradient **noise** as well as by gradient **size**.

## Two variants: centered and momentum

$$
\begin{aligned}\text{centered:}\quad&\bar g_t=\alpha\,\bar g_{t-1}+(1-\alpha)\,g_t, & \theta_{t+1}&=\theta_t-\frac{\eta\,g_t}{\sqrt{s_t-\bar g_t^{\,2}}+\epsilon}\\ \text{momentum:}\quad&b_t=\mu\,b_{t-1}+\frac{g_t}{\sqrt{s_t}+\epsilon}, & \theta_{t+1}&=\theta_t-\eta\,b_t\end{aligned}
$$

- $\bar g_t$: moving average of the gradient itself, so $s_t-\bar g_t^2$ estimates its **variance**
- $\mu$: momentum coefficient (`momentum` in PyTorch, default 0). The buffer accumulates already-normalized steps

**Centered** RMSProp (Graves, 2013) divides by the gradient's standard deviation instead of its RMS. A direction with a consistent gradient has small variance, so it gets a **larger** step than plain RMSProp would give it. This is closer to dividing by the noise level, and it costs one more tensor per parameter. **Momentum** is applied after normalization: the velocity adds up normalized steps, so a consistent direction moves by up to about $\eta/(1-\mu)$ per step. EfficientNet (Tan and Le, 2019) was trained with RMSProp using decay 0.9 and momentum 0.9.

> Figure: RMSProp (violet) and AdaGrad (orange) at the same learning rate 0.05 on the valley ½((x−a)² + 25(y−b)²), starting at (−4, 1.6). At t = 100 the minimum jumps from (0, 0) to (3, −1), like a shift in the data. RMSProp's first step is 0.5 per coordinate (10η, because nothing corrects s's zero start), and AdaGrad's is 0.05. By t = 100 RMSProp's loss is 1.8e-5 and AdaGrad's is still 12.6. The lower panel shows why: AdaGrad's effective rate along x falls from 0.0125 to 0.0015 and keeps falling (0.0007 at t = 200), while RMSProp's stays near 0.07. After the jump RMSProp gets back down from 10.2 to 2.8e-5 by t = 200, and AdaGrad only gets from 58 to 36. Try centering (3.8e-6 at t = 100), momentum 0.9, or α = 0.9.

## Worked example: constant gradient vs AdaGrad

One parameter with a constant gradient $g=2$ and $\eta=0.01$. AdaGrad's step is $\eta g/\sqrt{t\,g^2}=\eta/\sqrt t$. RMSProp's is $\eta g/\sqrt{(1-\alpha^t)g^2}=\eta/\sqrt{1-\alpha^t}$, since $s_t=(1-\alpha^t)g^2$ from a zero start. Neither depends on $g$ (checked in Node, ε negligible):

Step size per iteration under a constant gradient g = 2, η = 0.01

| step t | AdaGrad | RMSProp α = 0.99 | RMSProp α = 0.9 |
| --- | --- | --- | --- |
| 1 | 0.0100 | 0.1000 | 0.0316 |
| 2 | 0.00707 | 0.0709 | 0.0229 |
| 10 | 0.00316 | 0.0323 | 0.0124 |
| 100 | 0.00100 | 0.0126 | 0.0100 |
| 1000 | 0.000316 | 0.0100 | 0.0100 |
| 10000 | 0.000100 | 0.0100 | 0.0100 |

**Three RMSProp steps with α = 0.99 when the gradient suddenly shrinks**

1. $g_1=4$: $s_1=0.01\cdot16=0.16$, $\sqrt{s_1}=0.4$, step $=\eta\cdot4/0.4=10\eta$. With no bias correction, the first steps are large.
2. $g_2=4$: $s_2=0.99\cdot0.16+0.16=0.3184$, $\sqrt{s_2}=0.564$, step $=7.09\eta$. The step shrinks toward $\eta$ as $s$ fills up.
3. $g_3=0.4$ (ten times smaller): $s_3=0.3168$, step $=0.4/0.563\,\eta=0.71\eta$. The memory of the large gradients damps this step. Within roughly 100 steps $s$ forgets them and the step returns to about $\eta$. AdaGrad's $s$ would remember them forever.

The table also shows something useful. For a run shorter than $1/(1-\alpha)$ steps, $\alpha^t\approx1-(1-\alpha)t$ and $s_t\approx(1-\alpha)\sum_i g_i^2$. Over that horizon RMSProp **is** AdaGrad with learning rate $\eta/\sqrt{1-\alpha}$. The two only part ways once the average starts forgetting.

```python
import torch

opt = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.99, eps=1e-8)
opt = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.9, momentum=0.9, centered=True)

# First step under a constant gradient of 1: lr / sqrt(1 - alpha) = 10 * lr
p = torch.zeros(1, requires_grad=True)
opt = torch.optim.RMSprop([p], lr=0.01)       # alpha=0.99
p.sum().backward(); opt.step()
print(p.item())                               # about -0.1

# The update, written out (all element-wise), as in torch.optim.RMSprop
@torch.no_grad()
def rmsprop_step(p, g, s, lr, alpha=0.99, eps=1e-8, momentum=0.0, buf=None, centered=False, g_avg=None):
    s.mul_(alpha).addcmul_(g, g, value=1 - alpha)            # s = alpha*s + (1-alpha)*g^2
    if centered:
        g_avg.mul_(alpha).add_(g, alpha=1 - alpha)
        denom = (s - g_avg * g_avg).sqrt().add_(eps)         # std instead of RMS
    else:
        denom = s.sqrt().add_(eps)                           # eps outside the sqrt
    if momentum > 0:
        buf.mul_(momentum).addcdiv_(g, denom)                # momentum on normalized steps
        p.sub_(lr * buf)
    else:
        p.addcdiv_(g, denom, value=-lr)
```

RMSProp in PyTorch, its variants, and the rule written out

> **Frameworks disagree on the details:** PyTorch adds $\epsilon=10^{-8}$ **outside** the square root, starts $s$ at 0 and defaults to $\alpha=0.99$, $\eta=10^{-2}$. TensorFlow 1's RMSProp put $\epsilon$ (default $10^{-10}$) **inside** the square root and started $s$ at 1, and Keras defaults to `rho=0.9` and `learning_rate=1e-3`. These are not cosmetic. When gradients are tiny, $\epsilon$ decides the step size. Starting $s$ at 0 makes the first steps $1/\sqrt{1-\alpha}$ times too large; starting it at 1 makes them tiny when gradients are much smaller than 1. To reproduce a TF recipe (EfficientNet, MobileNet) in PyTorch, use a TF-compatible variant such as timm's `RMSpropTF`.

> **RMSProp is Adam without momentum or bias correction:** Set Adam's $\beta_1=0$: then $\hat m_t=g_t$, and Adam's $v_t$ is exactly RMSProp's $s_t$ with $\beta_2=\alpha$. The only remaining difference is Adam's bias correction $v_t/(1-\beta_2^t)$, which removes RMSProp's oversized early steps. Adam is, almost literally, RMSProp plus momentum plus that correction.

> **When to reach for it:** RMSProp is still common in reinforcement learning, where non-stationary targets reward a short memory, and in older recurrent-network code. For most new supervised work, Adam or AdamW does the same normalization with momentum and fewer surprises at step 1. Momentum-free RMSProp also stores one tensor fewer than Adam, which can matter at scale.

**Strengths**

- Per-parameter step of about η regardless of gradient scale: robust across layers
- Forgets old gradients, so the effective rate never decays to zero as in AdaGrad
- Adapts within about 1/(1−α) steps when the gradient scale or the target changes
- Cheap: one extra tensor per parameter without momentum

**Limitations**

- No bias correction: the first steps are up to $1/\sqrt{1-\alpha}$ (10×) too large
- Unpublished and loosely specified: ε placement, initial s and defaults differ between frameworks
- Per-coordinate scaling only: no help when the ill-conditioning is not axis-aligned
- Without momentum it does not average out gradient noise in the direction

**Question:** Training has run for 10,000 steps with a steady gradient. AdaGrad and RMSProp (α = 0.99) started with the same learning rate η. How do their current step sizes compare?

- A. Both are about η
- B. RMSProp's is about η; AdaGrad's is about η/100
- C. AdaGrad's is about η; RMSProp's is about η/100
- D. Both have decayed to about η/100

**Answer:** B. RMSProp's is about η; AdaGrad's is about η/100 — AdaGrad's accumulator is $\sum g^2=t\,g^2$, so its step is $\eta/\sqrt t=\eta/100$ at $t=10^4$. RMSProp's average settles at $g^2$, so its step stays about $\eta$. Forgetting is the whole point of RMSProp.

**Question:** What does centered=True change?

- A. It subtracts the running mean of the parameters
- B. It divides by an estimate of the gradient's standard deviation, √(s − ḡ²), instead of its RMS
- C. It applies bias correction to s
- D. It centres the learning-rate schedule

**Answer:** B. It divides by an estimate of the gradient's standard deviation, √(s − ḡ²), instead of its RMS — It also tracks $\bar g$, the moving average of $g$, and uses $s-\bar g^2\approx\mathrm{Var}(g)$. Consistent directions (low variance) then get larger steps than plain RMSProp would give them. It costs one more state tensor.

**Key points**

- $s_t=\alpha s_{t-1}+(1-\alpha)g_t^2$, $\theta\leftarrow\theta-\eta\,g_t/(\sqrt{s_t}+\epsilon)$: divide by the recent RMS of each parameter's gradient.
- It fixes AdaGrad's ever-shrinking step by averaging with decay instead of summing; memory is about 1/(1−α) steps.
- From Hinton's 2012 Coursera lecture 6e, unpublished. PyTorch defaults: lr 0.01, alpha 0.99, eps 1e-8 outside the sqrt.
- No bias correction: the first step is $\eta/\sqrt{1-\alpha}$, $10\eta$ at $\alpha=0.99$.
- Variants: centered (divide by the standard deviation) and momentum on the normalized step. Adam = RMSProp + momentum + bias correction.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
