# AdaDelta

> AdaGrad without a learning rate: each step is an RMS of recent updates divided by an RMS of recent gradients, so it has the parameter's units. The catch is that ε then decides how big the first steps are, so it acts as the learning rate.

Source: https://www.mlfoundry.in/learn/adadelta  
Section: Deep Learning › Optimization Algorithms  
Level: advanced · 10 min read · updated 2026-10-11

**AdaDelta:** is an adaptive optimizer that scales each parameter's gradient by $\mathrm{RMS}[\Delta\theta]_{t-1}/\mathrm{RMS}[g]_t$. Both are exponential moving averages: one of past **updates**, one of past **gradients**. The ratio stands in for a per-parameter learning rate, so the method has no global step size $\eta$.

Matthew Zeiler introduced AdaDelta in a 2012 arXiv note, *ADADELTA: An Adaptive Learning Rate Method*. He wanted to fix two things about **AdaGrad** (Duchi, Hazan and Singer, 2011). First, AdaGrad divides by the square root of a sum of **all** past squared gradients, so its effective learning rate can only fall and training eventually stalls. Second, AdaGrad still needs a hand-tuned global learning rate.

AdaDelta fixes the first problem the same way RMSProp does: it replaces the growing sum with a decaying average. For the second, it uses a units argument. The update $-\eta g$ has the units of a gradient, not of the parameter. Newton's step $-H^{-1}g$ does have the parameter's units, and AdaDelta builds a cheap diagonal stand-in for $H^{-1}$ out of quantities it already tracks.

## The update rule

$$
\begin{aligned}E[g^2]_t&=\rho\,E[g^2]_{t-1}+(1-\rho)\,g_t^2\\ \Delta\theta_t&=-\frac{\sqrt{E[\Delta\theta^2]_{t-1}+\epsilon}}{\sqrt{E[g^2]_t+\epsilon}}\;g_t=-\frac{\mathrm{RMS}[\Delta\theta]_{t-1}}{\mathrm{RMS}[g]_t}\,g_t\\ E[\Delta\theta^2]_t&=\rho\,E[\Delta\theta^2]_{t-1}+(1-\rho)\,\Delta\theta_t^2\\ \theta_{t+1}&=\theta_t+\Delta\theta_t\end{aligned}
$$

- $g_t$: the gradient at step $t$; every operation is element-wise, one copy per parameter
- $E[g^2]$: decaying average of squared gradients (RMSProp's accumulator); starts at 0
- $E[\Delta\theta^2]$: decaying average of squared **updates**; starts at 0
- $\rho$: decay rate. Zeiler used 0.95 on MNIST; PyTorch's default is 0.9
- $\epsilon$: added **inside** both square roots. Zeiler used $10^{-6}$ on MNIST, which is also PyTorch's default
- $\mathrm{RMS}[x]$: $\sqrt{E[x^2]+\epsilon}$

The numerator uses $t-1$ because $\Delta\theta_t$ is not known until the step has been computed. That lag is also what makes AdaDelta start slowly: at $t=1$ the numerator is $\sqrt{0+\epsilon}=\sqrt\epsilon$.

PyTorch's `torch.optim.Adadelta` adds one knob the paper does not have: `lr` (default 1.0) multiplies $\Delta\theta_t$ just before it is applied. The accumulator $E[\Delta\theta^2]$ still stores the **unscaled** $\Delta\theta_t$. With `lr=1.0` you get the paper's method. Keras's `Adadelta` defaults to `learning_rate=0.001`, so you have to pass 1.0 explicitly to get the original algorithm.

**At a glance**

- **Paper:** Zeiler, arXiv:1212.5701 (Dec 2012)
- **State per parameter:** two tensors, $E[g^2]$ and $E[\Delta\theta^2]$: the same memory as Adam
- **Hyperparameters:** $\rho$ and $\epsilon$; PyTorch adds an `lr` multiplier (default 1.0)
- **Units of the step:** those of $\theta$ (if $\epsilon$ is negligible), unlike SGD, AdaGrad and RMSProp
- **First step:** $|\Delta\theta_1|\approx\sqrt{\epsilon/(1-\rho)}$ for any gradient with $(1-\rho)g^2\gg\epsilon$: $0.0032$ at the defaults
- **PyTorch:** `torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-6, weight_decay=0)`

## Where the ratio comes from: a units argument

Give $\theta$ the unit "metres" and treat the loss as unitless. Then $g=\partial L/\partial\theta$ is in 1/metres, and an SGD step $\eta g$ is in 1/metres too unless $\eta$ is in metres². The learning rate quietly carries the conversion. AdaGrad and RMSProp divide $g$ by an RMS of $g$, so their step is unitless, and again $\eta$ has to supply the metres.

Newton's method gets the units right on its own. Its step $\Delta\theta=-g/H$ divides 1/metres by 1/metres², which gives metres. Rearranging Newton's step gives an expression for the inverse curvature that uses only first-order quantities:

$$
\Delta\theta=-\frac{\partial L/\partial\theta}{\partial^2L/\partial\theta^2}\quad\Longrightarrow\quad\frac{1}{\partial^2L/\partial\theta^2}=-\frac{\Delta\theta}{\partial L/\partial\theta}\;\approx\;\frac{\mathrm{RMS}[\Delta\theta]_{t-1}}{\mathrm{RMS}[g]_t}
$$

- $\partial^2L/\partial\theta^2$: the diagonal curvature, which AdaDelta never computes
- $\mathrm{RMS}[\Delta\theta]_{t-1}/\mathrm{RMS}[g]_t$: a running estimate of inverse curvature, in metres per (1/metre)

So AdaDelta is "RMSProp whose learning rate is $\mathrm{RMS}[\Delta\theta]_{t-1}$". Where the updates have recently been large, it keeps taking large steps. Where the gradient RMS is large (a steep direction), it shrinks them. The argument is a heuristic. The ratio is a crude, lagged, diagonal estimate, not a real Hessian, and it is only unit-correct when $\epsilon$ is small next to both averages.

> Figure: AdaDelta (mint) on the valley ½(x² + 25y²) from (−4, 1.6), 300 full-batch steps. The dashed curve is PyTorch's defaults (ρ 0.9, ε 1e-6, lr 1). At those settings the first step is 0.0032 in each coordinate, and the loss only falls from 40 to 25.4 by step 100 and to 6.5 by step 300: the effective learning rate creeps from 8e-4 to 2e-3. Raise ε to 1e-4 and the loss reaches 0.0014 at step 187. At 1e-2 it reaches 3e-26 by step 75, then climbs back to 7.3: with the gradients gone, the ratio tends to √ε/√ε = 1, which is above the stability limit 2/25 for the steep y direction, and y oscillates. Switch the parameter units to c = 100 with ε 1e-6 and the run barely moves (loss 39.3 at step 300). Switch to c = 0.01 and it behaves almost exactly like c = 1 with ε = 1e-2.

## ε is the hidden learning rate

Look at the first step. $E[\Delta\theta^2]_0=0$, so the numerator is $\sqrt\epsilon$, and $E[g^2]_1=(1-\rho)g_1^2$. For any gradient with $(1-\rho)g_1^2\gg\epsilon$:

$$
|\Delta\theta_1|=\frac{\sqrt{\epsilon}\,|g_1|}{\sqrt{(1-\rho)g_1^2+\epsilon}}\approx\sqrt{\frac{\epsilon}{1-\rho}}
$$

- $\sqrt{\epsilon/(1-\rho)}$: $0.0032$ for $\rho=0.9,\ \epsilon=10^{-6}$; $0.0045$ for Zeiler's $\rho=0.95$

The first step does not depend on the gradient's size at all, only on $\epsilon$ and $\rho$. It is measured in the parameter's own units, so the same $\epsilon$ is a tiny step for a weight of order 100 and a huge one for a weight of order 0.01. After that, $E[\Delta\theta^2]$ can only grow as fast as the steps feed it. Under a constant gradient the step grows roughly like $\sqrt{\epsilon\,(1+(1-\rho)t)}$, that is, like the square root of the step count.

Hence the figure's units buttons. Rescale the parameters by $c$ and the gradients scale by $1/c$. As long as the denominator's $\epsilon$ stays small next to $E[g^2]$, nothing changes there, but the numerator's $\epsilon$ now behaves like $\epsilon/c^2$ in the original units. A model whose weights are of order 0.01 sees PyTorch's default as if it were $10^{-2}$. **"No learning rate" really means "a learning rate written as $\epsilon$, in units of $\theta^2$".**

## Worked example: a constant gradient

One parameter, gradient $g=1$ at every step, $\rho=0.9$. Iterating the four lines of the update rule gives (checked in Node):

AdaDelta under a constant unit gradient: the step grows only like √t, and its scale is set by ε

| step t | $E[g^2]_t$ | $\mathrm{RMS}[\Delta\theta]_{t-1}$ (ε = 1e-6) | step $\|\Delta\theta_t\|$ (ε = 1e-6) | step $\|\Delta\theta_t\|$ (ε = 1e-2) |
| --- | --- | --- | --- | --- |
| 1 | 0.100 | 0.00100 | 0.00316 | 0.302 |
| 2 | 0.190 | 0.00141 | 0.00324 | 0.309 |
| 3 | 0.271 | 0.00172 | 0.00330 | 0.314 |
| 10 | 0.651 | 0.00283 | 0.00351 | 0.334 |
| 100 | 1.000 | 0.00466 | 0.00466 | 0.436 |
| 1000 | 1.000 | 0.0106 | 0.0106 | 0.817 |

**Step 1 at the PyTorch defaults**

1. $E[g^2]_1=0.9\cdot0+0.1\cdot1^2=0.1$, so $\mathrm{RMS}[g]_1=\sqrt{0.1+10^{-6}}=0.3162$.
2. $\mathrm{RMS}[\Delta\theta]_0=\sqrt{0+10^{-6}}=0.001$.
3. $\Delta\theta_1=-(0.001/0.3162)\cdot1=-0.00316$.
4. $E[\Delta\theta^2]_1=0.1\cdot0.00316^2=1.0\times10^{-6}$. The numerator for step 2 is $\sqrt{2\times10^{-6}}=0.00141$, so the step barely grows.
5. After 1000 steps the parameter has moved $7.6$ in total. With $\epsilon=10^{-2}$ it moves $644$. The gradient was identical in both runs; only $\epsilon$ differed.

## Near a minimum the ratio drifts to 1

As the gradients shrink, both averages decay toward 0 and both square roots approach $\sqrt\epsilon$, so the multiplier approaches $\sqrt\epsilon/\sqrt\epsilon=1$, times PyTorch's `lr`. AdaDelta then behaves like plain gradient descent with step 1. That step is unstable in any direction with curvature above 2. In the figure's steep direction (curvature 25) the iterate starts to oscillate until $E[g^2]$ grows enough to pull the ratio back to the edge of stability, $2/25=0.08$. With noisy minibatch gradients $E[g^2]$ never decays to 0, which hides the effect. When it does show up, decaying PyTorch's `lr` multiplier with a schedule removes it.

```python
import torch

# The paper's method is lr=1.0 (PyTorch's default). Keras defaults to learning_rate=0.001.
opt = torch.optim.Adadelta(model.parameters(), lr=1.0, rho=0.9, eps=1e-6)

# One parameter under a constant gradient of 1: the first step is sqrt(eps / (1 - rho)).
p = torch.zeros(1, requires_grad=True)
opt = torch.optim.Adadelta([p], lr=1.0, rho=0.9, eps=1e-6)
for t in range(1000):
    opt.zero_grad()
    p.sum().backward()          # dL/dp = 1
    opt.step()
    if t == 0:
        print(p.item())         # about -0.00316
print(p.item())                 # about -7.6 after 1000 steps

# The update, written out (all element-wise)
@torch.no_grad()
def adadelta_step(p, g, sq, acc, rho=0.9, eps=1e-6, lr=1.0):
    sq.mul_(rho).addcmul_(g, g, value=1 - rho)               # E[g^2]
    delta = (acc + eps).sqrt() / (sq + eps).sqrt() * g       # RMS[dθ]_{t-1} / RMS[g]_t * g
    acc.mul_(rho).addcmul_(delta, delta, value=1 - rho)      # E[dθ^2], unscaled by lr
    p.sub_(lr * delta)
```

AdaDelta in PyTorch, and the same rule written out

> **Default ε and badly scaled parameters:** AdaDelta's first steps are about $\sqrt{\epsilon/(1-\rho)}$ in absolute parameter units, $0.003$ at PyTorch's defaults. That is fine for weights of order 0.1 to 1. It is far too small for parameters of order 100 (an unnormalized bias, a physical constant, a large embedding scale), and it can be too large for tiny ones. If a run "does nothing" for hundreds of steps, raise $\epsilon$ or the `lr` multiplier before blaming the model.

> **Built-in warm-up:** Because $E[\Delta\theta^2]$ starts at 0 and fills slowly, AdaDelta's effective learning rate starts tiny and ramps up by itself. That self-warm-up is gentle on fragile early training. It is also why AdaDelta is often slow, and why modern recipes prefer Adam(W) with an explicit, controllable warm-up schedule.

> **Where it stands today:** AdaDelta was popular around 2013 to 2015, for example in early sequence-to-sequence and convolutional text classifiers. Adam (2014) offered similar per-parameter scaling with momentum and an interpretable step size of about $\eta$, and it largely replaced AdaDelta. You will still see it in older code and in tutorials. It is also a useful way to see that the units of an update are a real design question.

**Strengths**

- No global learning rate to tune in the paper's form; reasonable defaults often train
- Decaying averages: unlike AdaGrad, the effective step does not shrink to zero
- Unit-consistent step (when ε is small), loosely mimicking a diagonal Newton step
- Starts with small steps by construction, a built-in warm-up

**Limitations**

- $\epsilon$ secretly sets the step scale in parameter units, so it must be tuned to the parameters' magnitude
- Slow start: the step grows roughly like $\sqrt t$ from $\sqrt{\epsilon/(1-\rho)}$
- No momentum, and with full-batch gradients the ratio drifts to 1 near a minimum and can oscillate
- Two state tensors per parameter (as much as Adam) with no momentum term to show for it

**Question:** AdaDelta (PyTorch defaults ρ = 0.9, ε = 1e-6, lr = 1) takes its first step on a parameter whose gradient is 50. About how big is the step?

- A. 50
- B. 0.0032
- C. 1
- D. 0.05

**Answer:** B. 0.0032 — $|\Delta\theta_1|=\sqrt\epsilon\,|g|/\sqrt{(1-\rho)g^2+\epsilon}\approx\sqrt{\epsilon/(1-\rho)}=\sqrt{10^{-5}}=0.0032$. The gradient's magnitude cancels. Only $\epsilon$ and $\rho$ set the first step, which is why $\epsilon$ acts as a learning rate.

**Question:** Why does AdaDelta put RMS[Δθ] in the numerator of its per-parameter rate?

- A. To add momentum
- B. So the update has the same units as the parameter, mimicking the inverse curvature in Newton's step
- C. To make the effective learning rate decay like AdaGrad's
- D. To correct the bias of moving averages that start at zero

**Answer:** B. So the update has the same units as the parameter, mimicking the inverse curvature in Newton's step — Newton's step $-g/H$ has the parameter's units, and $1/H\approx\Delta\theta/g$. AdaDelta approximates that ratio with running RMS values of past updates and gradients, which removes the need for a global $\eta$. There is no bias correction: the zero start is exactly why the first steps are tiny.

**Key points**

- Update: $\Delta\theta_t=-\big(\mathrm{RMS}[\Delta\theta]_{t-1}/\mathrm{RMS}[g]_t\big)\,g_t$, with decaying averages of squared gradients and squared updates.
- It fixes AdaGrad's ever-shrinking step, like RMSProp, and drops the global learning rate with a units argument from Newton's method.
- $\epsilon$ sits inside both square roots and sets the first step, $\approx\sqrt{\epsilon/(1-\rho)}$ (0.0032 at the defaults), in the parameter's own units.
- Rescaling the parameters by $c$ acts like changing the numerator's $\epsilon$ to $\epsilon/c^2$: tune $\epsilon$ to the weights' magnitude.
- PyTorch: Adadelta(lr=1.0, rho=0.9, eps=1e-6); lr is an extra multiplier. Keras defaults to 0.001, so pass 1.0 for the paper's method.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
