AdaDelta

AdaGrad without a learning rate: each step is an RMS of recent updates divided by an RMS of recent gradients, so it has the parameter's units. The catch is that ε then decides how big the first steps are, so it acts as the learning rate.

10 min read1022 wordsadvanced0 viewsupdated 2026-10-11

Definition

AdaDelta is an adaptive optimizer that scales each parameter's gradient by RMS[Δθ]t−1/RMS[g]t\mathrm{RMS}[\Delta\theta]_{t-1}/\mathrm{RMS}[g]_t. Both are exponential moving averages: one of past updates, one of past gradients. The ratio stands in for a per-parameter learning rate, so the method has no global step size η\eta.

Matthew Zeiler introduced AdaDelta in a 2012 arXiv note, ADADELTA: An Adaptive Learning Rate Method. He wanted to fix two things about AdaGrad (Duchi, Hazan and Singer, 2011). First, AdaGrad divides by the square root of a sum of all past squared gradients, so its effective learning rate can only fall and training eventually stalls. Second, AdaGrad still needs a hand-tuned global learning rate.

AdaDelta fixes the first problem the same way RMSProp does: it replaces the growing sum with a decaying average. For the second, it uses a units argument. The update −ηg-\eta g has the units of a gradient, not of the parameter. Newton's step −H−1g-H^{-1}g does have the parameter's units, and AdaDelta builds a cheap diagonal stand-in for H−1H^{-1} out of quantities it already tracks.

The update rule

E[g2]t=ρ E[g2]t−1+(1−ρ) gt2Δθt=−E[Δθ2]t−1+ϵE[g2]t+ϵ  gt=−RMS[Δθ]t−1RMS[g]t gtE[Δθ2]t=ρ E[Δθ2]t−1+(1−ρ) Δθt2θt+1=θt+Δθt\begin{aligned}E[g^2]_t&=\rho\,E[g^2]_{t-1}+(1-\rho)\,g_t^2\\ \Delta\theta_t&=-\frac{\sqrt{E[\Delta\theta^2]_{t-1}+\epsilon}}{\sqrt{E[g^2]_t+\epsilon}}\;g_t=-\frac{\mathrm{RMS}[\Delta\theta]_{t-1}}{\mathrm{RMS}[g]_t}\,g_t\\ E[\Delta\theta^2]_t&=\rho\,E[\Delta\theta^2]_{t-1}+(1-\rho)\,\Delta\theta_t^2\\ \theta_{t+1}&=\theta_t+\Delta\theta_t\end{aligned}
gtg_t
the gradient at step tt; every operation is element-wise, one copy per parameter
E[g2]E[g^2]
decaying average of squared gradients (RMSProp's accumulator); starts at 0
E[Δθ2]E[\Delta\theta^2]
decaying average of squared updates; starts at 0
ρ\rho
decay rate. Zeiler used 0.95 on MNIST; PyTorch's default is 0.9
ϵ\epsilon
added inside both square roots. Zeiler used 10−610^{-6} on MNIST, which is also PyTorch's default
RMS[x]\mathrm{RMS}[x]
E[x2]+ϵ\sqrt{E[x^2]+\epsilon}

The numerator uses t−1t-1 because Δθt\Delta\theta_t is not known until the step has been computed. That lag is also what makes AdaDelta start slowly: at t=1t=1 the numerator is 0+ϵ=ϵ\sqrt{0+\epsilon}=\sqrt\epsilon.

PyTorch's torch.optim.Adadelta adds one knob the paper does not have: lr (default 1.0) multiplies Δθt\Delta\theta_t just before it is applied. The accumulator E[Δθ2]E[\Delta\theta^2] still stores the unscaled Δθt\Delta\theta_t. With lr=1.0 you get the paper's method. Keras's Adadelta defaults to learning_rate=0.001, so you have to pass 1.0 explicitly to get the original algorithm.

At a glance

Paper
Zeiler, arXiv:1212.5701 (Dec 2012)
State per parameter
two tensors, E[g2]E[g^2] and E[Δθ2]E[\Delta\theta^2]: the same memory as Adam
Hyperparameters
ρ\rho and ϵ\epsilon; PyTorch adds an lr multiplier (default 1.0)
Units of the step
those of θ\theta (if ϵ\epsilon is negligible), unlike SGD, AdaGrad and RMSProp
First step
∣Δθ1∣≈ϵ/(1−ρ)|\Delta\theta_1|\approx\sqrt{\epsilon/(1-\rho)} for any gradient with (1−ρ)g2≫ϵ(1-\rho)g^2\gg\epsilon: 0.00320.0032 at the defaults
PyTorch
torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-6, weight_decay=0)

Where the ratio comes from: a units argument

Give θ\theta the unit "metres" and treat the loss as unitless. Then g=∂L/∂θg=\partial L/\partial\theta is in 1/metres, and an SGD step ηg\eta g is in 1/metres too unless η\eta is in metres². The learning rate quietly carries the conversion. AdaGrad and RMSProp divide gg by an RMS of gg, so their step is unitless, and again η\eta has to supply the metres.

Newton's method gets the units right on its own. Its step Δθ=−g/H\Delta\theta=-g/H divides 1/metres by 1/metres², which gives metres. Rearranging Newton's step gives an expression for the inverse curvature that uses only first-order quantities:

Δθ=−∂L/∂θ∂2L/∂θ2⟹1∂2L/∂θ2=−Δθ∂L/∂θ  ≈  RMS[Δθ]t−1RMS[g]t\Delta\theta=-\frac{\partial L/\partial\theta}{\partial^2L/\partial\theta^2}\quad\Longrightarrow\quad\frac{1}{\partial^2L/\partial\theta^2}=-\frac{\Delta\theta}{\partial L/\partial\theta}\;\approx\;\frac{\mathrm{RMS}[\Delta\theta]_{t-1}}{\mathrm{RMS}[g]_t}
∂2L/∂θ2\partial^2L/\partial\theta^2
the diagonal curvature, which AdaDelta never computes
RMS[Δθ]t−1/RMS[g]t\mathrm{RMS}[\Delta\theta]_{t-1}/\mathrm{RMS}[g]_t
a running estimate of inverse curvature, in metres per (1/metre)

So AdaDelta is "RMSProp whose learning rate is RMS[Δθ]t−1\mathrm{RMS}[\Delta\theta]_{t-1}". Where the updates have recently been large, it keeps taking large steps. Where the gradient RMS is large (a steep direction), it shrinks them. The argument is a heuristic. The ratio is a crude, lagged, diagonal estimate, not a real Hessian, and it is only unit-correct when ϵ\epsilon is small next to both averages.

f = ½(x² + 25y²) in units θ/c, c = 1startminloss (log), ┄ defaults1e-81e-41eff. lr: x solid, y ┄1e-61e-41e-212/25: y unstable above0150300
|parameter units ct = 300
● AdaDelta loss 6.509 · best 6.509 at t = 300
effective lr = (0.002, 2.7e-4)
first step |Δθ|/c = 0.003 ≈ lr·√(ε/(1−ρ))/c = 0.003
┄ defaults (ρ 0.9, ε 1e-6, lr 1): loss 6.509
No learning rate to tune, but ε sets the size of the first steps in parameter units, so it acts as one.
fig.AdaDelta (mint) on the valley ½(x² + 25y²) from (−4, 1.6), 300 full-batch steps. The dashed curve is PyTorch's defaults (ρ 0.9, ε 1e-6, lr 1). At those settings the first step is 0.0032 in each coordinate, and the loss only falls from 40 to 25.4 by step 100 and to 6.5 by step 300: the effective learning rate creeps from 8e-4 to 2e-3. Raise ε to 1e-4 and the loss reaches 0.0014 at step 187. At 1e-2 it reaches 3e-26 by step 75, then climbs back to 7.3: with the gradients gone, the ratio tends to √ε/√ε = 1, which is above the stability limit 2/25 for the steep y direction, and y oscillates. Switch the parameter units to c = 100 with ε 1e-6 and the run barely moves (loss 39.3 at step 300). Switch to c = 0.01 and it behaves almost exactly like c = 1 with ε = 1e-2.

ε is the hidden learning rate

Look at the first step. E[Δθ2]0=0E[\Delta\theta^2]_0=0, so the numerator is ϵ\sqrt\epsilon, and E[g2]1=(1−ρ)g12E[g^2]_1=(1-\rho)g_1^2. For any gradient with (1−ρ)g12≫ϵ(1-\rho)g_1^2\gg\epsilon:

∣Δθ1∣=ϵ ∣g1∣(1−ρ)g12+ϵ≈ϵ1−ρ|\Delta\theta_1|=\frac{\sqrt{\epsilon}\,|g_1|}{\sqrt{(1-\rho)g_1^2+\epsilon}}\approx\sqrt{\frac{\epsilon}{1-\rho}}
ϵ/(1−ρ)\sqrt{\epsilon/(1-\rho)}
0.00320.0032 for ρ=0.9, ϵ=10−6\rho=0.9,\ \epsilon=10^{-6}; 0.00450.0045 for Zeiler's ρ=0.95\rho=0.95

The first step does not depend on the gradient's size at all, only on ϵ\epsilon and ρ\rho. It is measured in the parameter's own units, so the same ϵ\epsilon is a tiny step for a weight of order 100 and a huge one for a weight of order 0.01. After that, E[Δθ2]E[\Delta\theta^2] can only grow as fast as the steps feed it. Under a constant gradient the step grows roughly like ϵ (1+(1−ρ)t)\sqrt{\epsilon\,(1+(1-\rho)t)}, that is, like the square root of the step count.

Hence the figure's units buttons. Rescale the parameters by cc and the gradients scale by 1/c1/c. As long as the denominator's ϵ\epsilon stays small next to E[g2]E[g^2], nothing changes there, but the numerator's ϵ\epsilon now behaves like ϵ/c2\epsilon/c^2 in the original units. A model whose weights are of order 0.01 sees PyTorch's default as if it were 10−210^{-2}. "No learning rate" really means "a learning rate written as ϵ\epsilon, in units of θ2\theta^2".

Worked example: a constant gradient

One parameter, gradient g=1g=1 at every step, ρ=0.9\rho=0.9. Iterating the four lines of the update rule gives (checked in Node):

step tE[g2]tE[g^2]_tRMS[Δθ]t−1\mathrm{RMS}[\Delta\theta]_{t-1} (ε = 1e-6)step ∣Δθt∣|\Delta\theta_t| (ε = 1e-6)step ∣Δθt∣|\Delta\theta_t| (ε = 1e-2)
10.1000.001000.003160.302
20.1900.001410.003240.309
30.2710.001720.003300.314
100.6510.002830.003510.334
1001.0000.004660.004660.436
10001.0000.01060.01060.817
AdaDelta under a constant unit gradient: the step grows only like √t, and its scale is set by ε

Step 1 at the PyTorch defaults

  1. 1E[g2]1=0.9⋅0+0.1⋅12=0.1E[g^2]_1=0.9\cdot0+0.1\cdot1^2=0.1, so RMS[g]1=0.1+10−6=0.3162\mathrm{RMS}[g]_1=\sqrt{0.1+10^{-6}}=0.3162.
  2. 2RMS[Δθ]0=0+10−6=0.001\mathrm{RMS}[\Delta\theta]_0=\sqrt{0+10^{-6}}=0.001.
  3. 3Δθ1=−(0.001/0.3162)⋅1=−0.00316\Delta\theta_1=-(0.001/0.3162)\cdot1=-0.00316.
  4. 4E[Δθ2]1=0.1⋅0.003162=1.0×10−6E[\Delta\theta^2]_1=0.1\cdot0.00316^2=1.0\times10^{-6}. The numerator for step 2 is 2×10−6=0.00141\sqrt{2\times10^{-6}}=0.00141, so the step barely grows.
  5. 5After 1000 steps the parameter has moved 7.67.6 in total. With ϵ=10−2\epsilon=10^{-2} it moves 644644. The gradient was identical in both runs; only ϵ\epsilon differed.

Near a minimum the ratio drifts to 1

As the gradients shrink, both averages decay toward 0 and both square roots approach ϵ\sqrt\epsilon, so the multiplier approaches ϵ/ϵ=1\sqrt\epsilon/\sqrt\epsilon=1, times PyTorch's lr. AdaDelta then behaves like plain gradient descent with step 1. That step is unstable in any direction with curvature above 2. In the figure's steep direction (curvature 25) the iterate starts to oscillate until E[g2]E[g^2] grows enough to pull the ratio back to the edge of stability, 2/25=0.082/25=0.08. With noisy minibatch gradients E[g2]E[g^2] never decays to 0, which hides the effect. When it does show up, decaying PyTorch's lr multiplier with a schedule removes it.

python
1import torch
2
3# The paper's method is lr=1.0 (PyTorch's default). Keras defaults to learning_rate=0.001.
4opt = torch.optim.Adadelta(model.parameters(), lr=1.0, rho=0.9, eps=1e-6)
5
6# One parameter under a constant gradient of 1: the first step is sqrt(eps / (1 - rho)).
7p = torch.zeros(1, requires_grad=True)
8opt = torch.optim.Adadelta([p], lr=1.0, rho=0.9, eps=1e-6)
9for t in range(1000):
10 opt.zero_grad()
11 p.sum().backward() # dL/dp = 1
12 opt.step()
13 if t == 0:
14 print(p.item()) # about -0.00316
15print(p.item()) # about -7.6 after 1000 steps
16
17# The update, written out (all element-wise)
18@torch.no_grad()
19def adadelta_step(p, g, sq, acc, rho=0.9, eps=1e-6, lr=1.0):
20 sq.mul_(rho).addcmul_(g, g, value=1 - rho) # E[g^2]
21 delta = (acc + eps).sqrt() / (sq + eps).sqrt() * g # RMS[dθ]_{t-1} / RMS[g]_t * g
22 acc.mul_(rho).addcmul_(delta, delta, value=1 - rho) # E[dθ^2], unscaled by lr
23 p.sub_(lr * delta)
AdaDelta in PyTorch, and the same rule written out

Use when

  • +No global learning rate to tune in the paper's form; reasonable defaults often train
  • +Decaying averages: unlike AdaGrad, the effective step does not shrink to zero
  • +Unit-consistent step (when ε is small), loosely mimicking a diagonal Newton step
  • +Starts with small steps by construction, a built-in warm-up

Avoid when

  • −ϵ\epsilon secretly sets the step scale in parameter units, so it must be tuned to the parameters' magnitude
  • −Slow start: the step grows roughly like t\sqrt t from ϵ/(1−ρ)\sqrt{\epsilon/(1-\rho)}
  • −No momentum, and with full-batch gradients the ratio drifts to 1 near a minimum and can oscillate
  • −Two state tensors per parameter (as much as Adam) with no momentum term to show for it

Check yourself

AdaDelta (PyTorch defaults ρ = 0.9, ε = 1e-6, lr = 1) takes its first step on a parameter whose gradient is 50. About how big is the step?

Check yourself

Why does AdaDelta put RMS[Δθ] in the numerator of its per-parameter rate?

Remember

  • ✓Update: Δθt=−(RMS[Δθ]t−1/RMS[g]t) gt\Delta\theta_t=-\big(\mathrm{RMS}[\Delta\theta]_{t-1}/\mathrm{RMS}[g]_t\big)\,g_t, with decaying averages of squared gradients and squared updates.
  • ✓It fixes AdaGrad's ever-shrinking step, like RMSProp, and drops the global learning rate with a units argument from Newton's method.
  • ✓ϵ\epsilon sits inside both square roots and sets the first step, ≈ϵ/(1−ρ)\approx\sqrt{\epsilon/(1-\rho)} (0.0032 at the defaults), in the parameter's own units.
  • ✓Rescaling the parameters by cc acts like changing the numerator's ϵ\epsilon to ϵ/c2\epsilon/c^2: tune ϵ\epsilon to the weights' magnitude.
  • ✓PyTorch: Adadelta(lr=1.0, rho=0.9, eps=1e-6); lr is an extra multiplier. Keras defaults to 0.001, so pass 1.0 for the paper's method.
← back to Deep Learning roadmap