Optimization algorithms

Every optimizer steps against the gradient; momentum smooths the direction and Adam also rescales each parameter's step by its gradient history.

9 min read807 wordscore0 viewsupdated 2026-10-09

Definition

An optimizer is the rule that turns the gradient of the loss into a parameter update. Backpropagation gives you g=∇θLg=\nabla_\theta L; the optimizer decides how far, and in exactly which direction, to move θ\theta.

Every optimizer in common use is a variation on one move: step against the gradient. They differ in two ways. Some keep a running average of past gradients to smooth the direction (momentum). Some keep a running average of past squared gradients and divide by it, giving each parameter its own step size (adaptive methods). Adam does both.

Why bother? Deep-learning loss surfaces are ill-conditioned: steep in some directions, nearly flat in others. A single global step size that is safe for the steep directions crawls along the flat ones. The figure below shows exactly this on a 2-D valley.

Gradient descent and SGD

θt+1=θt−η gt,gt=∇θ1∣B∣∑i∈Bℓ(xi,yi;θt)\theta_{t+1}=\theta_t-\eta\,g_t,\qquad g_t=\nabla_\theta \frac{1}{|B|}\sum_{i\in B}\ell(x_i,y_i;\theta_t)
θ\theta
all trainable parameters
η\eta
learning rate (step size)
BB
the mini-batch; the full dataset gives plain gradient descent, a random batch gives SGD

The batch gradient is a noisy but unbiased estimate of the full gradient, and it costs a tiny fraction as much. Batches and epochs get their own article; here the question is what to do with gtg_t once you have it.

The learning rate is the knob that matters most. On a quadratic with largest curvature λmax⁡\lambda_{\max}, gradient descent converges only if η<2/λmax⁡\eta<2/\lambda_{\max}. Progress along the flattest direction then shrinks by a factor of at best about 1−2/κ1-2/\kappa per step, where κ=λmax⁡/λmin⁡\kappa=\lambda_{\max}/\lambda_{\min} is the condition number. The steep direction caps η\eta; the flat direction pays for it.

Momentum and Nesterov

vt+1=β vt+gt,θt+1=θt−η vt+1v_{t+1}=\beta\,v_t+g_t,\qquad \theta_{t+1}=\theta_t-\eta\,v_{t+1}
vv
velocity: an exponentially weighted sum of past gradients
β\beta
momentum coefficient, typically 0.90.9

Gradient components that flip sign every step (across the valley) cancel in vv. Components that keep pointing the same way (along the valley) add up, to about 1/(1−β)=10×1/(1-\beta)=10\times the single-step size. Momentum also raises the stability limit on a quadratic to η<2(1+β)/λmax⁡\eta<2(1+\beta)/\lambda_{\max}.

Nesterov momentum evaluates the gradient at the look-ahead point θt−ηβvt\theta_t-\eta\beta v_t instead of at θt\theta_t, so it corrects an overshoot one step earlier. In PyTorch it is SGD(..., momentum=0.9, nesterov=True).

Adaptive step sizes: AdaGrad and RMSProp

st+1=ρ st+(1−ρ) gt2,θt+1=θt−ηst+1+ϵ gts_{t+1}=\rho\,s_t+(1-\rho)\,g_t^2,\qquad \theta_{t+1}=\theta_t-\frac{\eta}{\sqrt{s_{t+1}}+\epsilon}\,g_t
ss
running average of the squared gradient, per parameter (all operations are element-wise)
ρ\rho
decay rate, typically 0.990.99 (PyTorch default) or 0.90.9
ϵ\epsilon
tiny constant (e.g. 10−810^{-8}) to avoid division by zero

Dividing by s\sqrt{s} (roughly the gradient's RMS) normalizes the step: a parameter with large gradients takes smaller steps, a parameter with tiny gradients takes larger ones. Each coordinate moves at roughly η\eta per step regardless of its gradient scale.

AdaGrad, the predecessor, sums instead of averaging: st+1=st+gt2s_{t+1}=s_t+g_t^2. That suits sparse features, which see few gradients, but ss only grows, so the effective learning rate decays to zero. RMSProp fixes this by forgetting old gradients.

Adam and AdamW

mt=β1mt−1+(1−β1) gt,m^t=mt/(1−β1 t)vt=β2vt−1+(1−β2) gt2,v^t=vt/(1−β2 t)θt=θt−1−η m^tv^t+ϵ\begin{aligned}m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, & \hat m_t&=m_t/(1-\beta_1^{\,t})\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2, & \hat v_t&=v_t/(1-\beta_2^{\,t})\\ \theta_t&=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\end{aligned}
mtm_t
first moment: momentum, as an average rather than a sum
vtv_t
second moment: RMSProp's squared-gradient average
m^t,v^t\hat m_t,\hat v_t
bias-corrected moments
β1,β2\beta_1,\beta_2
decay rates, defaults 0.90.9 and 0.9990.999
tt
step counter, starting at 1

Bias correction. Both moments start at 0, so early on they are biased toward 0. At t=1t=1, m1=0.1 gm_1=0.1\,g and v1=0.001 g2v_1=0.001\,g^2. Dividing by 1−βt1-\beta^t restores m^1=g\hat m_1=g and v^1=g2\hat v_1=g^2, so the first step is exactly η⋅sign⁡(g)\eta\cdot\operatorname{sign}(g). Without the correction the ratio would be 0.1/0.001≈3.160.1/\sqrt{0.001}\approx3.16 times too large. The correction fades as βt→0\beta^t\to0.

AdamW fixes how Adam handles weight decay. Adding an L2 term λθ\lambda\theta to the gradient sends it through the 1/v^1/\sqrt{\hat v} rescaling, so parameters with large gradient history are barely regularized. AdamW applies the decay directly to the weights instead:

θt=θt−1−η(m^tv^t+ϵ+λ θt−1)\theta_t=\theta_{t-1}-\eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}+\lambda\,\theta_{t-1}\right)
λ\lambda
decoupled weight decay, typically 0.010.01 to 0.10.1
f = ½(u² + 25v²), valley along the x-axisstartminloss (log scale) vs step1e-61e-311e3060120step
t = 120|
SGD is stable only for lr < 2/25 = 0.08 (the steep direction's curvature is 25). Click an optimizer to hide it.
fig.The loss is a valley 25 times steeper across than along. Press play. At lr 0.06, SGD zigzags across the valley and momentum overshoots and swings. RMSProp and Adam move diagonally because each coordinate gets a step of about lr. Raise lr past 0.08 and SGD diverges; past about 0.15 momentum diverges too, while RMSProp and Adam survive 0.3. Then switch to the rotated valley: SGD and momentum behave identically, but the per-coordinate methods lose their edge because the steep direction no longer lines up with an axis.

Two lessons from the figure. Adam is not automatically fastest: its step is about η\eta in parameter units, so at η=0.06\eta=0.06 it needs roughly 4/0.06≈674/0.06\approx67 steps just to travel the 4 units from start to minimum. Its strength is robustness: the step size no longer depends on the scale of the gradient, so one learning rate works across layers whose gradients differ by orders of magnitude.

Worked example: the first two steps

optimizerfirst update Δθθ after 1 steploss after 1loss after 2
SGD(0.20, −2.00)(−3.80, −0.40)9.226.64
Momentum(0.20, −2.00)(−3.80, −0.40)9.2242.01
RMSProp(0.50, −0.50)(−3.50, 1.10)21.2513.34
Adam(0.05, −0.05)(−3.95, 1.55)37.8335.73
The figure's valley f=12(x2+25y2)f=\tfrac12(x^2+25y^2), start (−4, 1.6)(-4,\,1.6), f=40f=40, gradient (−4, 40)(-4,\,40), η=0.05\eta=0.05, default betas.
  1. 1SGD moves 10 times further in yy than in xx because the gradient is 10 times larger there. It overshoots the valley floor from y=1.6y=1.6 to y=−0.4y=-0.4.
  2. 2Momentum's first step equals SGD's (v1=g1v_1=g_1). On step 2 the stored velocity carries it further across the valley, to y=−1.7y=-1.7, and the loss jumps to 42. It needs many more steps to damp that swing.
  3. 3RMSProp's first step is η g/0.01 g2=10η\eta\,g/\sqrt{0.01\,g^2}=10\eta per coordinate: it has no bias correction, so its first steps are large.
  4. 4Adam's first step is exactly η=0.05\eta=0.05 in both coordinates, despite the 10× gradient ratio. That is the rescaling in action, and the reason it is so small is the bias correction.

Which one to use

optimizertypical settingsuse it for
SGD + momentum (Nesterov)η≈0.1\eta\approx0.1 (batch 256), β=0.9\beta=0.9, weight decay 10−410^{-4}, plus a scheduleCNNs for vision (ResNets); often generalizes slightly better but needs tuning
Adamη=10−3\eta=10^{-3} or 3×10−43\times10^{-4}, β=(0.9,0.999)\beta=(0.9,0.999)a strong default for prototyping, small models and RL
AdamWη≈10−4\eta\approx10^{-4} to 10−310^{-3} with warmup, λ=0.01\lambda=0.01 to 0.10.1; LLMs often use β2=0.95\beta_2=0.95transformers and almost all large-scale training
RMSPropη≈10−3\eta\approx10^{-3}, ρ=0.9\rho=0.9 to 0.990.99older RNN and reinforcement-learning setups
AdaGradη≈10−2\eta\approx10^{-2}sparse features (linear models, embeddings); the step size decays permanently

Adam-family optimizers store two extra tensors (mm and vv) per parameter, so optimizer state takes twice the memory of the weights themselves. SGD with momentum stores one. At billions of parameters that difference matters.

python
1import torch
2
3opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.999), weight_decay=0.01)
4# opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, nesterov=True, weight_decay=1e-4)
5
6for x, y in loader:
7 opt.zero_grad() # gradients accumulate otherwise
8 loss = loss_fn(model(x), y)
9 loss.backward() # fills p.grad for every parameter
10 opt.step() # applies the update rule

Check yourself

On its very first step, with ε negligible, how large is Adam's update to a parameter whose gradient is g?

Check yourself

Why is AdamW preferred over Adam with an L2 penalty?

Remember

  • ✓All optimizers step against the gradient; they differ in how they smooth and scale it.
  • ✓Plain gradient descent is capped at η < 2/λ_max, so ill-conditioned valleys are slow.
  • ✓Momentum averages directions: oscillations cancel, consistent components accumulate.
  • ✓RMSProp and Adam divide by the RMS gradient, giving every parameter a step of about η.
  • ✓Adam = momentum + RMSProp + bias correction; AdamW decouples weight decay and is the transformer default.
  • ✓The learning rate matters more than the choice of optimizer.
← back to Deep Learning roadmap