# Optimization algorithms

> Every optimizer steps against the gradient; momentum smooths the direction and Adam also rescales each parameter's step by its gradient history.

Source: https://www.mlfoundry.in/learn/optimization-algorithms  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 9 min read · updated 2026-10-09

**An optimizer:** is the rule that turns the gradient of the loss into a parameter update. Backpropagation gives you $g=\nabla_\theta L$; the optimizer decides how far, and in exactly which direction, to move $\theta$.

Every optimizer in common use is a variation on one move: step **against** the gradient. They differ in two ways. Some keep a running average of past gradients to smooth the direction (**momentum**). Some keep a running average of past *squared* gradients and divide by it, giving each parameter its own step size (**adaptive** methods). Adam does both.

Why bother? Deep-learning loss surfaces are **ill-conditioned**: steep in some directions, nearly flat in others. A single global step size that is safe for the steep directions crawls along the flat ones. The figure below shows exactly this on a 2-D valley.

## Gradient descent and SGD

$$
\theta_{t+1}=\theta_t-\eta\,g_t,\qquad g_t=\nabla_\theta \frac{1}{|B|}\sum_{i\in B}\ell(x_i,y_i;\theta_t)
$$

- $\theta$: all trainable parameters
- $\eta$: learning rate (step size)
- $B$: the mini-batch; the full dataset gives plain gradient descent, a random batch gives SGD

The batch gradient is a noisy but unbiased estimate of the full gradient, and it costs a tiny fraction as much. Batches and epochs get their own article; here the question is what to do with $g_t$ once you have it.

**The learning rate is the knob that matters most.** On a quadratic with largest curvature $\lambda_{\max}$, gradient descent converges only if $\eta<2/\lambda_{\max}$. Progress along the flattest direction then shrinks by a factor of at best about $1-2/\kappa$ per step, where $\kappa=\lambda_{\max}/\lambda_{\min}$ is the condition number. The steep direction caps $\eta$; the flat direction pays for it.

## Momentum and Nesterov

$$
v_{t+1}=\beta\,v_t+g_t,\qquad \theta_{t+1}=\theta_t-\eta\,v_{t+1}
$$

- $v$: velocity: an exponentially weighted sum of past gradients
- $\beta$: momentum coefficient, typically $0.9$

Gradient components that flip sign every step (across the valley) cancel in $v$. Components that keep pointing the same way (along the valley) add up, to about $1/(1-\beta)=10\times$ the single-step size. Momentum also raises the stability limit on a quadratic to $\eta<2(1+\beta)/\lambda_{\max}$.

**Nesterov momentum** evaluates the gradient at the look-ahead point $\theta_t-\eta\beta v_t$ instead of at $\theta_t$, so it corrects an overshoot one step earlier. In PyTorch it is `SGD(..., momentum=0.9, nesterov=True)`.

## Adaptive step sizes: AdaGrad and RMSProp

$$
s_{t+1}=\rho\,s_t+(1-\rho)\,g_t^2,\qquad \theta_{t+1}=\theta_t-\frac{\eta}{\sqrt{s_{t+1}}+\epsilon}\,g_t
$$

- $s$: running average of the squared gradient, per parameter (all operations are element-wise)
- $\rho$: decay rate, typically $0.99$ (PyTorch default) or $0.9$
- $\epsilon$: tiny constant (e.g. $10^{-8}$) to avoid division by zero

Dividing by $\sqrt{s}$ (roughly the gradient's RMS) normalizes the step: a parameter with large gradients takes smaller steps, a parameter with tiny gradients takes larger ones. Each coordinate moves at roughly $\eta$ per step regardless of its gradient scale.

**AdaGrad**, the predecessor, sums instead of averaging: $s_{t+1}=s_t+g_t^2$. That suits sparse features, which see few gradients, but $s$ only grows, so the effective learning rate decays to zero. **RMSProp** fixes this by forgetting old gradients.

## Adam and AdamW

$$
\begin{aligned}m_t&=\beta_1 m_{t-1}+(1-\beta_1)\,g_t, & \hat m_t&=m_t/(1-\beta_1^{\,t})\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^2, & \hat v_t&=v_t/(1-\beta_2^{\,t})\\ \theta_t&=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\end{aligned}
$$

- $m_t$: first moment: momentum, as an average rather than a sum
- $v_t$: second moment: RMSProp's squared-gradient average
- $\hat m_t,\hat v_t$: bias-corrected moments
- $\beta_1,\beta_2$: decay rates, defaults $0.9$ and $0.999$
- $t$: step counter, starting at 1

**Bias correction.** Both moments start at 0, so early on they are biased toward 0. At $t=1$, $m_1=0.1\,g$ and $v_1=0.001\,g^2$. Dividing by $1-\beta^t$ restores $\hat m_1=g$ and $\hat v_1=g^2$, so the first step is exactly $\eta\cdot\operatorname{sign}(g)$. Without the correction the ratio would be $0.1/\sqrt{0.001}\approx3.16$ times too large. The correction fades as $\beta^t\to0$.

**AdamW** fixes how Adam handles weight decay. Adding an L2 term $\lambda\theta$ to the gradient sends it through the $1/\sqrt{\hat v}$ rescaling, so parameters with large gradient history are barely regularized. AdamW applies the decay directly to the weights instead:

$$
\theta_t=\theta_{t-1}-\eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}+\lambda\,\theta_{t-1}\right)
$$

- $\lambda$: decoupled weight decay, typically $0.01$ to $0.1$

> **Adam's weight_decay is not AdamW's:** `torch.optim.Adam(..., weight_decay=0.01)` is coupled L2 regularization; `torch.optim.AdamW(..., weight_decay=0.01)` is decoupled decay. They give different models. Transformers are trained with AdamW.

> Figure: The loss is a valley 25 times steeper across than along. Press play. At lr 0.06, SGD zigzags across the valley and momentum overshoots and swings. RMSProp and Adam move diagonally because each coordinate gets a step of about lr. Raise lr past 0.08 and SGD diverges; past about 0.15 momentum diverges too, while RMSProp and Adam survive 0.3. Then switch to the rotated valley: SGD and momentum behave identically, but the per-coordinate methods lose their edge because the steep direction no longer lines up with an axis.

Two lessons from the figure. Adam is not automatically fastest: its step is about $\eta$ in parameter units, so at $\eta=0.06$ it needs roughly $4/0.06\approx67$ steps just to travel the 4 units from start to minimum. Its strength is **robustness**: the step size no longer depends on the scale of the gradient, so one learning rate works across layers whose gradients differ by orders of magnitude.

## Worked example: the first two steps

The figure's valley $f=\tfrac12(x^2+25y^2)$, start $(-4,\,1.6)$, $f=40$, gradient $(-4,\,40)$, $\eta=0.05$, default betas.

| optimizer | first update Δθ | θ after 1 step | loss after 1 | loss after 2 |
| --- | --- | --- | --- | --- |
| SGD | (0.20, −2.00) | (−3.80, −0.40) | 9.22 | 6.64 |
| Momentum | (0.20, −2.00) | (−3.80, −0.40) | 9.22 | 42.01 |
| RMSProp | (0.50, −0.50) | (−3.50, 1.10) | 21.25 | 13.34 |
| Adam | (0.05, −0.05) | (−3.95, 1.55) | 37.83 | 35.73 |

1. SGD moves 10 times further in $y$ than in $x$ because the gradient is 10 times larger there. It overshoots the valley floor from $y=1.6$ to $y=-0.4$.
2. Momentum's first step equals SGD's ($v_1=g_1$). On step 2 the stored velocity carries it further across the valley, to $y=-1.7$, and the loss jumps to 42. It needs many more steps to damp that swing.
3. RMSProp's first step is $\eta\,g/\sqrt{0.01\,g^2}=10\eta$ per coordinate: it has no bias correction, so its first steps are large.
4. Adam's first step is exactly $\eta=0.05$ in both coordinates, despite the 10× gradient ratio. That is the rescaling in action, and the reason it is so small is the bias correction.

## Which one to use

| optimizer | typical settings | use it for |
| --- | --- | --- |
| SGD + momentum (Nesterov) | $\eta\approx0.1$ (batch 256), $\beta=0.9$, weight decay $10^{-4}$, plus a schedule | CNNs for vision (ResNets); often generalizes slightly better but needs tuning |
| Adam | $\eta=10^{-3}$ or $3\times10^{-4}$, $\beta=(0.9,0.999)$ | a strong default for prototyping, small models and RL |
| AdamW | $\eta\approx10^{-4}$ to $10^{-3}$ with warmup, $\lambda=0.01$ to $0.1$; LLMs often use $\beta_2=0.95$ | transformers and almost all large-scale training |
| RMSProp | $\eta\approx10^{-3}$, $\rho=0.9$ to $0.99$ | older RNN and reinforcement-learning setups |
| AdaGrad | $\eta\approx10^{-2}$ | sparse features (linear models, embeddings); the step size decays permanently |

Adam-family optimizers store two extra tensors ($m$ and $v$) per parameter, so optimizer state takes twice the memory of the weights themselves. SGD with momentum stores one. At billions of parameters that difference matters.

```python
import torch

opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.999), weight_decay=0.01)
# opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, nesterov=True, weight_decay=1e-4)

for x, y in loader:
    opt.zero_grad()            # gradients accumulate otherwise
    loss = loss_fn(model(x), y)
    loss.backward()            # fills p.grad for every parameter
    opt.step()                 # applies the update rule
```

> **Tune the learning rate first:** Switching optimizers rarely matters as much as getting $\eta$ right. Sweep it on a log scale ($10^{-4},3\times10^{-4},10^{-3},\dots$) before touching anything else. Schedules (warmup, cosine decay) get their own article.

**Question:** On its very first step, with ε negligible, how large is Adam's update to a parameter whose gradient is g?

- A. η · g
- B. η, in the direction of −g
- C. 3.16 · η
- D. zero, because m starts at 0

**Answer:** B. η, in the direction of −g — Bias correction gives $\hat m_1=g$ and $\hat v_1=g^2$, so the update is $-\eta\,g/|g|=-\eta\operatorname{sign}(g)$, whatever the size of $g$. Without bias correction it would be about $3.16\,\eta$.

**Question:** Why is AdamW preferred over Adam with an L2 penalty?

- A. AdamW uses less memory
- B. With L2 inside Adam, the decay term is divided by √v̂, so parameters with large gradients are barely regularized
- C. AdamW does not need a learning rate
- D. Adam cannot use weight decay at all

**Answer:** B. With L2 inside Adam, the decay term is divided by √v̂, so parameters with large gradients are barely regularized — L2 adds $\lambda\theta$ to the gradient, and Adam then rescales it per parameter. AdamW subtracts $\eta\lambda\theta$ directly, so every weight decays at the same relative rate.

**Key points**

- All optimizers step against the gradient; they differ in how they smooth and scale it.
- Plain gradient descent is capped at η < 2/λ_max, so ill-conditioned valleys are slow.
- Momentum averages directions: oscillations cancel, consistent components accumulate.
- RMSProp and Adam divide by the RMS gradient, giving every parameter a step of about η.
- Adam = momentum + RMSProp + bias correction; AdamW decouples weight decay and is the transformer default.
- The learning rate matters more than the choice of optimizer.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
