Definition
An optimizer is the rule that turns the gradient of the loss into a parameter update. Backpropagation gives you ; the optimizer decides how far, and in exactly which direction, to move .
Every optimizer in common use is a variation on one move: step against the gradient. They differ in two ways. Some keep a running average of past gradients to smooth the direction (momentum). Some keep a running average of past squared gradients and divide by it, giving each parameter its own step size (adaptive methods). Adam does both.
Why bother? Deep-learning loss surfaces are ill-conditioned: steep in some directions, nearly flat in others. A single global step size that is safe for the steep directions crawls along the flat ones. The figure below shows exactly this on a 2-D valley.
Gradient descent and SGD
- all trainable parameters
- learning rate (step size)
- the mini-batch; the full dataset gives plain gradient descent, a random batch gives SGD
The batch gradient is a noisy but unbiased estimate of the full gradient, and it costs a tiny fraction as much. Batches and epochs get their own article; here the question is what to do with once you have it.
The learning rate is the knob that matters most. On a quadratic with largest curvature , gradient descent converges only if . Progress along the flattest direction then shrinks by a factor of at best about per step, where is the condition number. The steep direction caps ; the flat direction pays for it.
Momentum and Nesterov
- velocity: an exponentially weighted sum of past gradients
- momentum coefficient, typically
Gradient components that flip sign every step (across the valley) cancel in . Components that keep pointing the same way (along the valley) add up, to about the single-step size. Momentum also raises the stability limit on a quadratic to .
Nesterov momentum evaluates the gradient at the look-ahead point instead of at , so it corrects an overshoot one step earlier. In PyTorch it is SGD(..., momentum=0.9, nesterov=True).
Adaptive step sizes: AdaGrad and RMSProp
- running average of the squared gradient, per parameter (all operations are element-wise)
- decay rate, typically (PyTorch default) or
- tiny constant (e.g. ) to avoid division by zero
Dividing by (roughly the gradient's RMS) normalizes the step: a parameter with large gradients takes smaller steps, a parameter with tiny gradients takes larger ones. Each coordinate moves at roughly per step regardless of its gradient scale.
AdaGrad, the predecessor, sums instead of averaging: . That suits sparse features, which see few gradients, but only grows, so the effective learning rate decays to zero. RMSProp fixes this by forgetting old gradients.
Adam and AdamW
- first moment: momentum, as an average rather than a sum
- second moment: RMSProp's squared-gradient average
- bias-corrected moments
- decay rates, defaults and
- step counter, starting at 1
Bias correction. Both moments start at 0, so early on they are biased toward 0. At , and . Dividing by restores and , so the first step is exactly . Without the correction the ratio would be times too large. The correction fades as .
AdamW fixes how Adam handles weight decay. Adding an L2 term to the gradient sends it through the rescaling, so parameters with large gradient history are barely regularized. AdamW applies the decay directly to the weights instead:
- decoupled weight decay, typically to
Two lessons from the figure. Adam is not automatically fastest: its step is about in parameter units, so at it needs roughly steps just to travel the 4 units from start to minimum. Its strength is robustness: the step size no longer depends on the scale of the gradient, so one learning rate works across layers whose gradients differ by orders of magnitude.
Worked example: the first two steps
| optimizer | first update Δθ | θ after 1 step | loss after 1 | loss after 2 |
|---|---|---|---|---|
| SGD | (0.20, −2.00) | (−3.80, −0.40) | 9.22 | 6.64 |
| Momentum | (0.20, −2.00) | (−3.80, −0.40) | 9.22 | 42.01 |
| RMSProp | (0.50, −0.50) | (−3.50, 1.10) | 21.25 | 13.34 |
| Adam | (0.05, −0.05) | (−3.95, 1.55) | 37.83 | 35.73 |
- 1SGD moves 10 times further in than in because the gradient is 10 times larger there. It overshoots the valley floor from to .
- 2Momentum's first step equals SGD's (). On step 2 the stored velocity carries it further across the valley, to , and the loss jumps to 42. It needs many more steps to damp that swing.
- 3RMSProp's first step is per coordinate: it has no bias correction, so its first steps are large.
- 4Adam's first step is exactly in both coordinates, despite the 10× gradient ratio. That is the rescaling in action, and the reason it is so small is the bias correction.
Which one to use
| optimizer | typical settings | use it for |
|---|---|---|
| SGD + momentum (Nesterov) | (batch 256), , weight decay , plus a schedule | CNNs for vision (ResNets); often generalizes slightly better but needs tuning |
| Adam | or , | a strong default for prototyping, small models and RL |
| AdamW | to with warmup, to ; LLMs often use | transformers and almost all large-scale training |
| RMSProp | , to | older RNN and reinforcement-learning setups |
| AdaGrad | sparse features (linear models, embeddings); the step size decays permanently |
Adam-family optimizers store two extra tensors ( and ) per parameter, so optimizer state takes twice the memory of the weights themselves. SGD with momentum stores one. At billions of parameters that difference matters.
1import torch23opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.999), weight_decay=0.01)4# opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, nesterov=True, weight_decay=1e-4)56for x, y in loader:7 opt.zero_grad() # gradients accumulate otherwise8 loss = loss_fn(model(x), y)9 loss.backward() # fills p.grad for every parameter10 opt.step() # applies the update ruleCheck yourself
On its very first step, with ε negligible, how large is Adam's update to a parameter whose gradient is g?
Check yourself
Why is AdamW preferred over Adam with an L2 penalty?
Remember
- ✓All optimizers step against the gradient; they differ in how they smooth and scale it.
- ✓Plain gradient descent is capped at η < 2/λ_max, so ill-conditioned valleys are slow.
- ✓Momentum averages directions: oscillations cancel, consistent components accumulate.
- ✓RMSProp and Adam divide by the RMS gradient, giving every parameter a step of about η.
- ✓Adam = momentum + RMSProp + bias correction; AdamW decouples weight decay and is the transformer default.
- ✓The learning rate matters more than the choice of optimizer.