# Learning rate scheduling

> Change the learning rate during training: large early for fast progress, small late so SGD's noise ball shrinks. Step, exponential, polynomial, inverse-sqrt, WSD and plateau schedules all do this; in PyTorch they live in torch.optim.lr_scheduler.

Source: https://www.mlfoundry.in/learn/learning-rate-scheduling  
Section: Deep Learning › Optimization Algorithms  
Level: core · 12 min read · updated 2026-10-11

**A learning-rate schedule:** is a rule $\eta_t=\eta_0\,s(t)$ that sets the step size at every iteration $t$ (or epoch) instead of keeping it fixed. The optimizer's update rule is unchanged; only the scalar in front of it moves.

Almost every serious training run uses one. ResNets were trained with SGD at $\eta=0.1$, divided by 10 twice. The original Transformer used a warmup followed by inverse-square-root decay. Most large language models since GPT-3 use warmup plus cosine decay, and recent ones increasingly use warmup–stable–decay. The schedule is usually the second hyperparameter you tune, right after the peak learning rate itself.

The theory is old. Robbins and Monro (1951) showed that stochastic approximation converges if the steps satisfy $\sum_t\eta_t=\infty$ (you can still travel any distance) and $\sum_t\eta_t^2<\infty$ (the injected noise has finite total variance), for example $\eta_t\propto1/t$. Deep-learning schedules are much less aggressive than $1/t$, but they follow the same logic: big steps first, small steps last.

## Why decay helps: the noise ball

A mini-batch gradient is the full gradient plus noise, $g_t=\nabla L(\theta_t)+\xi_t$. With a constant learning rate SGD does not converge to the minimum. It converges to a cloud around it whose size is set by $\eta$: each step pulls the iterate in by a factor $(1-\eta\lambda)$ and kicks it out by $\eta\xi_t$. On a quadratic you can solve for the stationary spread exactly.

$$
\theta_{t+1}=(1-\eta\lambda)\,\theta_t-\eta\,\xi_t\;\Rightarrow\;\operatorname{Var}_\infty(\theta)=\frac{\eta\,\sigma^2}{\lambda\,(2-\eta\lambda)},\qquad \mathbb E\big[L-L^\star\big]_\infty=\frac{\eta\,\sigma^2}{2\,(2-\eta\lambda)}\approx\frac{\eta\sigma^2}{4}
$$

- $\lambda$: curvature of the direction (one eigenvalue of the Hessian), loss $\tfrac12\lambda\theta^2$
- $\sigma^2$: variance of the gradient noise in that direction; it grows as the batch shrinks
- $\eta$: the learning rate; stability needs $\eta<2/\lambda$

So the **excess loss is proportional to $\eta$**, independently of the curvature for small $\eta\lambda$. Halve the learning rate and the noise floor roughly halves. But a small $\eta$ makes progress along flat directions (small $\lambda$) slow, because each step only shrinks the distance by $\eta\lambda$. A constant learning rate must choose between fast progress and a low floor. A schedule gets both: a large $\eta$ while you are far from the minimum, a small one when noise is what is left.

That is why loss curves under step decay look like staircases: each drop of $\eta$ produces an almost immediate drop of the training loss to a new, lower floor. The drop is noise being removed, not new progress, and it happens fastest in the steep directions.

> Figure: SGD on f = ½(0.05x² + y²) with Gaussian gradient noise σ = 0.3, 300 steps, loss averaged over 24 seeds. The dashed orange line is the noise floor of the current learning rate, ησ²/(2(2 − ηλ)) summed over both directions. At base lr 1.0, constant SGD hovers at its floor (final loss 0.069), while step decay drops to 0.016 in two stairs, exponential to 0.0056 and polynomial (p = 2) to 0.0052. Lower the base lr to 0.1 and nothing decays usefully: the constant run (0.082) is now limited by slow progress along x, and every decaying schedule does the same or worse. Pick ReduceLROnPlateau to see it halve the lr at steps 120, 200 and 270 when the averaged loss stops improving.

## The common schedules

Closed forms with base rate $\eta_0$, step $t$ and run length $T$, as configured in the figure ($T=300$); lr values checked against the figure's code for $\eta_0=1$

| schedule | η_t | PyTorch | t = 0 | t = 150 | t = 299 |
| --- | --- | --- | --- | --- | --- |
| step | $\eta_0\,\gamma^{\#\{m_i\le t\}}$, $\gamma=0.1$, milestones 150, 225 | `MultiStepLR` / `StepLR` | 1 | 0.1 | 0.01 |
| exponential | $\eta_0\,\gamma^t$, $\gamma=0.01^{1/300}=0.98477$ | `ExponentialLR` | 1 | 0.1 | 0.0102 |
| polynomial | $\eta_0\,(1-t/T)^p$, $p=2$ | `PolynomialLR` | 1 | 0.25 | 0.00001 |
| inverse sqrt | $\eta_0\sqrt{\tau/\max(t,\tau)}$, $\tau=15$ | `LambdaLR` | 1 | 0.316 | 0.224 |
| WSD / trapezoid | linear warmup 15 steps, constant, linear decay to 0 over the last 60 | `SequentialLR` of `LinearLR` pieces | 0.067 | 1 | 0.017 |
| plateau | multiply by factor 0.5 when the monitored loss has not improved for patience 2 epochs | `ReduceLROnPlateau` | 1 | 0.5 | 0.125 |

**Step decay** (He et al. 2016 trained CIFAR-10 ResNets with $\eta=0.1$, divided by 10 at 32k and 48k iterations) is simple and still common in vision. **Exponential** decay is smooth but needs $\gamma$ tuned to the run length. **Polynomial** decay with $p=1$ is linear decay to zero, the schedule BERT used after its warmup. **Inverse square root**, $\eta\propto1/\sqrt t$, came with the Transformer (Vaswani et al. 2017) and decays slowly, so it does not need to know $T$. **Cosine** decay is the subject of its own article.

**Warmup–stable–decay** (WSD, also called trapezoidal): warm up, hold the peak for most of training, then decay quickly to zero in a short final "cooldown". MiniCPM (Hu et al. 2024) popularized the name, and Hägele et al. (2024) showed that a constant rate plus a cooldown of roughly 20% of training matches cosine. The practical win is that the stable phase does not depend on $T$: you can branch a cooldown off any checkpoint, or keep training longer, without restarting.

**ReduceLROnPlateau** is the only adaptive one: it watches a metric (usually validation loss) and multiplies the rate by `factor` once the metric has not improved by more than `threshold` for `patience` evaluations. It needs no knowledge of $T$, but it reacts late by design and is sensitive to noise in the metric.

## Worked example: what each schedule buys

The figure's problem at base lr 1.0: final loss = mean over the last 20 steps of the seed-averaged loss (checked by running the figure's code in Node)

| schedule | loss at step 150 | final loss | vs constant |
| --- | --- | --- | --- |
| constant | 0.104 | 0.069 | 1× |
| step (×0.1 at 150, 225) | 0.104 | 0.016 | 4.3× lower |
| exponential | 0.011 | 0.0056 | 12× lower |
| polynomial p = 2 | 0.016 | 0.0052 | 13× lower |
| inverse sqrt | 0.020 | 0.015 | 4.7× lower |
| WSD | 0.104 | 0.024 | 2.9× lower |
| ReduceLROnPlateau | 0.038 | 0.012 | 5.7× lower |

**Reading the numbers**

1. The floor formula at $\eta=1$: steep direction $0.09/(2\cdot1)=0.045$, flat direction $0.09/(2\cdot1.95)=0.023$, total $0.068$. The constant run ends at $0.069$, right on it.
2. At $\eta=0.1$ the floor would be $0.0046$, fifteen times lower, but a constant $0.1$ only shrinks $x$ by $1-0.1\cdot0.05=0.995$ per step: after 300 steps $x$ is still 22% of its start and the loss is $0.082$.
3. Exponential and polynomial decay win because they spend the first half at large $\eta$ (fast travel along $x$) and the second half sweeping the floor down. Step decay removes noise just as well at each drop, but after its first drop at step 150 the rate (0.1) is too small for the remaining spread along the flat direction to contract in time.
4. WSD's 60-step cooldown is short for this problem: the flat direction's noise relaxes at rate $\eta\lambda=0.05$ per step at best, so the $x$-cloud has no time to contract. Inverse sqrt never gets the rate low enough (it ends at $0.22\eta_0$).

> **Big early steps are not only about speed:** In real networks the high-learning-rate phase also matters for *where* training ends up. Large steps cannot settle in sharp minima (a minimum is stable only if $\eta<2/\lambda_{\max}$), so the early phase biases training toward flatter regions, and decaying too early often gives a lower training loss but a worse test accuracy. A convex quadratic cannot show this; it only shows the noise half of the story.

## PyTorch: the scheduler API

```python
import math, torch
from torch.optim.lr_scheduler import (MultiStepLR, ExponentialLR, PolynomialLR,
                                      LambdaLR, LinearLR, ConstantLR, SequentialLR,
                                      ReduceLROnPlateau)

opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
total = epochs * len(loader)                     # schedule in iterations, not epochs

# Pick one:
sched = MultiStepLR(opt, milestones=[total // 2, 3 * total // 4], gamma=0.1)
sched = ExponentialLR(opt, gamma=0.01 ** (1 / total))
sched = PolynomialLR(opt, total_iters=total, power=2.0)
sched = LambdaLR(opt, lambda t: min(1.0, math.sqrt(1000 / max(t, 1))))   # factor x base lr

# WSD: 5% linear warmup, constant, 20% linear decay to zero
w, d = int(0.05 * total), int(0.2 * total)
sched = SequentialLR(opt, schedulers=[
    LinearLR(opt, start_factor=1e-3, end_factor=1.0, total_iters=w),
    ConstantLR(opt, factor=1.0, total_iters=total - w - d),
    LinearLR(opt, start_factor=1.0, end_factor=0.0, total_iters=d),
], milestones=[w, total - d])

for epoch in range(epochs):
    for x, y in loader:
        opt.zero_grad()
        loss_fn(model(x), y).backward()
        opt.step()
        sched.step()                 # AFTER opt.step(), once per iteration
    print(epoch, sched.get_last_lr())

# Plateau: step once per evaluation, with the metric
plateau = ReduceLROnPlateau(opt, mode="min", factor=0.5, patience=2, threshold=1e-4)
for epoch in range(epochs):
    train_one_epoch(model, opt)
    plateau.step(validate(model))    # pass the monitored metric
```

Per-iteration scheduling, composite schedules, and plateau scheduling

> **Where scheduler.step() goes:** Since PyTorch 1.1, call `optimizer.step()` **before** `scheduler.step()`; the reverse order skips the first value of the schedule and triggers a warning. Then decide the unit: a scheduler built with milestones or `total_iters` in **epochs** must be stepped once per epoch, one built in **iterations** once per batch. Mixing them (epoch milestones, stepped every batch) decays the rate hundreds of times too early, the most common scheduling bug.

> **Resuming training:** A scheduler has state (`last_epoch`, plateau counters). Save `sched.state_dict()` with the model and optimizer and restore it on resume; otherwise the schedule restarts from step 0 and you get a second warmup at full learning rate, or no decay at all.

> **Schedulers multiply, they do not override:** Every scheduler rescales each param group's `initial_lr`, so per-group learning rates (for example a 10× smaller rate for a pretrained backbone) keep their ratio through the whole schedule. `LambdaLR`'s function returns a factor, not a learning rate.

**Strengths**

- Lower final loss for free: same steps, same model, often several times lower training loss than the best constant rate
- Lets you use a larger peak rate, which speeds up early training
- WSD and plateau schedules do not need the run length in advance

**Limitations**

- More hyperparameters: decay shape, milestones, final rate, and they interact with the peak rate and batch size
- Schedules tied to $T$ (step, polynomial, cosine) are wrong if you later train longer or stop early
- ReduceLROnPlateau reacts late and to noise; patience and threshold need care

**Question:** With a constant learning rate η, SGD on a noisy quadratic settles at an expected excess loss of about ησ²/4 per direction. What happens to the final loss if you halve η and train much longer?

- A. It roughly halves
- B. It stays the same: only the batch size matters
- C. It quarters, because the loss is quadratic
- D. It doubles, because progress is slower

**Answer:** A. It roughly halves — The stationary excess loss is $\eta\sigma^2/(2(2-\eta\lambda))$, linear in $\eta$ for small $\eta\lambda$. Halving $\eta$ halves the floor; it just takes about twice as many steps to get there. Decaying schedules exploit exactly this.

**Question:** You build MultiStepLR(opt, milestones=[30, 60]) for a 90-epoch run, but call scheduler.step() after every batch (1000 batches per epoch). What happens?

- A. The lr drops by 10× after 30 and 60 batches, so almost all training runs at 1% of the base lr
- B. Nothing: PyTorch detects the unit
- C. The lr drops at epochs 30 and 60 as intended
- D. PyTorch raises an error

**Answer:** A. The lr drops by 10× after 30 and 60 batches, so almost all training runs at 1% of the base lr — Schedulers count calls to `step()`, not epochs. With per-batch calls the milestones are reached after 30 and 60 iterations, so the run spends 89.9 of its 90 epochs at $0.01\eta_0$. Either step per epoch or convert the milestones to iterations.

**Key points**

- Constant-$\eta$ SGD converges to a noise ball with excess loss $\approx\eta\sigma^2/4$ per direction: proportional to $\eta$.
- Large early rates travel fast along flat directions; small late rates shrink the noise floor. Schedules get both.
- Step, exponential, polynomial, inverse-sqrt, WSD and ReduceLROnPlateau differ in shape and in whether they need the run length.
- In PyTorch: `optimizer.step()` then `scheduler.step()`, and step in the same unit (iterations or epochs) the schedule was built in.
- Compose schedules with `SequentialLR` or write any shape as a factor with `LambdaLR`; save the scheduler state when checkpointing.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
