Learning rate scheduling

Change the learning rate during training: large early for fast progress, small late so SGD's noise ball shrinks. Step, exponential, polynomial, inverse-sqrt, WSD and plateau schedules all do this; in PyTorch they live in torch.optim.lr_scheduler.

12 min read1021 wordscore0 viewsupdated 2026-10-11

Definition

A learning-rate schedule is a rule ηt=η0 s(t)\eta_t=\eta_0\,s(t) that sets the step size at every iteration tt (or epoch) instead of keeping it fixed. The optimizer's update rule is unchanged; only the scalar in front of it moves.

Almost every serious training run uses one. ResNets were trained with SGD at η=0.1\eta=0.1, divided by 10 twice. The original Transformer used a warmup followed by inverse-square-root decay. Most large language models since GPT-3 use warmup plus cosine decay, and recent ones increasingly use warmup–stable–decay. The schedule is usually the second hyperparameter you tune, right after the peak learning rate itself.

The theory is old. Robbins and Monro (1951) showed that stochastic approximation converges if the steps satisfy ∑tηt=∞\sum_t\eta_t=\infty (you can still travel any distance) and ∑tηt2<∞\sum_t\eta_t^2<\infty (the injected noise has finite total variance), for example ηt∝1/t\eta_t\propto1/t. Deep-learning schedules are much less aggressive than 1/t1/t, but they follow the same logic: big steps first, small steps last.

Why decay helps: the noise ball

A mini-batch gradient is the full gradient plus noise, gt=∇L(θt)+ξtg_t=\nabla L(\theta_t)+\xi_t. With a constant learning rate SGD does not converge to the minimum. It converges to a cloud around it whose size is set by η\eta: each step pulls the iterate in by a factor (1−ηλ)(1-\eta\lambda) and kicks it out by ηξt\eta\xi_t. On a quadratic you can solve for the stationary spread exactly.

θt+1=(1−ηλ) θt−η ξt  ⇒  Var⁡∞(θ)=η σ2λ (2−ηλ),E[L−L⋆]∞=η σ22 (2−ηλ)≈ησ24\theta_{t+1}=(1-\eta\lambda)\,\theta_t-\eta\,\xi_t\;\Rightarrow\;\operatorname{Var}_\infty(\theta)=\frac{\eta\,\sigma^2}{\lambda\,(2-\eta\lambda)},\qquad \mathbb E\big[L-L^\star\big]_\infty=\frac{\eta\,\sigma^2}{2\,(2-\eta\lambda)}\approx\frac{\eta\sigma^2}{4}
λ\lambda
curvature of the direction (one eigenvalue of the Hessian), loss 12λθ2\tfrac12\lambda\theta^2
σ2\sigma^2
variance of the gradient noise in that direction; it grows as the batch shrinks
η\eta
the learning rate; stability needs η<2/λ\eta<2/\lambda

So the excess loss is proportional to η\eta, independently of the curvature for small ηλ\eta\lambda. Halve the learning rate and the noise floor roughly halves. But a small η\eta makes progress along flat directions (small λ\lambda) slow, because each step only shrinks the distance by ηλ\eta\lambda. A constant learning rate must choose between fast progress and a low floor. A schedule gets both: a large η\eta while you are far from the minimum, a small one when noise is what is left.

That is why loss curves under step decay look like staircases: each drop of η\eta produces an almost immediate drop of the training loss to a new, lower floor. The drop is noise being removed, not new progress, and it happens fastest in the steep directions.

learning rate η(t): step (MultiStepLR)0.000.501.00loss, mean of 24 noisy runs (log)1e-31e-21e-11100100200300one run on f = ½(0.05x² + y²)min at (0, 0); last 60 steps drawnstep300 / 300η now0.0100loss now0.016floor(η)4.5e-4final, this0.016final, const0.069━ this schedule ━ constant η₀ ┄ noise floor of the current η · σ = 0.3
SGD, gradient = ∇f + N(0, σ²) noise; stable for η < 2
fig.SGD on f = ½(0.05x² + y²) with Gaussian gradient noise σ = 0.3, 300 steps, loss averaged over 24 seeds. The dashed orange line is the noise floor of the current learning rate, ησ²/(2(2 − ηλ)) summed over both directions. At base lr 1.0, constant SGD hovers at its floor (final loss 0.069), while step decay drops to 0.016 in two stairs, exponential to 0.0056 and polynomial (p = 2) to 0.0052. Lower the base lr to 0.1 and nothing decays usefully: the constant run (0.082) is now limited by slow progress along x, and every decaying schedule does the same or worse. Pick ReduceLROnPlateau to see it halve the lr at steps 120, 200 and 270 when the averaged loss stops improving.

The common schedules

scheduleη_tPyTorcht = 0t = 150t = 299
stepη0 γ#{mi≤t}\eta_0\,\gamma^{\#\{m_i\le t\}}, γ=0.1\gamma=0.1, milestones 150, 225MultiStepLR / StepLR10.10.01
exponentialη0 γt\eta_0\,\gamma^t, γ=0.011/300=0.98477\gamma=0.01^{1/300}=0.98477ExponentialLR10.10.0102
polynomialη0 (1−t/T)p\eta_0\,(1-t/T)^p, p=2p=2PolynomialLR10.250.00001
inverse sqrtη0τ/max⁡(t,τ)\eta_0\sqrt{\tau/\max(t,\tau)}, τ=15\tau=15LambdaLR10.3160.224
WSD / trapezoidlinear warmup 15 steps, constant, linear decay to 0 over the last 60SequentialLR of LinearLR pieces0.06710.017
plateaumultiply by factor 0.5 when the monitored loss has not improved for patience 2 epochsReduceLROnPlateau10.50.125
Closed forms with base rate η0\eta_0, step tt and run length TT, as configured in the figure (T=300T=300); lr values checked against the figure's code for η0=1\eta_0=1

Step decay (He et al. 2016 trained CIFAR-10 ResNets with η=0.1\eta=0.1, divided by 10 at 32k and 48k iterations) is simple and still common in vision. Exponential decay is smooth but needs γ\gamma tuned to the run length. Polynomial decay with p=1p=1 is linear decay to zero, the schedule BERT used after its warmup. Inverse square root, η∝1/t\eta\propto1/\sqrt t, came with the Transformer (Vaswani et al. 2017) and decays slowly, so it does not need to know TT. Cosine decay is the subject of its own article.

Warmup–stable–decay (WSD, also called trapezoidal): warm up, hold the peak for most of training, then decay quickly to zero in a short final "cooldown". MiniCPM (Hu et al. 2024) popularized the name, and Hägele et al. (2024) showed that a constant rate plus a cooldown of roughly 20% of training matches cosine. The practical win is that the stable phase does not depend on TT: you can branch a cooldown off any checkpoint, or keep training longer, without restarting.

ReduceLROnPlateau is the only adaptive one: it watches a metric (usually validation loss) and multiplies the rate by factor once the metric has not improved by more than threshold for patience evaluations. It needs no knowledge of TT, but it reacts late by design and is sensitive to noise in the metric.

Worked example: what each schedule buys

scheduleloss at step 150final lossvs constant
constant0.1040.0691×
step (×0.1 at 150, 225)0.1040.0164.3× lower
exponential0.0110.005612× lower
polynomial p = 20.0160.005213× lower
inverse sqrt0.0200.0154.7× lower
WSD0.1040.0242.9× lower
ReduceLROnPlateau0.0380.0125.7× lower
The figure's problem at base lr 1.0: final loss = mean over the last 20 steps of the seed-averaged loss (checked by running the figure's code in Node)

Reading the numbers

  1. 1The floor formula at η=1\eta=1: steep direction 0.09/(2⋅1)=0.0450.09/(2\cdot1)=0.045, flat direction 0.09/(2⋅1.95)=0.0230.09/(2\cdot1.95)=0.023, total 0.0680.068. The constant run ends at 0.0690.069, right on it.
  2. 2At η=0.1\eta=0.1 the floor would be 0.00460.0046, fifteen times lower, but a constant 0.10.1 only shrinks xx by 1−0.1⋅0.05=0.9951-0.1\cdot0.05=0.995 per step: after 300 steps xx is still 22% of its start and the loss is 0.0820.082.
  3. 3Exponential and polynomial decay win because they spend the first half at large η\eta (fast travel along xx) and the second half sweeping the floor down. Step decay removes noise just as well at each drop, but after its first drop at step 150 the rate (0.1) is too small for the remaining spread along the flat direction to contract in time.
  4. 4WSD's 60-step cooldown is short for this problem: the flat direction's noise relaxes at rate ηλ=0.05\eta\lambda=0.05 per step at best, so the xx-cloud has no time to contract. Inverse sqrt never gets the rate low enough (it ends at 0.22η00.22\eta_0).

PyTorch: the scheduler API

python
1import math, torch
2from torch.optim.lr_scheduler import (MultiStepLR, ExponentialLR, PolynomialLR,
3 LambdaLR, LinearLR, ConstantLR, SequentialLR,
4 ReduceLROnPlateau)
5
6opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
7total = epochs * len(loader) # schedule in iterations, not epochs
8
9# Pick one:
10sched = MultiStepLR(opt, milestones=[total // 2, 3 * total // 4], gamma=0.1)
11sched = ExponentialLR(opt, gamma=0.01 ** (1 / total))
12sched = PolynomialLR(opt, total_iters=total, power=2.0)
13sched = LambdaLR(opt, lambda t: min(1.0, math.sqrt(1000 / max(t, 1)))) # factor x base lr
14
15# WSD: 5% linear warmup, constant, 20% linear decay to zero
16w, d = int(0.05 * total), int(0.2 * total)
17sched = SequentialLR(opt, schedulers=[
18 LinearLR(opt, start_factor=1e-3, end_factor=1.0, total_iters=w),
19 ConstantLR(opt, factor=1.0, total_iters=total - w - d),
20 LinearLR(opt, start_factor=1.0, end_factor=0.0, total_iters=d),
21], milestones=[w, total - d])
22
23for epoch in range(epochs):
24 for x, y in loader:
25 opt.zero_grad()
26 loss_fn(model(x), y).backward()
27 opt.step()
28 sched.step() # AFTER opt.step(), once per iteration
29 print(epoch, sched.get_last_lr())
30
31# Plateau: step once per evaluation, with the metric
32plateau = ReduceLROnPlateau(opt, mode="min", factor=0.5, patience=2, threshold=1e-4)
33for epoch in range(epochs):
34 train_one_epoch(model, opt)
35 plateau.step(validate(model)) # pass the monitored metric
Per-iteration scheduling, composite schedules, and plateau scheduling

Use when

  • +Lower final loss for free: same steps, same model, often several times lower training loss than the best constant rate
  • +Lets you use a larger peak rate, which speeds up early training
  • +WSD and plateau schedules do not need the run length in advance

Avoid when

  • −More hyperparameters: decay shape, milestones, final rate, and they interact with the peak rate and batch size
  • −Schedules tied to TT (step, polynomial, cosine) are wrong if you later train longer or stop early
  • −ReduceLROnPlateau reacts late and to noise; patience and threshold need care

Check yourself

With a constant learning rate η, SGD on a noisy quadratic settles at an expected excess loss of about ησ²/4 per direction. What happens to the final loss if you halve η and train much longer?

Check yourself

You build MultiStepLR(opt, milestones=[30, 60]) for a 90-epoch run, but call scheduler.step() after every batch (1000 batches per epoch). What happens?

Remember

  • ✓Constant-η\eta SGD converges to a noise ball with excess loss ≈ησ2/4\approx\eta\sigma^2/4 per direction: proportional to η\eta.
  • ✓Large early rates travel fast along flat directions; small late rates shrink the noise floor. Schedules get both.
  • ✓Step, exponential, polynomial, inverse-sqrt, WSD and ReduceLROnPlateau differ in shape and in whether they need the run length.
  • ✓In PyTorch: optimizer.step() then scheduler.step(), and step in the same unit (iterations or epochs) the schedule was built in.
  • ✓Compose schedules with SequentialLR or write any shape as a factor with LambdaLR; save the scheduler state when checkpointing.
← back to Deep Learning roadmap