Definition
A learning-rate schedule is a rule that sets the step size at every iteration (or epoch) instead of keeping it fixed. The optimizer's update rule is unchanged; only the scalar in front of it moves.
Almost every serious training run uses one. ResNets were trained with SGD at , divided by 10 twice. The original Transformer used a warmup followed by inverse-square-root decay. Most large language models since GPT-3 use warmup plus cosine decay, and recent ones increasingly use warmup–stable–decay. The schedule is usually the second hyperparameter you tune, right after the peak learning rate itself.
The theory is old. Robbins and Monro (1951) showed that stochastic approximation converges if the steps satisfy (you can still travel any distance) and (the injected noise has finite total variance), for example . Deep-learning schedules are much less aggressive than , but they follow the same logic: big steps first, small steps last.
Why decay helps: the noise ball
A mini-batch gradient is the full gradient plus noise, . With a constant learning rate SGD does not converge to the minimum. It converges to a cloud around it whose size is set by : each step pulls the iterate in by a factor and kicks it out by . On a quadratic you can solve for the stationary spread exactly.
- curvature of the direction (one eigenvalue of the Hessian), loss
- variance of the gradient noise in that direction; it grows as the batch shrinks
- the learning rate; stability needs
So the excess loss is proportional to , independently of the curvature for small . Halve the learning rate and the noise floor roughly halves. But a small makes progress along flat directions (small ) slow, because each step only shrinks the distance by . A constant learning rate must choose between fast progress and a low floor. A schedule gets both: a large while you are far from the minimum, a small one when noise is what is left.
That is why loss curves under step decay look like staircases: each drop of produces an almost immediate drop of the training loss to a new, lower floor. The drop is noise being removed, not new progress, and it happens fastest in the steep directions.
The common schedules
| schedule | η_t | PyTorch | t = 0 | t = 150 | t = 299 |
|---|---|---|---|---|---|
| step | , , milestones 150, 225 | MultiStepLR / StepLR | 1 | 0.1 | 0.01 |
| exponential | , | ExponentialLR | 1 | 0.1 | 0.0102 |
| polynomial | , | PolynomialLR | 1 | 0.25 | 0.00001 |
| inverse sqrt | , | LambdaLR | 1 | 0.316 | 0.224 |
| WSD / trapezoid | linear warmup 15 steps, constant, linear decay to 0 over the last 60 | SequentialLR of LinearLR pieces | 0.067 | 1 | 0.017 |
| plateau | multiply by factor 0.5 when the monitored loss has not improved for patience 2 epochs | ReduceLROnPlateau | 1 | 0.5 | 0.125 |
Step decay (He et al. 2016 trained CIFAR-10 ResNets with , divided by 10 at 32k and 48k iterations) is simple and still common in vision. Exponential decay is smooth but needs tuned to the run length. Polynomial decay with is linear decay to zero, the schedule BERT used after its warmup. Inverse square root, , came with the Transformer (Vaswani et al. 2017) and decays slowly, so it does not need to know . Cosine decay is the subject of its own article.
Warmup–stable–decay (WSD, also called trapezoidal): warm up, hold the peak for most of training, then decay quickly to zero in a short final "cooldown". MiniCPM (Hu et al. 2024) popularized the name, and Hägele et al. (2024) showed that a constant rate plus a cooldown of roughly 20% of training matches cosine. The practical win is that the stable phase does not depend on : you can branch a cooldown off any checkpoint, or keep training longer, without restarting.
ReduceLROnPlateau is the only adaptive one: it watches a metric (usually validation loss) and multiplies the rate by factor once the metric has not improved by more than threshold for patience evaluations. It needs no knowledge of , but it reacts late by design and is sensitive to noise in the metric.
Worked example: what each schedule buys
| schedule | loss at step 150 | final loss | vs constant |
|---|---|---|---|
| constant | 0.104 | 0.069 | 1× |
| step (×0.1 at 150, 225) | 0.104 | 0.016 | 4.3× lower |
| exponential | 0.011 | 0.0056 | 12× lower |
| polynomial p = 2 | 0.016 | 0.0052 | 13× lower |
| inverse sqrt | 0.020 | 0.015 | 4.7× lower |
| WSD | 0.104 | 0.024 | 2.9× lower |
| ReduceLROnPlateau | 0.038 | 0.012 | 5.7× lower |
Reading the numbers
- 1The floor formula at : steep direction , flat direction , total . The constant run ends at , right on it.
- 2At the floor would be , fifteen times lower, but a constant only shrinks by per step: after 300 steps is still 22% of its start and the loss is .
- 3Exponential and polynomial decay win because they spend the first half at large (fast travel along ) and the second half sweeping the floor down. Step decay removes noise just as well at each drop, but after its first drop at step 150 the rate (0.1) is too small for the remaining spread along the flat direction to contract in time.
- 4WSD's 60-step cooldown is short for this problem: the flat direction's noise relaxes at rate per step at best, so the -cloud has no time to contract. Inverse sqrt never gets the rate low enough (it ends at ).
PyTorch: the scheduler API
1import math, torch2from torch.optim.lr_scheduler import (MultiStepLR, ExponentialLR, PolynomialLR,3 LambdaLR, LinearLR, ConstantLR, SequentialLR,4 ReduceLROnPlateau)56opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)7total = epochs * len(loader) # schedule in iterations, not epochs89# Pick one:10sched = MultiStepLR(opt, milestones=[total // 2, 3 * total // 4], gamma=0.1)11sched = ExponentialLR(opt, gamma=0.01 ** (1 / total))12sched = PolynomialLR(opt, total_iters=total, power=2.0)13sched = LambdaLR(opt, lambda t: min(1.0, math.sqrt(1000 / max(t, 1)))) # factor x base lr1415# WSD: 5% linear warmup, constant, 20% linear decay to zero16w, d = int(0.05 * total), int(0.2 * total)17sched = SequentialLR(opt, schedulers=[18 LinearLR(opt, start_factor=1e-3, end_factor=1.0, total_iters=w),19 ConstantLR(opt, factor=1.0, total_iters=total - w - d),20 LinearLR(opt, start_factor=1.0, end_factor=0.0, total_iters=d),21], milestones=[w, total - d])2223for epoch in range(epochs):24 for x, y in loader:25 opt.zero_grad()26 loss_fn(model(x), y).backward()27 opt.step()28 sched.step() # AFTER opt.step(), once per iteration29 print(epoch, sched.get_last_lr())3031# Plateau: step once per evaluation, with the metric32plateau = ReduceLROnPlateau(opt, mode="min", factor=0.5, patience=2, threshold=1e-4)33for epoch in range(epochs):34 train_one_epoch(model, opt)35 plateau.step(validate(model)) # pass the monitored metricUse when
- +Lower final loss for free: same steps, same model, often several times lower training loss than the best constant rate
- +Lets you use a larger peak rate, which speeds up early training
- +WSD and plateau schedules do not need the run length in advance
Avoid when
- −More hyperparameters: decay shape, milestones, final rate, and they interact with the peak rate and batch size
- −Schedules tied to (step, polynomial, cosine) are wrong if you later train longer or stop early
- −ReduceLROnPlateau reacts late and to noise; patience and threshold need care
Check yourself
With a constant learning rate η, SGD on a noisy quadratic settles at an expected excess loss of about ησ²/4 per direction. What happens to the final loss if you halve η and train much longer?
Check yourself
You build MultiStepLR(opt, milestones=[30, 60]) for a 90-epoch run, but call scheduler.step() after every batch (1000 batches per epoch). What happens?
Remember
- ✓Constant- SGD converges to a noise ball with excess loss per direction: proportional to .
- ✓Large early rates travel fast along flat directions; small late rates shrink the noise floor. Schedules get both.
- ✓Step, exponential, polynomial, inverse-sqrt, WSD and ReduceLROnPlateau differ in shape and in whether they need the run length.
- ✓In PyTorch:
optimizer.step()thenscheduler.step(), and step in the same unit (iterations or epochs) the schedule was built in. - ✓Compose schedules with
SequentialLRor write any shape as a factor withLambdaLR; save the scheduler state when checkpointing.