Definition
Cosine annealing sets the learning rate to : it starts at , follows half a cosine wave, and lands at after steps. With warm restarts the counter is reset periodically and the rate jumps back to .
Loshchilov and Hutter introduced it in SGDR: Stochastic Gradient Descent with Warm Restarts (ICLR 2017). Their motivation was restarts: a periodic schedule gives a good model at the end of every cycle (anytime performance) and lets SGD explore again after settling. The restarts turned out to be optional. A single cosine cycle over the whole run, usually after a linear warmup, became the standard decay for ImageNet models and for most large language models (GPT-3, LLaMA and Chinchilla all decay cosine to 10% of the peak).
The formula
- steps (or epochs, possibly fractional) since the last restart
- length of cycle : , then , , …
- the rate at the start and at the end of a cycle; PyTorch takes from the optimizer's lr and calls
eta_min
Its derivative is : zero at both ends and largest in the middle. So cosine spends a long time near the peak (fast progress), decays fastest mid-run, and then lingers near (noise removal) instead of crashing. At a quarter of the run the rate is still , at half it is , at three quarters (for ).
Compared with step decay it has one hyperparameter, , and no milestones to tune, and it has no discontinuities, so the loss curve has no staircase. Compared with linear decay it keeps the rate high for longer and is gentler at the end. In practice cosine and linear-to-zero perform similarly, and both are usually at least as good as a well-tuned step decay.
Warm restarts (SGDR)
A restart sets , so the rate jumps from back to . The loss rises for a while (the noise ball expands again), then the next, longer, cycle brings it lower than before. With the cycles double (), so the restarts become rare as training goes on; SGDR's CIFAR experiments used schedules such as epochs with .
Each cycle end is a good model, which is what Snapshot Ensembles (Huang et al. 2017) exploit: save the weights at the end of every cycle and average the predictions, getting an ensemble for the cost of one training run. SWA (Izmailov et al. 2018) averages the weights themselves along a cyclical or constant-rate tail.
| T_0, T_mult | cycle ends (restarts) | loss at cycle ends | final loss (last 20 steps) |
|---|---|---|---|
| 50, 2 | 50, 150 (next would be 350) | 0.130, 0.010 | 0.019 (stopped mid-cycle) |
| 20, 2 | 20, 60, 140, 300 | 0.460, 0.077, 0.011 | 0.012 |
| 100, 1 | 100, 200, 300 | 0.024, 0.012 | 0.018 |
| no restart, T_max = 300 | 300 | — | 0.0076 |
In these runs on a convex problem, restarts never beat a single cycle of the same total length: every restart throws away the noise reduction of the previous cycle. Restarts pay off when you want intermediate models (snapshots, early stopping, anytime evaluation), or on non-convex landscapes where re-exploration can find a better basin. That is why the field kept the cosine shape and mostly dropped the restarts.
Worked example: CosineAnnealingLR(T_max = 300)
- 1At : , so . With : .
- 2At : , . At : , .
- 3Warm restarts with , : cycle lengths 10, 20, 40, 80, so restarts at steps 10, 30, 70 and 150. At step 40, of : .
- 4To make the last cycle end exactly at the end of training with , choose : e.g. gives .
1import torch2from torch.optim.lr_scheduler import (CosineAnnealingLR, CosineAnnealingWarmRestarts,3 LinearLR, SequentialLR)45opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)6iters = len(loader)7total = epochs * iters89# 1) One cosine cycle over the whole run (after a short warmup), stepped per iteration10warm = 5 * iters11sched = SequentialLR(opt, [12 LinearLR(opt, start_factor=0.01, total_iters=warm),13 CosineAnnealingLR(opt, T_max=total - warm, eta_min=0.0), # T_max = remaining steps14], milestones=[warm])1516# 2) SGDR: T_0 and T_mult are in the unit you pass to step(); epochs here17sgdr = CosineAnnealingWarmRestarts(opt, T_0=10, T_mult=2, eta_min=1e-4)18for epoch in range(epochs):19 for i, (x, y) in enumerate(loader):20 opt.zero_grad()21 loss_fn(model(x), y).backward()22 opt.step()23 sgdr.step(epoch + i / iters) # fractional epoch: smooth within an epoch24 # cycle ends at epochs 10, 30, 70, 150: good moments to save a snapshotUse when
- +One knob (T) instead of milestones and decay factors; smooth, no loss staircases
- +Keeps the rate high early and low late: usually at or near the best simple decay
- +Warm restarts give anytime snapshots and cheap ensembles
Avoid when
- −Needs the run length in advance; extending training means re-planning the schedule (WSD avoids this)
- −Mis-set silently gives the wrong shape (rising rate, or never annealing)
- −Restarts temporarily raise the loss and, on a fixed budget, rarely beat one cycle
Check yourself
You run CosineAnnealingLR(opt, T_max=100) and call scheduler.step() 200 times. What is the learning rate at the end?
Check yourself
CosineAnnealingWarmRestarts(T_0=20, T_mult=2) on a 300-step run: where are the restarts?
Remember
- ✓: high for long, fastest decay mid-run, gentle landing.
- ✓SGDR (Loshchilov and Hutter 2017) adds warm restarts with cycle lengths ; each cycle end is a usable snapshot.
- ✓
CosineAnnealingLR(T_max, eta_min)for one cycle (it rises again past );CosineAnnealingWarmRestarts(T_0, T_mult, eta_min)for SGDR, steppable with fractional epochs. - ✓Warmup + single-cycle cosine to 0–10% of the peak is the default for transformers and LLMs.
- ✓Match the cycle to the run: too short climbs back up, too long never anneals.