Cosine annealing

Decay the learning rate along half a cosine, from η_max to η_min over T steps: slow at first, fastest in the middle, flat at the end. SGDR adds warm restarts every T_i steps; without restarts, cosine is the default decay for transformers and LLMs.

10 min read749 wordscore0 viewsupdated 2026-10-11

Definition

Cosine annealing sets the learning rate to ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡(π t/T))\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\big(1+\cos(\pi\,t/T)\big): it starts at ηmax⁡\eta_{\max}, follows half a cosine wave, and lands at ηmin⁡\eta_{\min} after TT steps. With warm restarts the counter tt is reset periodically and the rate jumps back to ηmax⁡\eta_{\max}.

Loshchilov and Hutter introduced it in SGDR: Stochastic Gradient Descent with Warm Restarts (ICLR 2017). Their motivation was restarts: a periodic schedule gives a good model at the end of every cycle (anytime performance) and lets SGD explore again after settling. The restarts turned out to be optional. A single cosine cycle over the whole run, usually after a linear warmup, became the standard decay for ImageNet models and for most large language models (GPT-3, LLaMA and Chinchilla all decay cosine to 10% of the peak).

The formula

ηt=ηmin⁡i+12(ηmax⁡i−ηmin⁡i)(1+cos⁡(π TcurTi))\eta_t=\eta_{\min}^{i}+\tfrac12\big(\eta_{\max}^{i}-\eta_{\min}^{i}\big)\Big(1+\cos\Big(\pi\,\frac{T_{cur}}{T_i}\Big)\Big)
TcurT_{cur}
steps (or epochs, possibly fractional) since the last restart
TiT_i
length of cycle ii: T0T_0, then T0TmultT_0T_{mult}, T0Tmult2T_0T_{mult}^2, …
ηmax⁡, ηmin⁡\eta_{\max},\ \eta_{\min}
the rate at the start and at the end of a cycle; PyTorch takes ηmax⁡\eta_{\max} from the optimizer's lr and calls ηmin⁡\eta_{\min} eta_min

Its derivative is −π2T(ηmax⁡−ηmin⁡)sin⁡(πt/T)-\tfrac{\pi}{2T}(\eta_{\max}-\eta_{\min})\sin(\pi t/T): zero at both ends and largest in the middle. So cosine spends a long time near the peak (fast progress), decays fastest mid-run, and then lingers near ηmin⁡\eta_{\min} (noise removal) instead of crashing. At a quarter of the run the rate is still 0.854 ηmax⁡0.854\,\eta_{\max}, at half it is 0.5 ηmax⁡0.5\,\eta_{\max}, at three quarters 0.146 ηmax⁡0.146\,\eta_{\max} (for ηmin⁡=0\eta_{\min}=0).

Compared with step decay it has one hyperparameter, TT, and no milestones to tune, and it has no discontinuities, so the loss curve has no staircase. Compared with linear decay it keeps the rate high for longer and is gentler at the end. In practice cosine and linear-to-zero perform similarly, and both are usually at least as good as a well-tuned step decay.

CosineAnnealingLR(T_max=300, eta_min=0)0.000.501.00loss, mean of 24 noisy SGD runs (log)1e-31e-21e-11100100200300one run, last 60 stepsstep300 / 300η now0.0000274T_cur / T_max299/300loss now7.5e-3final, this7.6e-3final, step decay0.016━ cosine ━ step decay ×0.1 at 150, 225
run length 300 steps; f = ½(0.05x² + y²) with gradient noise
fig.SGD on the noisy quadratic f = ½(0.05x² + y²), gradient noise σ = 0.3, 300 steps, η₀ = 1, loss averaged over 24 seeds; the grey line is step decay (×0.1 at 150 and 225). A single cosine cycle with T_max = 300 ends at a final loss of 0.0076, against 0.016 for step decay. eta_min = 0.1 stops the decay early and ends at 0.011. Set T_max = 150 (too short): PyTorch's cosine keeps going and climbs back to the peak by step 300, ending at 0.066; T_max = 450 (too long) leaves the lr at 0.25 and ends at 0.020. Switch to warm restarts: with T_0 = 50, T_mult = 2 the cycles end at steps 50 and 150 (circles: 0.130, then 0.010), but the third cycle would end at 350, so the run stops mid-cycle at lr 0.15 with loss 0.019. T_0 = 20, T_mult = 2 ends cycles at 20, 60, 140 and exactly 300, finishing at 0.012.

Warm restarts (SGDR)

A restart sets Tcur=0T_{cur}=0, so the rate jumps from ηmin⁡\eta_{\min} back to ηmax⁡\eta_{\max}. The loss rises for a while (the noise ball expands again), then the next, longer, cycle brings it lower than before. With Tmult=2T_{mult}=2 the cycles double (T0,2T0,4T0,…T_0,2T_0,4T_0,\dots), so the restarts become rare as training goes on; SGDR's CIFAR experiments used schedules such as T0=10T_0=10 epochs with Tmult=2T_{mult}=2.

Each cycle end is a good model, which is what Snapshot Ensembles (Huang et al. 2017) exploit: save the weights at the end of every cycle and average the predictions, getting an ensemble for the cost of one training run. SWA (Izmailov et al. 2018) averages the weights themselves along a cyclical or constant-rate tail.

T_0, T_multcycle ends (restarts)loss at cycle endsfinal loss (last 20 steps)
50, 250, 150 (next would be 350)0.130, 0.0100.019 (stopped mid-cycle)
20, 220, 60, 140, 3000.460, 0.077, 0.0110.012
100, 1100, 200, 3000.024, 0.0120.018
no restart, T_max = 300300—0.0076
Restart steps and the loss at the end of each cycle in the figure (η₀ = 1, η_min = 0, 300-step run)

In these runs on a convex problem, restarts never beat a single cycle of the same total length: every restart throws away the noise reduction of the previous cycle. Restarts pay off when you want intermediate models (snapshots, early stopping, anytime evaluation), or on non-convex landscapes where re-exploration can find a better basin. That is why the field kept the cosine shape and mostly dropped the restarts.

Worked example: CosineAnnealingLR(T_max = 300)

  1. 1At t=75t=75: cos⁡(π/4)=0.7071\cos(\pi/4)=0.7071, so η=12(1+0.7071)=0.8536\eta=\tfrac12(1+0.7071)=0.8536. With ηmin⁡=0.1\eta_{\min}=0.1: 0.1+0.9⋅0.8536=0.86820.1+0.9\cdot0.8536=0.8682.
  2. 2At t=150t=150: cos⁡(π/2)=0\cos(\pi/2)=0, η=0.5\eta=0.5. At t=225t=225: cos⁡(3π/4)=−0.7071\cos(3\pi/4)=-0.7071, η=0.1464\eta=0.1464.
  3. 3Warm restarts with T0=10T_0=10, Tmult=2T_{mult}=2: cycle lengths 10, 20, 40, 80, so restarts at steps 10, 30, 70 and 150. At step 40, Tcur=10T_{cur}=10 of Ti=40T_i=40: η=12(1+cos⁡(π/4))ηmax⁡=0.854 ηmax⁡\eta=\tfrac12(1+\cos(\pi/4))\eta_{\max}=0.854\,\eta_{\max}.
  4. 4To make the last cycle end exactly at the end of training with Tmult=2T_{mult}=2, choose T=T0(2k−1)T=T_0(2^k-1): e.g. T0=20T_0=20 gives 20⋅15=30020\cdot15=300.
python
1import torch
2from torch.optim.lr_scheduler import (CosineAnnealingLR, CosineAnnealingWarmRestarts,
3 LinearLR, SequentialLR)
4
5opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
6iters = len(loader)
7total = epochs * iters
8
9# 1) One cosine cycle over the whole run (after a short warmup), stepped per iteration
10warm = 5 * iters
11sched = SequentialLR(opt, [
12 LinearLR(opt, start_factor=0.01, total_iters=warm),
13 CosineAnnealingLR(opt, T_max=total - warm, eta_min=0.0), # T_max = remaining steps
14], milestones=[warm])
15
16# 2) SGDR: T_0 and T_mult are in the unit you pass to step(); epochs here
17sgdr = CosineAnnealingWarmRestarts(opt, T_0=10, T_mult=2, eta_min=1e-4)
18for epoch in range(epochs):
19 for i, (x, y) in enumerate(loader):
20 opt.zero_grad()
21 loss_fn(model(x), y).backward()
22 opt.step()
23 sgdr.step(epoch + i / iters) # fractional epoch: smooth within an epoch
24 # cycle ends at epochs 10, 30, 70, 150: good moments to save a snapshot
Cosine with warmup, and warm restarts stepped per iteration

Use when

  • +One knob (T) instead of milestones and decay factors; smooth, no loss staircases
  • +Keeps the rate high early and low late: usually at or near the best simple decay
  • +Warm restarts give anytime snapshots and cheap ensembles

Avoid when

  • −Needs the run length in advance; extending training means re-planning the schedule (WSD avoids this)
  • −Mis-set TmaxT_{max} silently gives the wrong shape (rising rate, or never annealing)
  • −Restarts temporarily raise the loss and, on a fixed budget, rarely beat one cycle

Check yourself

You run CosineAnnealingLR(opt, T_max=100) and call scheduler.step() 200 times. What is the learning rate at the end?

Check yourself

CosineAnnealingWarmRestarts(T_0=20, T_mult=2) on a 300-step run: where are the restarts?

Remember

  • ✓ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡(πt/T))\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})(1+\cos(\pi t/T)): high for long, fastest decay mid-run, gentle landing.
  • ✓SGDR (Loshchilov and Hutter 2017) adds warm restarts with cycle lengths T0TmultiT_0T_{mult}^i; each cycle end is a usable snapshot.
  • ✓CosineAnnealingLR(T_max, eta_min) for one cycle (it rises again past TmaxT_{max}); CosineAnnealingWarmRestarts(T_0, T_mult, eta_min) for SGDR, steppable with fractional epochs.
  • ✓Warmup + single-cycle cosine to 0–10% of the peak is the default for transformers and LLMs.
  • ✓Match the cycle to the run: too short climbs back up, too long never anneals.
← back to Deep Learning roadmap