# Cosine annealing

> Decay the learning rate along half a cosine, from η_max to η_min over T steps: slow at first, fastest in the middle, flat at the end. SGDR adds warm restarts every T_i steps; without restarts, cosine is the default decay for transformers and LLMs.

Source: https://www.mlfoundry.in/learn/cosine-annealing  
Section: Deep Learning › Optimization Algorithms  
Level: core · 10 min read · updated 2026-10-11

**Cosine annealing:** sets the learning rate to $\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\big(1+\cos(\pi\,t/T)\big)$: it starts at $\eta_{\max}$, follows half a cosine wave, and lands at $\eta_{\min}$ after $T$ steps. With **warm restarts** the counter $t$ is reset periodically and the rate jumps back to $\eta_{\max}$.

Loshchilov and Hutter introduced it in **SGDR: Stochastic Gradient Descent with Warm Restarts** (ICLR 2017). Their motivation was restarts: a periodic schedule gives a good model at the end of every cycle (anytime performance) and lets SGD explore again after settling. The restarts turned out to be optional. A single cosine cycle over the whole run, usually after a linear warmup, became the standard decay for ImageNet models and for most large language models (GPT-3, LLaMA and Chinchilla all decay cosine to 10% of the peak).

## The formula

$$
\eta_t=\eta_{\min}^{i}+\tfrac12\big(\eta_{\max}^{i}-\eta_{\min}^{i}\big)\Big(1+\cos\Big(\pi\,\frac{T_{cur}}{T_i}\Big)\Big)
$$

- $T_{cur}$: steps (or epochs, possibly fractional) since the last restart
- $T_i$: length of cycle $i$: $T_0$, then $T_0T_{mult}$, $T_0T_{mult}^2$, …
- $\eta_{\max},\ \eta_{\min}$: the rate at the start and at the end of a cycle; PyTorch takes $\eta_{\max}$ from the optimizer's lr and calls $\eta_{\min}$ `eta_min`

Its derivative is $-\tfrac{\pi}{2T}(\eta_{\max}-\eta_{\min})\sin(\pi t/T)$: zero at both ends and largest in the middle. So cosine spends a long time near the peak (fast progress), decays fastest mid-run, and then lingers near $\eta_{\min}$ (noise removal) instead of crashing. At a quarter of the run the rate is still $0.854\,\eta_{\max}$, at half it is $0.5\,\eta_{\max}$, at three quarters $0.146\,\eta_{\max}$ (for $\eta_{\min}=0$).

Compared with step decay it has one hyperparameter, $T$, and no milestones to tune, and it has no discontinuities, so the loss curve has no staircase. Compared with linear decay it keeps the rate high for longer and is gentler at the end. In practice cosine and linear-to-zero perform similarly, and both are usually at least as good as a well-tuned step decay.

> Figure: SGD on the noisy quadratic f = ½(0.05x² + y²), gradient noise σ = 0.3, 300 steps, η₀ = 1, loss averaged over 24 seeds; the grey line is step decay (×0.1 at 150 and 225). A single cosine cycle with T_max = 300 ends at a final loss of 0.0076, against 0.016 for step decay. eta_min = 0.1 stops the decay early and ends at 0.011. Set T_max = 150 (too short): PyTorch's cosine keeps going and climbs back to the peak by step 300, ending at 0.066; T_max = 450 (too long) leaves the lr at 0.25 and ends at 0.020. Switch to warm restarts: with T_0 = 50, T_mult = 2 the cycles end at steps 50 and 150 (circles: 0.130, then 0.010), but the third cycle would end at 350, so the run stops mid-cycle at lr 0.15 with loss 0.019. T_0 = 20, T_mult = 2 ends cycles at 20, 60, 140 and exactly 300, finishing at 0.012.

## Warm restarts (SGDR)

A restart sets $T_{cur}=0$, so the rate jumps from $\eta_{\min}$ back to $\eta_{\max}$. The loss rises for a while (the noise ball expands again), then the next, longer, cycle brings it lower than before. With $T_{mult}=2$ the cycles double ($T_0,2T_0,4T_0,\dots$), so the restarts become rare as training goes on; SGDR's CIFAR experiments used schedules such as $T_0=10$ epochs with $T_{mult}=2$.

Each cycle end is a good model, which is what **Snapshot Ensembles** (Huang et al. 2017) exploit: save the weights at the end of every cycle and average the predictions, getting an ensemble for the cost of one training run. **SWA** (Izmailov et al. 2018) averages the weights themselves along a cyclical or constant-rate tail.

Restart steps and the loss at the end of each cycle in the figure (η₀ = 1, η_min = 0, 300-step run)

| T_0, T_mult | cycle ends (restarts) | loss at cycle ends | final loss (last 20 steps) |
| --- | --- | --- | --- |
| 50, 2 | 50, 150 (next would be 350) | 0.130, 0.010 | 0.019 (stopped mid-cycle) |
| 20, 2 | 20, 60, 140, 300 | 0.460, 0.077, 0.011 | 0.012 |
| 100, 1 | 100, 200, 300 | 0.024, 0.012 | 0.018 |
| no restart, T_max = 300 | 300 | — | 0.0076 |

In these runs on a convex problem, restarts never beat a single cycle of the same total length: every restart throws away the noise reduction of the previous cycle. Restarts pay off when you want **intermediate** models (snapshots, early stopping, anytime evaluation), or on non-convex landscapes where re-exploration can find a better basin. That is why the field kept the cosine shape and mostly dropped the restarts.

## Worked example: CosineAnnealingLR(T_max = 300)

1. At $t=75$: $\cos(\pi/4)=0.7071$, so $\eta=\tfrac12(1+0.7071)=0.8536$. With $\eta_{\min}=0.1$: $0.1+0.9\cdot0.8536=0.8682$.
2. At $t=150$: $\cos(\pi/2)=0$, $\eta=0.5$. At $t=225$: $\cos(3\pi/4)=-0.7071$, $\eta=0.1464$.
3. Warm restarts with $T_0=10$, $T_{mult}=2$: cycle lengths 10, 20, 40, 80, so restarts at steps 10, 30, 70 and 150. At step 40, $T_{cur}=10$ of $T_i=40$: $\eta=\tfrac12(1+\cos(\pi/4))\eta_{\max}=0.854\,\eta_{\max}$.
4. To make the last cycle end exactly at the end of training with $T_{mult}=2$, choose $T=T_0(2^k-1)$: e.g. $T_0=20$ gives $20\cdot15=300$.

```python
import torch
from torch.optim.lr_scheduler import (CosineAnnealingLR, CosineAnnealingWarmRestarts,
                                      LinearLR, SequentialLR)

opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
iters = len(loader)
total = epochs * iters

# 1) One cosine cycle over the whole run (after a short warmup), stepped per iteration
warm = 5 * iters
sched = SequentialLR(opt, [
    LinearLR(opt, start_factor=0.01, total_iters=warm),
    CosineAnnealingLR(opt, T_max=total - warm, eta_min=0.0),   # T_max = remaining steps
], milestones=[warm])

# 2) SGDR: T_0 and T_mult are in the unit you pass to step(); epochs here
sgdr = CosineAnnealingWarmRestarts(opt, T_0=10, T_mult=2, eta_min=1e-4)
for epoch in range(epochs):
    for i, (x, y) in enumerate(loader):
        opt.zero_grad()
        loss_fn(model(x), y).backward()
        opt.step()
        sgdr.step(epoch + i / iters)       # fractional epoch: smooth within an epoch
    # cycle ends at epochs 10, 30, 70, 150: good moments to save a snapshot
```

Cosine with warmup, and warm restarts stepped per iteration

> **T_max is a half period, not a stop:** `CosineAnnealingLR` does not stop at $T_{max}$. Keep stepping and the rate follows the cosine back **up**, reaching $\eta_{\max}$ again at $2T_{max}$ (in the figure, $T_{max}=150$ on a 300-step run ends at the peak rate and a 9× worse loss). Set $T_{max}$ to the number of times you will call `step()`, minus the warmup if it is chained after one. Hoffmann et al. (2022) found that a cosine cycle more than about 25% longer than the run noticeably hurts LLM loss, because the rate never gets low.

> **Restarts that do not land on the end:** With $T_{mult}=2$ the cycle boundaries are at $T_0(2^k-1)$. If training stops in the middle of a cycle you get the high-rate model, not the annealed one (the figure's $T_0=50$ run ends at lr 0.15). Choose $T_0$ so a boundary falls on the last step.

> **eta_min:** Decaying all the way to 0 is best for the final training loss here, but many recipes stop at 1–10% of the peak so the last steps still make progress and a later fine-tuning or continued-training phase can start from a non-zero rate. LLM recipes commonly use $\eta_{\min}=0.1\,\eta_{\max}$.

**Strengths**

- One knob (T) instead of milestones and decay factors; smooth, no loss staircases
- Keeps the rate high early and low late: usually at or near the best simple decay
- Warm restarts give anytime snapshots and cheap ensembles

**Limitations**

- Needs the run length in advance; extending training means re-planning the schedule (WSD avoids this)
- Mis-set $T_{max}$ silently gives the wrong shape (rising rate, or never annealing)
- Restarts temporarily raise the loss and, on a fixed budget, rarely beat one cycle

**Question:** You run CosineAnnealingLR(opt, T_max=100) and call scheduler.step() 200 times. What is the learning rate at the end?

- A. Back at the initial lr η_max
- B. eta_min
- C. Zero, the scheduler clamps
- D. Half of η_max

**Answer:** A. Back at the initial lr η_max — The schedule is $\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})(1+\cos(\pi t/T_{max}))$ for all $t$, with no clamp. At $t=200$, $\cos(2\pi)=1$, so the rate is back at $\eta_{\max}$. Set $T_{max}$ to the total number of steps.

**Question:** CosineAnnealingWarmRestarts(T_0=20, T_mult=2) on a 300-step run: where are the restarts?

- A. Steps 20, 60, 140 (and the third cycle ends exactly at 300)
- B. Every 20 steps
- C. Steps 20, 40, 80, 160
- D. Steps 40, 80, 160

**Answer:** A. Steps 20, 60, 140 (and the third cycle ends exactly at 300) — Cycle lengths are 20, 40, 80, 160, so the boundaries are the cumulative sums 20, 60, 140, 300. Because $300=20(2^4-1)$, the last cycle ends exactly at the end of training.

**Key points**

- $\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})(1+\cos(\pi t/T))$: high for long, fastest decay mid-run, gentle landing.
- SGDR (Loshchilov and Hutter 2017) adds warm restarts with cycle lengths $T_0T_{mult}^i$; each cycle end is a usable snapshot.
- `CosineAnnealingLR(T_max, eta_min)` for one cycle (it rises again past $T_{max}$); `CosineAnnealingWarmRestarts(T_0, T_mult, eta_min)` for SGDR, steppable with fractional epochs.
- Warmup + single-cycle cosine to 0–10% of the peak is the default for transformers and LLMs.
- Match the cycle to the run: too short climbs back up, too long never anneals.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
