# One-cycle policy

> Leslie Smith's schedule: one LR range test picks max_lr, then the rate climbs from max_lr/25 to max_lr and anneals far below the start while momentum moves the opposite way. It trains fast with large rates; in PyTorch it is OneCycleLR, stepped every batch.

Source: https://www.mlfoundry.in/learn/one-cycle-policy  
Section: Deep Learning › Optimization Algorithms  
Level: advanced · 12 min read · updated 2026-10-11

**The one-cycle policy:** is a learning-rate schedule made of a single cycle: the rate rises from a low value to a large $\eta_{\max}$, then decreases to a value far below where it started, all within one training run. Momentum is cycled **inversely**: high while the rate is low, low while the rate is high. $\eta_{\max}$ is chosen with an **LR range test**.

It comes from three papers by Leslie Smith. **Cyclical Learning Rates for Training Neural Networks** (WACV 2017, first posted 2015) proposed letting the rate oscillate between bounds and introduced the range test for finding those bounds. **Super-Convergence** (Smith and Topin, first posted 2017, published 2019) reported that some networks, such as ResNets on CIFAR-10, train an order of magnitude faster with a single cycle that reaches very large rates, and argued that the large rates act as a regularizer. **A disciplined approach to neural network hyper-parameters: Part 1** (Smith 2018) turned this into a recipe: one cycle, cyclical momentum from 0.95 to 0.85, and a short final phase that drops the rate well below the starting value. fastai made it the default way to train (`fit_one_cycle`), and PyTorch ships it as `OneCycleLR`.

## Step 1: the LR range test

Train for a few hundred iterations while increasing the learning rate exponentially, from far too small (say $10^{-7}$) to far too large (say $10$), and record the loss at each rate. Plotted on a log axis, the loss is flat while the rate is too small to matter, falls while the rate is useful, reaches a minimum, and then blows up once training becomes unstable. One run, no full training needed.

$$
\eta_k=\eta_{\text{lo}}\Big(\frac{\eta_{\text{hi}}}{\eta_{\text{lo}}}\Big)^{k/(K-1)},\qquad k=0,\dots,K-1;\qquad\text{stop when }L_k>4\min_{j\le k}L_j
$$

- $K$: number of test iterations (often 100 to a few hundred)
- $\eta_{\text{lo}},\eta_{\text{hi}}$: the sweep range; the exponential spacing gives every decade the same number of steps
- $4\min L$: a common divergence cutoff (fastai uses it, with an exponentially smoothed loss)

Pick $\eta_{\max}$ a bit **below** the minimum of the curve, where the loss is still falling clearly: the minimum itself is already close to the edge, because the test reaches it after only a few steps at that rate. Common rules of thumb are "a factor of a few to 10 below the minimum" or "the steepest point of the descent". Smith's own guidance is the largest rate at which the loss still decreases.

## Step 2: the cycle

$$
\eta_0=\frac{\eta_{\max}}{\texttt{div\_factor}},\qquad \eta_{\text{end}}=\frac{\eta_0}{\texttt{final\_div\_factor}},\qquad \text{anneal}_{\cos}(a,b,p)=b+\frac{a-b}{2}\big(1+\cos\pi p\big)
$$

- $\texttt{pct\_start}$: fraction of the steps spent rising (PyTorch default 0.3; phase 1 ends at step $\texttt{pct\_start}\cdot N-1$)
- $\texttt{div\_factor}$: start rate $=\eta_{\max}/25$ by default
- $\texttt{final\_div\_factor}$: end rate $=\eta_0/10^4$ by default, i.e. $\eta_{\max}/(2.5\times10^5)$
- $p$: progress through the current phase, from 0 to 1

Phase 1 anneals the rate from $\eta_0$ **up** to $\eta_{\max}$ and the momentum from `max_momentum` (0.95) down to `base_momentum` (0.85). Phase 2 anneals the rate from $\eta_{\max}$ down to $\eta_{\text{end}}$ and the momentum back up to 0.95. `anneal_strategy="cos"` (default) or `"linear"` picks the interpolation. `three_phase=True` gives Smith's original shape: up, symmetrically back down to $\eta_0$, then a final drop to $\eta_{\text{end}}$.

**Why cycle momentum the other way?** With heavy-ball momentum the effective step is about $\eta/(1-\mu)$. At the peak, $\mu=0.85$ makes it $6.7\eta$ instead of $20\eta$ at $\mu=0.95$, which keeps the combined step from overshooting just when the rate is largest. When the rate is small, high momentum helps keep progress going. Smith (2018) found this combination trained faster and more stably than a fixed momentum in his CNN experiments.

> Figure: Momentum SGD on the noisy quadratic f = ½(0.01x² + y²), gradient noise σ = 0.1, 200 steps, loss averaged over 24 seeds. Top left, the range test (200 steps, lr 10⁻³ → 10, momentum 0.9): the lowest loss is at lr 0.57 and the test stops at 1.57. With max_lr = 0.3 and PyTorch's defaults the cycle runs lr 0.012 → 0.3 (step 59) → 1.2·10⁻⁶ and reaches a final loss of 3.94·10⁻³, against 1.54·10⁻² for a constant 0.3 and 6.15·10⁻³ for the best constant rate on a grid (0.141). Too small a max_lr (0.1) is worse than constant (0.044 vs 0.0081); a large one (1.0) still trains but ends at 0.010; at 3.8 the run diverges because 3.8 > 2(1 + 0.85) = 3.7, the stability limit at the peak's momentum.

The toy is a convex quadratic, so it shows only the optimization side: the high phase travels quickly along the flat direction, the long anneal removes the gradient noise. It cannot show what Smith emphasized for real networks, that the large rates also regularize. It is also honest about momentum cycling: here fixing momentum at 0.9 gives about the same result ($3.75\times10^{-3}$), so the benefit Smith reports is not a property of quadratics.

## Worked example: OneCycleLR(max_lr=0.3, total_steps=200)

PyTorch defaults (pct_start 0.3, div_factor 25, final_div_factor 1e4, cos, momentum 0.95 ↔ 0.85); values from the figure's code, which mirrors OneCycleLR.get_lr()

| step | phase, progress p | lr | momentum | effective lr/(1−μ) |
| --- | --- | --- | --- | --- |
| 0 | 1, p = 0 | 0.0120 | 0.950 | 0.24 |
| 29 | 1, p = 29/59 | 0.152 | 0.901 | 1.54 |
| 59 | 1 → 2, p = 1 | 0.300 | 0.850 | 2.00 |
| 100 | 2, p = 41/140 | 0.241 | 0.870 | 1.85 |
| 129 | 2, p = 70/140 | 0.150 | 0.900 | 1.50 |
| 150 | 2, p = 91/140 | 0.0819 | 0.923 | 1.06 |
| 199 | 2, p = 1 | 1.2e−6 | 0.950 | 2.4e−5 |

1. Phase 1 ends at step $0.3\cdot200-1=59$. At step 29 the cosine is halfway: $0.3+\tfrac{0.012-0.3}{2}(1+\cos(\pi\cdot29/59))=0.152$.
2. The phase-2 midpoint is step $59+70=129$, where the rate is $(0.3+1.2\times10^{-6})/2\approx0.150$ and momentum $0.90$.
3. The final rate is $0.3/25/10^4=1.2\times10^{-6}$: 10 000 times below the start, which is what "annihilation" means in Smith's recipe.
4. Linear annealing instead of cosine gives 0.154 at step 29, 0.212 at step 100 and 0.105 at step 150 (it leaves the peak sooner), and a slightly worse final loss here ($4.39\times10^{-3}$).

Final loss (mean of the last 10 steps) versus max_lr, defaults otherwise; the constant run uses the same lr with momentum 0.9

| max_lr | one-cycle | constant at max_lr |
| --- | --- | --- |
| 0.1 | 4.42e−2 | 8.11e−3 |
| 0.3 | 3.94e−3 | 1.54e−2 |
| 1.0 | 1.03e−2 | 7.16e−2 |
| 3.5 | 1.84e−2 | 1.04 |
| 3.8 | diverged | 2.8e3 |

```python
import math, copy, torch
from torch.optim.lr_scheduler import OneCycleLR

def lr_range_test(model, loader, loss_fn, lo=1e-7, hi=10.0, iters=200, beta=0.98):
    model = copy.deepcopy(model)                         # do not touch the real weights
    opt = torch.optim.SGD(model.parameters(), lr=lo, momentum=0.9)
    gamma = (hi / lo) ** (1 / (iters - 1))
    avg, best, out, it = 0.0, math.inf, [], iter(loader)
    for k in range(iters):
        try: x, y = next(it)
        except StopIteration: it = iter(loader); x, y = next(it)
        lr = lo * gamma ** k
        for g in opt.param_groups: g["lr"] = lr
        opt.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        opt.step()
        avg = beta * avg + (1 - beta) * loss.item()      # smoothed, bias-corrected
        smooth = avg / (1 - beta ** (k + 1))
        out.append((lr, smooth))
        best = min(best, smooth)
        if smooth > 4 * best: break
    return out                                           # plot loss vs lr on a log axis

opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
sched = OneCycleLR(opt, max_lr=0.3, epochs=epochs, steps_per_epoch=len(loader),
                   pct_start=0.3, anneal_strategy="cos", div_factor=25.0,
                   final_div_factor=1e4, cycle_momentum=True,
                   base_momentum=0.85, max_momentum=0.95, three_phase=False)
for epoch in range(epochs):
    for x, y in loader:
        opt.zero_grad()
        loss_fn(model(x), y).backward()
        opt.step()
        sched.step()                 # every batch; exactly epochs * len(loader) calls
```

LR range test, then OneCycleLR stepped once per batch

> **OneCycleLR counts batches and is not resumable by accident:** It needs `total_steps` (or `epochs` and `steps_per_epoch`) and must be stepped once per batch. Stepping per epoch leaves the run stuck in the first few percent of the warm-up; stepping more than `total_steps` times raises an error. With gradient accumulation, pass the number of **optimizer** steps. It also overwrites the optimizer's lr and momentum at construction, so the `lr=` you gave the optimizer is ignored.

> **With Adam:** When the optimizer has `betas` instead of `momentum`, `cycle_momentum=True` cycles $\beta_1$ between `base_momentum` and `max_momentum`. Many Adam users set `cycle_momentum=False` or narrow the range; the original evidence for momentum cycling is from SGD on CNNs.

> **One-cycle is warmup + annealing, chosen together:** Its rising phase is a warmup ($\eta_{\max}/25$ up to $\eta_{\max}$ over 30% of training, far longer than a typical warmup) and its falling phase is a cosine decay to almost zero. What it adds is a procedure: the range test fixes the peak, defaults fix everything else, and the high peak is what buys the speed.

**Strengths**

- One tuned number (max_lr, from a cheap range test); good defaults for the rest
- Large peak rates train fast; Smith reports super-convergence and a regularizing effect on CNNs
- Ends at a tiny rate, so the final noise floor is low

**Limitations**

- Tied to a fixed number of steps; you cannot simply train longer
- The range test is noisy and the choice of max_lr is a judgement call
- Super-convergence is architecture and data dependent; momentum cycling matters less with Adam

**Question:** OneCycleLR(max_lr=0.3, total_steps=200) with defaults. What are the first and last learning rates?

- A. 0.012 and 1.2e−6
- B. 0.03 and 0
- C. 0.3 and 0.012
- D. 0.012 and 0.012

**Answer:** A. 0.012 and 1.2e−6 — The start is $\texttt{max\_lr}/\texttt{div\_factor}=0.3/25=0.012$, and the end is the start divided by `final_div_factor`: $0.012/10^4=1.2\times10^{-6}$.

**Question:** Why does the one-cycle policy lower momentum while the learning rate is at its peak?

- A. The effective step is about η/(1 − μ); lowering μ from 0.95 to 0.85 cuts it by 3× when η is largest
- B. Momentum is not used at large learning rates in PyTorch
- C. To make the range test easier
- D. Lower momentum increases the gradient noise, which helps convergence

**Answer:** A. The effective step is about η/(1 − μ); lowering μ from 0.95 to 0.85 cuts it by 3× when η is largest — $1/(1-0.95)=20$ versus $1/(1-0.85)=6.7$. Lowering momentum at the peak keeps the combined step manageable exactly when the learning rate pushes it up, and raising it again when the rate is small keeps progress going.

**Key points**

- LR range test: one short run with exponentially increasing lr; pick max_lr somewhat below the loss minimum.
- Cycle: $\eta_{\max}/25\to\eta_{\max}$ over `pct_start` (0.3) of the steps, then cosine down to $\eta_{\max}/(25\cdot10^4)$.
- Momentum moves inversely, 0.95 → 0.85 → 0.95, keeping the effective step η/(1 − μ) in check at the peak.
- PyTorch: `OneCycleLR(opt, max_lr, total_steps | epochs + steps_per_epoch, pct_start, div_factor, final_div_factor, anneal_strategy, three_phase)`, stepped every batch.
- Smith (2017, 2018) and Smith and Topin (super-convergence): large rates train faster and regularize; on a convex toy only the speed-and-annealing part shows.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
