One-cycle policy

Leslie Smith's schedule: one LR range test picks max_lr, then the rate climbs from max_lr/25 to max_lr and anneals far below the start while momentum moves the opposite way. It trains fast with large rates; in PyTorch it is OneCycleLR, stepped every batch.

12 min read863 wordsadvanced0 viewsupdated 2026-10-11

Definition

The one-cycle policy is a learning-rate schedule made of a single cycle: the rate rises from a low value to a large ηmax⁡\eta_{\max}, then decreases to a value far below where it started, all within one training run. Momentum is cycled inversely: high while the rate is low, low while the rate is high. ηmax⁡\eta_{\max} is chosen with an LR range test.

It comes from three papers by Leslie Smith. Cyclical Learning Rates for Training Neural Networks (WACV 2017, first posted 2015) proposed letting the rate oscillate between bounds and introduced the range test for finding those bounds. Super-Convergence (Smith and Topin, first posted 2017, published 2019) reported that some networks, such as ResNets on CIFAR-10, train an order of magnitude faster with a single cycle that reaches very large rates, and argued that the large rates act as a regularizer. A disciplined approach to neural network hyper-parameters: Part 1 (Smith 2018) turned this into a recipe: one cycle, cyclical momentum from 0.95 to 0.85, and a short final phase that drops the rate well below the starting value. fastai made it the default way to train (fit_one_cycle), and PyTorch ships it as OneCycleLR.

Step 1: the LR range test

Train for a few hundred iterations while increasing the learning rate exponentially, from far too small (say 10−710^{-7}) to far too large (say 1010), and record the loss at each rate. Plotted on a log axis, the loss is flat while the rate is too small to matter, falls while the rate is useful, reaches a minimum, and then blows up once training becomes unstable. One run, no full training needed.

ηk=ηlo(ηhiηlo)k/(K−1),k=0,…,K−1;stop when Lk>4min⁡j≤kLj\eta_k=\eta_{\text{lo}}\Big(\frac{\eta_{\text{hi}}}{\eta_{\text{lo}}}\Big)^{k/(K-1)},\qquad k=0,\dots,K-1;\qquad\text{stop when }L_k>4\min_{j\le k}L_j
KK
number of test iterations (often 100 to a few hundred)
ηlo,ηhi\eta_{\text{lo}},\eta_{\text{hi}}
the sweep range; the exponential spacing gives every decade the same number of steps
4min⁡L4\min L
a common divergence cutoff (fastai uses it, with an exponentially smoothed loss)

Pick ηmax⁡\eta_{\max} a bit below the minimum of the curve, where the loss is still falling clearly: the minimum itself is already close to the edge, because the test reaches it after only a few steps at that rate. Common rules of thumb are "a factor of a few to 10 below the minimum" or "the steepest point of the descent". Smith's own guidance is the largest rate at which the loss still decreases.

Step 2: the cycle

η0=ηmax⁡div_factor,ηend=η0final_div_factor,annealcos⁡(a,b,p)=b+a−b2(1+cos⁡πp)\eta_0=\frac{\eta_{\max}}{\texttt{div\_factor}},\qquad \eta_{\text{end}}=\frac{\eta_0}{\texttt{final\_div\_factor}},\qquad \text{anneal}_{\cos}(a,b,p)=b+\frac{a-b}{2}\big(1+\cos\pi p\big)
pct_start\texttt{pct\_start}
fraction of the steps spent rising (PyTorch default 0.3; phase 1 ends at step pct_start⋅N−1\texttt{pct\_start}\cdot N-1)
div_factor\texttt{div\_factor}
start rate =ηmax⁡/25=\eta_{\max}/25 by default
final_div_factor\texttt{final\_div\_factor}
end rate =η0/104=\eta_0/10^4 by default, i.e. ηmax⁡/(2.5×105)\eta_{\max}/(2.5\times10^5)
pp
progress through the current phase, from 0 to 1

Phase 1 anneals the rate from η0\eta_0 up to ηmax⁡\eta_{\max} and the momentum from max_momentum (0.95) down to base_momentum (0.85). Phase 2 anneals the rate from ηmax⁡\eta_{\max} down to ηend\eta_{\text{end}} and the momentum back up to 0.95. anneal_strategy="cos" (default) or "linear" picks the interpolation. three_phase=True gives Smith's original shape: up, symmetrically back down to η0\eta_0, then a final drop to ηend\eta_{\text{end}}.

Why cycle momentum the other way? With heavy-ball momentum the effective step is about η/(1−μ)\eta/(1-\mu). At the peak, μ=0.85\mu=0.85 makes it 6.7η6.7\eta instead of 20η20\eta at μ=0.95\mu=0.95, which keeps the combined step from overshooting just when the rate is largest. When the rate is small, high momentum helps keep progress going. Smith (2018) found this combination trained faster and more stably than a fixed momentum in his CNN experiments.

LR range test (momentum 0.9)0.010.11101e−31e−21e−1110max_lrlr →OneCycleLR: lr and momentum0.3000.850.950100200training loss, mean of 24 noisy momentum-SGD runs (log)1e-31e-21e-1110050100150200━ one-cycle final 3.94e-3 ━ constant max_lr 0.015 ━ best constant (0.141) 6.15e-3range test: lowest loss at lr 0.57, blew up by 1.57 · lr 0.012 → 0.30 → 1.2e-6
anneal_strategycycle_momentum
fig.Momentum SGD on the noisy quadratic f = ½(0.01x² + y²), gradient noise σ = 0.1, 200 steps, loss averaged over 24 seeds. Top left, the range test (200 steps, lr 10⁻³ → 10, momentum 0.9): the lowest loss is at lr 0.57 and the test stops at 1.57. With max_lr = 0.3 and PyTorch's defaults the cycle runs lr 0.012 → 0.3 (step 59) → 1.2·10⁻⁶ and reaches a final loss of 3.94·10⁻³, against 1.54·10⁻² for a constant 0.3 and 6.15·10⁻³ for the best constant rate on a grid (0.141). Too small a max_lr (0.1) is worse than constant (0.044 vs 0.0081); a large one (1.0) still trains but ends at 0.010; at 3.8 the run diverges because 3.8 > 2(1 + 0.85) = 3.7, the stability limit at the peak's momentum.

The toy is a convex quadratic, so it shows only the optimization side: the high phase travels quickly along the flat direction, the long anneal removes the gradient noise. It cannot show what Smith emphasized for real networks, that the large rates also regularize. It is also honest about momentum cycling: here fixing momentum at 0.9 gives about the same result (3.75×10−33.75\times10^{-3}), so the benefit Smith reports is not a property of quadratics.

Worked example: OneCycleLR(max_lr=0.3, total_steps=200)

stepphase, progress plrmomentumeffective lr/(1−μ)
01, p = 00.01200.9500.24
291, p = 29/590.1520.9011.54
591 → 2, p = 10.3000.8502.00
1002, p = 41/1400.2410.8701.85
1292, p = 70/1400.1500.9001.50
1502, p = 91/1400.08190.9231.06
1992, p = 11.2e−60.9502.4e−5
PyTorch defaults (pct_start 0.3, div_factor 25, final_div_factor 1e4, cos, momentum 0.95 ↔ 0.85); values from the figure's code, which mirrors OneCycleLR.get_lr()
  1. 1Phase 1 ends at step 0.3⋅200−1=590.3\cdot200-1=59. At step 29 the cosine is halfway: 0.3+0.012−0.32(1+cos⁡(π⋅29/59))=0.1520.3+\tfrac{0.012-0.3}{2}(1+\cos(\pi\cdot29/59))=0.152.
  2. 2The phase-2 midpoint is step 59+70=12959+70=129, where the rate is (0.3+1.2×10−6)/2≈0.150(0.3+1.2\times10^{-6})/2\approx0.150 and momentum 0.900.90.
  3. 3The final rate is 0.3/25/104=1.2×10−60.3/25/10^4=1.2\times10^{-6}: 10 000 times below the start, which is what "annihilation" means in Smith's recipe.
  4. 4Linear annealing instead of cosine gives 0.154 at step 29, 0.212 at step 100 and 0.105 at step 150 (it leaves the peak sooner), and a slightly worse final loss here (4.39×10−34.39\times10^{-3}).
max_lrone-cycleconstant at max_lr
0.14.42e−28.11e−3
0.33.94e−31.54e−2
1.01.03e−27.16e−2
3.51.84e−21.04
3.8diverged2.8e3
Final loss (mean of the last 10 steps) versus max_lr, defaults otherwise; the constant run uses the same lr with momentum 0.9
python
1import math, copy, torch
2from torch.optim.lr_scheduler import OneCycleLR
3
4def lr_range_test(model, loader, loss_fn, lo=1e-7, hi=10.0, iters=200, beta=0.98):
5 model = copy.deepcopy(model) # do not touch the real weights
6 opt = torch.optim.SGD(model.parameters(), lr=lo, momentum=0.9)
7 gamma = (hi / lo) ** (1 / (iters - 1))
8 avg, best, out, it = 0.0, math.inf, [], iter(loader)
9 for k in range(iters):
10 try: x, y = next(it)
11 except StopIteration: it = iter(loader); x, y = next(it)
12 lr = lo * gamma ** k
13 for g in opt.param_groups: g["lr"] = lr
14 opt.zero_grad()
15 loss = loss_fn(model(x), y)
16 loss.backward()
17 opt.step()
18 avg = beta * avg + (1 - beta) * loss.item() # smoothed, bias-corrected
19 smooth = avg / (1 - beta ** (k + 1))
20 out.append((lr, smooth))
21 best = min(best, smooth)
22 if smooth > 4 * best: break
23 return out # plot loss vs lr on a log axis
24
25opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
26sched = OneCycleLR(opt, max_lr=0.3, epochs=epochs, steps_per_epoch=len(loader),
27 pct_start=0.3, anneal_strategy="cos", div_factor=25.0,
28 final_div_factor=1e4, cycle_momentum=True,
29 base_momentum=0.85, max_momentum=0.95, three_phase=False)
30for epoch in range(epochs):
31 for x, y in loader:
32 opt.zero_grad()
33 loss_fn(model(x), y).backward()
34 opt.step()
35 sched.step() # every batch; exactly epochs * len(loader) calls
LR range test, then OneCycleLR stepped once per batch

Use when

  • +One tuned number (max_lr, from a cheap range test); good defaults for the rest
  • +Large peak rates train fast; Smith reports super-convergence and a regularizing effect on CNNs
  • +Ends at a tiny rate, so the final noise floor is low

Avoid when

  • −Tied to a fixed number of steps; you cannot simply train longer
  • −The range test is noisy and the choice of max_lr is a judgement call
  • −Super-convergence is architecture and data dependent; momentum cycling matters less with Adam

Check yourself

OneCycleLR(max_lr=0.3, total_steps=200) with defaults. What are the first and last learning rates?

Check yourself

Why does the one-cycle policy lower momentum while the learning rate is at its peak?

Remember

  • ✓LR range test: one short run with exponentially increasing lr; pick max_lr somewhat below the loss minimum.
  • ✓Cycle: ηmax⁡/25→ηmax⁡\eta_{\max}/25\to\eta_{\max} over pct_start (0.3) of the steps, then cosine down to ηmax⁡/(25⋅104)\eta_{\max}/(25\cdot10^4).
  • ✓Momentum moves inversely, 0.95 → 0.85 → 0.95, keeping the effective step η/(1 − μ) in check at the peak.
  • ✓PyTorch: OneCycleLR(opt, max_lr, total_steps | epochs + steps_per_epoch, pct_start, div_factor, final_div_factor, anneal_strategy, three_phase), stepped every batch.
  • ✓Smith (2017, 2018) and Smith and Topin (super-convergence): large rates train faster and regularize; on a convex toy only the speed-and-annealing part shows.
← back to Deep Learning roadmap