Definition
The one-cycle policy is a learning-rate schedule made of a single cycle: the rate rises from a low value to a large , then decreases to a value far below where it started, all within one training run. Momentum is cycled inversely: high while the rate is low, low while the rate is high. is chosen with an LR range test.
It comes from three papers by Leslie Smith. Cyclical Learning Rates for Training Neural Networks (WACV 2017, first posted 2015) proposed letting the rate oscillate between bounds and introduced the range test for finding those bounds. Super-Convergence (Smith and Topin, first posted 2017, published 2019) reported that some networks, such as ResNets on CIFAR-10, train an order of magnitude faster with a single cycle that reaches very large rates, and argued that the large rates act as a regularizer. A disciplined approach to neural network hyper-parameters: Part 1 (Smith 2018) turned this into a recipe: one cycle, cyclical momentum from 0.95 to 0.85, and a short final phase that drops the rate well below the starting value. fastai made it the default way to train (fit_one_cycle), and PyTorch ships it as OneCycleLR.
Step 1: the LR range test
Train for a few hundred iterations while increasing the learning rate exponentially, from far too small (say ) to far too large (say ), and record the loss at each rate. Plotted on a log axis, the loss is flat while the rate is too small to matter, falls while the rate is useful, reaches a minimum, and then blows up once training becomes unstable. One run, no full training needed.
- number of test iterations (often 100 to a few hundred)
- the sweep range; the exponential spacing gives every decade the same number of steps
- a common divergence cutoff (fastai uses it, with an exponentially smoothed loss)
Pick a bit below the minimum of the curve, where the loss is still falling clearly: the minimum itself is already close to the edge, because the test reaches it after only a few steps at that rate. Common rules of thumb are "a factor of a few to 10 below the minimum" or "the steepest point of the descent". Smith's own guidance is the largest rate at which the loss still decreases.
Step 2: the cycle
- fraction of the steps spent rising (PyTorch default 0.3; phase 1 ends at step )
- start rate by default
- end rate by default, i.e.
- progress through the current phase, from 0 to 1
Phase 1 anneals the rate from up to and the momentum from max_momentum (0.95) down to base_momentum (0.85). Phase 2 anneals the rate from down to and the momentum back up to 0.95. anneal_strategy="cos" (default) or "linear" picks the interpolation. three_phase=True gives Smith's original shape: up, symmetrically back down to , then a final drop to .
Why cycle momentum the other way? With heavy-ball momentum the effective step is about . At the peak, makes it instead of at , which keeps the combined step from overshooting just when the rate is largest. When the rate is small, high momentum helps keep progress going. Smith (2018) found this combination trained faster and more stably than a fixed momentum in his CNN experiments.
The toy is a convex quadratic, so it shows only the optimization side: the high phase travels quickly along the flat direction, the long anneal removes the gradient noise. It cannot show what Smith emphasized for real networks, that the large rates also regularize. It is also honest about momentum cycling: here fixing momentum at 0.9 gives about the same result (), so the benefit Smith reports is not a property of quadratics.
Worked example: OneCycleLR(max_lr=0.3, total_steps=200)
| step | phase, progress p | lr | momentum | effective lr/(1−μ) |
|---|---|---|---|---|
| 0 | 1, p = 0 | 0.0120 | 0.950 | 0.24 |
| 29 | 1, p = 29/59 | 0.152 | 0.901 | 1.54 |
| 59 | 1 → 2, p = 1 | 0.300 | 0.850 | 2.00 |
| 100 | 2, p = 41/140 | 0.241 | 0.870 | 1.85 |
| 129 | 2, p = 70/140 | 0.150 | 0.900 | 1.50 |
| 150 | 2, p = 91/140 | 0.0819 | 0.923 | 1.06 |
| 199 | 2, p = 1 | 1.2e−6 | 0.950 | 2.4e−5 |
- 1Phase 1 ends at step . At step 29 the cosine is halfway: .
- 2The phase-2 midpoint is step , where the rate is and momentum .
- 3The final rate is : 10 000 times below the start, which is what "annihilation" means in Smith's recipe.
- 4Linear annealing instead of cosine gives 0.154 at step 29, 0.212 at step 100 and 0.105 at step 150 (it leaves the peak sooner), and a slightly worse final loss here ().
| max_lr | one-cycle | constant at max_lr |
|---|---|---|
| 0.1 | 4.42e−2 | 8.11e−3 |
| 0.3 | 3.94e−3 | 1.54e−2 |
| 1.0 | 1.03e−2 | 7.16e−2 |
| 3.5 | 1.84e−2 | 1.04 |
| 3.8 | diverged | 2.8e3 |
1import math, copy, torch2from torch.optim.lr_scheduler import OneCycleLR34def lr_range_test(model, loader, loss_fn, lo=1e-7, hi=10.0, iters=200, beta=0.98):5 model = copy.deepcopy(model) # do not touch the real weights6 opt = torch.optim.SGD(model.parameters(), lr=lo, momentum=0.9)7 gamma = (hi / lo) ** (1 / (iters - 1))8 avg, best, out, it = 0.0, math.inf, [], iter(loader)9 for k in range(iters):10 try: x, y = next(it)11 except StopIteration: it = iter(loader); x, y = next(it)12 lr = lo * gamma ** k13 for g in opt.param_groups: g["lr"] = lr14 opt.zero_grad()15 loss = loss_fn(model(x), y)16 loss.backward()17 opt.step()18 avg = beta * avg + (1 - beta) * loss.item() # smoothed, bias-corrected19 smooth = avg / (1 - beta ** (k + 1))20 out.append((lr, smooth))21 best = min(best, smooth)22 if smooth > 4 * best: break23 return out # plot loss vs lr on a log axis2425opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)26sched = OneCycleLR(opt, max_lr=0.3, epochs=epochs, steps_per_epoch=len(loader),27 pct_start=0.3, anneal_strategy="cos", div_factor=25.0,28 final_div_factor=1e4, cycle_momentum=True,29 base_momentum=0.85, max_momentum=0.95, three_phase=False)30for epoch in range(epochs):31 for x, y in loader:32 opt.zero_grad()33 loss_fn(model(x), y).backward()34 opt.step()35 sched.step() # every batch; exactly epochs * len(loader) callsUse when
- +One tuned number (max_lr, from a cheap range test); good defaults for the rest
- +Large peak rates train fast; Smith reports super-convergence and a regularizing effect on CNNs
- +Ends at a tiny rate, so the final noise floor is low
Avoid when
- −Tied to a fixed number of steps; you cannot simply train longer
- −The range test is noisy and the choice of max_lr is a judgement call
- −Super-convergence is architecture and data dependent; momentum cycling matters less with Adam
Check yourself
OneCycleLR(max_lr=0.3, total_steps=200) with defaults. What are the first and last learning rates?
Check yourself
Why does the one-cycle policy lower momentum while the learning rate is at its peak?
Remember
- ✓LR range test: one short run with exponentially increasing lr; pick max_lr somewhat below the loss minimum.
- ✓Cycle: over
pct_start(0.3) of the steps, then cosine down to . - ✓Momentum moves inversely, 0.95 → 0.85 → 0.95, keeping the effective step η/(1 − μ) in check at the peak.
- ✓PyTorch:
OneCycleLR(opt, max_lr, total_steps | epochs + steps_per_epoch, pct_start, div_factor, final_div_factor, anneal_strategy, three_phase), stepped every batch. - ✓Smith (2017, 2018) and Smith and Topin (super-convergence): large rates train faster and regularize; on a convex toy only the speed-and-annealing part shows.