Warmup scheduling

Start the learning rate near zero and ramp it to its peak over the first steps. Early on the loss is sharpest and Adam's second-moment estimates are least reliable, so the full rate would blow up; warmup lets training reach a region that tolerates it.

11 min read986 wordscore0 viewsupdated 2026-10-11

Definition

Learning-rate warmup multiplies the learning rate by a factor that grows from (nearly) 0 to 1 over the first WW steps, ηt=ηpeak⋅w(t)\eta_t=\eta_{\text{peak}}\cdot w(t), after which the main schedule (constant, cosine, linear or inverse-sqrt decay) takes over. Linear warmup uses w(t)=min⁡(1, t/W)w(t)=\min(1,\,t/W).

Warmup is now part of nearly every large training recipe: transformers almost never train well without it, and large-batch CNN training depends on it. It costs a few hundred to a few thousand steps at a reduced rate, and in exchange you can use a peak learning rate that would otherwise diverge in the first iterations.

Three papers made it standard. He et al. (2016) trained their 110-layer CIFAR ResNet at η=0.01\eta=0.01 until training error fell below 80% (about 400 iterations) before switching to 0.10.1, a constant warmup. Goyal et al. (2017), training ResNet-50 on ImageNet with batch 8192 in one hour, scaled the learning rate linearly with the batch (η=0.1⋅B/256=3.2\eta=0.1\cdot B/256=3.2) and found that a gradual warmup, ramping linearly from 0.10.1 to 3.23.2 over the first 5 epochs, was needed: constant warmup still produced an error spike at the switch. Vaswani et al. (2017) built warmup into the Transformer's schedule.

The Transformer schedule

ηt=dmodel−0.5⋅min⁡ ⁣(t−0.5,  t⋅W−1.5)\eta_t=d_{\text{model}}^{-0.5}\cdot\min\!\big(t^{-0.5},\;t\cdot W^{-1.5}\big)
tt
step number, starting at 1
WW
warmup steps; the paper used W=4000W=4000
dmodeld_{\text{model}}
model width (512 for the base model)

The two branches meet at t=Wt=W: linear growth up to there, inverse-square-root decay after it. With dmodel=512d_{\text{model}}=512 and W=4000W=4000 the peak is 1/512⋅4000=6.99×10−41/\sqrt{512\cdot4000}=6.99\times10^{-4}, reached at step 4000; at step 400 the rate is a tenth of that, 6.99×10−56.99\times10^{-5}, and by step 16000 it has halved to 3.49×10−43.49\times10^{-4}. Modern recipes keep the linear warmup and replace the decay with cosine or WSD.

Why the start is dangerous

Reason 1: the landscape is sharpest at initialization. Gradient descent on a direction with curvature λ\lambda is stable only for η<2/λ\eta<2/\lambda. Measurements in real networks (Gilmer et al. 2022; Kalra and Barkeshli 2024) show that the largest Hessian eigenvalue λmax⁡\lambda_{\max} is often large at initialization and falls during early training, especially in transformers and with large learning rates. A peak learning rate that is perfectly stable an hour into training can exceed 2/λmax⁡2/\lambda_{\max} at step 0. Warmup keeps ηt\eta_t small while λmax⁡\lambda_{\max} comes down, and it does so without having to know either curve in advance.

Reason 2: Adam's denominator is unreliable at first. Adam divides by v^t\sqrt{\hat v_t}, an average of squared gradients over roughly 1/(1−β2)=10001/(1-\beta_2)=1000 steps when β2=0.999\beta_2=0.999. After only a handful of steps it is an average of a few samples, so its variance is large and some coordinates get a tiny v^\hat v and a huge step. Liu et al. (2020) showed this variance is what warmup mainly compensates for in Adam, and proposed RAdam, which switches the adaptive rate on only once enough samples have accumulated. Ma and Yarats (2021) suggested a rule of thumb of linear warmup over 2/(1−β2)2/(1-\beta_2) steps (2000 for β2=0.999\beta_2=0.999).

Reason 3: large batches and big models amplify both. The linear scaling rule multiplies the rate by the batch-size ratio, so a batch of 8192 means 32×32\times the base rate on the same initial landscape. Post-LayerNorm transformers are particularly fragile early: Xiong et al. (2020) traced it to large gradients near the output layers at initialization and showed that Pre-LN transformers can train with little or no warmup.

η(t) vs stability limit 2/λmax(θₜ), log0.010.11warmup endsloss (log)1e-81e-411e4050100150200sharp at start, flat at the min×x →, across-valley y ↑; × = minimumstep200 / 200η0.0000964λmax1.29η·λmax0.00loss3.0e-30worst spike×8.5smooth fromw ≥ 49━ η ┄ 2/λmax: above it the iterate is unstable ━ loss · GD, warmup then cosine
start: λmax = 16.7, limit 0.120; at the min: limit 1.55
fig.Gradient descent on f = ½·0.3(x + 2)² + ½·s(x)·y², where the across-valley curvature s(x) = 1 + 16·sigmoid(2x) is 16.7 at the start (x = 2) and 1.29 at the minimum (x = −2). The top panel compares η (blue) with the stability limit 2/λmax at the current point (red dashes): 0.12 at the start, 1.55 at the minimum. At peak lr 1.0 with no warmup the run diverges at step 2. Linear warmup over 40 steps survives, but η·λmax passes 2 at step 4, the y-oscillation grows and the loss spikes 8.5× (to 12.3 at step 11) before the iterate is thrown into the flat region. From 49 steps on the spike never exceeds 1.5× (at 50 it vanishes entirely). Geometric (exponential) warmup from 1% of the peak needs 21 steps; constant warmup at 10% needs only 12, but only because 0.1 sits below the initial limit 0.12: at peak 1.5 no constant warmup length is smooth.

The toy shows the mechanism, including the catapult: when η\eta exceeds 2/λmax⁡2/\lambda_{\max}, the unstable direction grows, the loss jumps, and the jump pushes the parameters into a flatter region where training continues (Lewkowycz et al. 2020). A short warmup in a real network does the same thing, and if the jump is too violent it never comes back. The longer and gentler the ramp, the smaller the spike.

Worked example: the 40-step linear warmup

step tθ = (x, y)η_tλmaxη·λmaxloss
0(2.00, 1.00)0.02516.70.4210.76
4(1.70, 0.016)0.12516.52.062.05
8(1.03, 0.094)0.22515.23.421.44
10(0.58, 0.59)0.27514.03.843.29
11(0.09, −1.55)0.30017.65.2912.3
12(−2.95, 2.95)0.3251.300.424.68
40(−2.00, 0.00)1.0001.291.297.9e−7
The figure's default run (peak 1.0, ηt=(t+1)/40\eta_t=(t+1)/40 during warmup); λmax⁡\lambda_{\max} is evaluated at the iterate before step tt. Values from the figure's code.
  1. 1Steps 0–3: ηλmax⁡<2\eta\lambda_{\max}<2, so the steep yy direction is damped and the loss falls from 10.8 to 2.2 as y→0y\to0.
  2. 2Step 4: ηλmax⁡=2.06\eta\lambda_{\max}=2.06. The yy direction now flips sign and grows each step, by a factor ∣1−ηλ∣|1-\eta\lambda|, from 0.0160.016 to 1.551.55 by step 11. The loss climbs from its minimum of 1.44 to 12.3: an 8.5× spike.
  3. 3Step 11 → 12: the large yy produces a large xx-gradient (12s′(x) y2\tfrac12 s'(x)\,y^2) that throws xx to −2.95-2.95, deep in the flat region, where λmax⁡=1.30\lambda_{\max}=1.30 and ηλmax⁡=0.42\eta\lambda_{\max}=0.42. That is the catapult.
  4. 4From there the rate keeps growing to 1.0, which is safe because the limit is now 2/1.29=1.552/1.29=1.55. With W≥49W\ge49 the iterate drifts into the flat region before the instability has time to grow much: the worst spike is at most 1.5×, and at W=50W=50 there is none.

Shapes and lengths

warmupfactor w(t), t < Wnotes
constantcc (e.g. 0.1), then jumps to 1He et al. 2016; needs c below the initial stability limit, and the jump itself can spike (Goyal et al. 2017)
lineart/Wt/Wthe default almost everywhere; PyTorch LinearLR(start_factor, total_iters=W)
exponential (geometric)c 1−t/Wc^{\,1-t/W} from c≪1c\ll1spends most of the warmup at very small rates; gentle but slow
exponential (saturating)1−e−t/τ1-e^{-t/\tau}front-loaded: half the peak after 0.7τ0.7\tau; Ma and Yarats (2021) use τ=1/(1−β2)\tau=1/(1-\beta_2)
Transformert/Wt/W as part of min⁡(t−0.5,tW−1.5)\min(t^{-0.5},tW^{-1.5})W=4000W=4000 in Vaswani et al.

Typical lengths: a few hundred to a few thousand steps for transformers (BERT used 10 000, GPT-3 375 million tokens), often quoted as 1–5% of training for LLMs; about 5 epochs for large-batch ImageNet (Goyal et al.); and often none at all for small CNNs with batch norm trained by SGD at moderate rates. If you raise the peak rate or the batch size, lengthen the warmup.

python
1import math, torch
2from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR, LambdaLR
3
4opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
5total, warm = 100_000, 2_000
6
7# Built-in pieces: LinearLR multiplies the base lr by start_factor -> 1.0 over warm steps
8sched = SequentialLR(opt, schedulers=[
9 LinearLR(opt, start_factor=1e-3, end_factor=1.0, total_iters=warm),
10 CosineAnnealingLR(opt, T_max=total - warm, eta_min=3e-5),
11], milestones=[warm])
12
13# The same idea as one lambda (factor of the base lr), plus the Transformer schedule
14def warmup_cosine(t):
15 if t < warm:
16 return (t + 1) / warm
17 p = (t - warm) / (total - warm)
18 return 0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * p)) # decays to 10% of peak
19sched = LambdaLR(opt, warmup_cosine)
20
21d_model, W = 512, 4000
22noam = LambdaLR(torch.optim.Adam(model.parameters(), lr=1.0, betas=(0.9, 0.98), eps=1e-9),
23 lambda t: d_model ** -0.5 * min((t + 1) ** -0.5, (t + 1) * W ** -1.5))
24
25for step, (x, y) in enumerate(loader):
26 opt.zero_grad()
27 loss_fn(model(x), y).backward()
28 opt.step()
29 sched.step() # once per iteration: warmup is counted in steps
Linear warmup followed by cosine decay, stepped per iteration

Use when

  • +Allows a higher peak learning rate and larger batches without early divergence
  • +Cheap: a small fraction of training at a reduced rate
  • +Compensates for Adam's noisy second-moment estimate in the first steps

Avoid when

  • −One more hyperparameter (length, and a shape) that interacts with the peak rate and batch size
  • −Too long wastes steps at a low rate; too short still spikes
  • −Hides rather than removes instability caused by poor initialization or normalization

Check yourself

Why does the original Transformer schedule divide by √W·√d_model only at t = W, and what is the learning rate at step 400 with W = 4000, d_model = 512?

Check yourself

In the figure, constant warmup at 10% of the peak works at peak 1.0 but never becomes smooth at peak 1.5. Why?

Remember

  • ✓Warmup ramps η\eta from near 0 to the peak over WW steps, then hands over to the decay schedule.
  • ✓Stability needs η<2/λmax⁡\eta<2/\lambda_{\max}, and λmax⁡\lambda_{\max} is often largest at initialization; warmup keeps η\eta below a limit that is still rising.
  • ✓Adam's v^t\hat v_t averages few samples early on; warmup (or RAdam) protects against its variance. Rule of thumb: 2/(1−β2)2/(1-\beta_2) steps.
  • ✓Linear warmup is the default; constant warmup needs a known safe rate, and too short a ramp produces a loss spike (catapult) or divergence.
  • ✓Transformer: η=d−0.5min⁡(t−0.5,tW−1.5)\eta=d^{-0.5}\min(t^{-0.5},tW^{-1.5}), W=4000W=4000. Count warmup in optimizer steps.
← back to Deep Learning roadmap