Definition
Learning-rate warmup multiplies the learning rate by a factor that grows from (nearly) 0 to 1 over the first steps, , after which the main schedule (constant, cosine, linear or inverse-sqrt decay) takes over. Linear warmup uses .
Warmup is now part of nearly every large training recipe: transformers almost never train well without it, and large-batch CNN training depends on it. It costs a few hundred to a few thousand steps at a reduced rate, and in exchange you can use a peak learning rate that would otherwise diverge in the first iterations.
Three papers made it standard. He et al. (2016) trained their 110-layer CIFAR ResNet at until training error fell below 80% (about 400 iterations) before switching to , a constant warmup. Goyal et al. (2017), training ResNet-50 on ImageNet with batch 8192 in one hour, scaled the learning rate linearly with the batch () and found that a gradual warmup, ramping linearly from to over the first 5 epochs, was needed: constant warmup still produced an error spike at the switch. Vaswani et al. (2017) built warmup into the Transformer's schedule.
The Transformer schedule
- step number, starting at 1
- warmup steps; the paper used
- model width (512 for the base model)
The two branches meet at : linear growth up to there, inverse-square-root decay after it. With and the peak is , reached at step 4000; at step 400 the rate is a tenth of that, , and by step 16000 it has halved to . Modern recipes keep the linear warmup and replace the decay with cosine or WSD.
Why the start is dangerous
Reason 1: the landscape is sharpest at initialization. Gradient descent on a direction with curvature is stable only for . Measurements in real networks (Gilmer et al. 2022; Kalra and Barkeshli 2024) show that the largest Hessian eigenvalue is often large at initialization and falls during early training, especially in transformers and with large learning rates. A peak learning rate that is perfectly stable an hour into training can exceed at step 0. Warmup keeps small while comes down, and it does so without having to know either curve in advance.
Reason 2: Adam's denominator is unreliable at first. Adam divides by , an average of squared gradients over roughly steps when . After only a handful of steps it is an average of a few samples, so its variance is large and some coordinates get a tiny and a huge step. Liu et al. (2020) showed this variance is what warmup mainly compensates for in Adam, and proposed RAdam, which switches the adaptive rate on only once enough samples have accumulated. Ma and Yarats (2021) suggested a rule of thumb of linear warmup over steps (2000 for ).
Reason 3: large batches and big models amplify both. The linear scaling rule multiplies the rate by the batch-size ratio, so a batch of 8192 means the base rate on the same initial landscape. Post-LayerNorm transformers are particularly fragile early: Xiong et al. (2020) traced it to large gradients near the output layers at initialization and showed that Pre-LN transformers can train with little or no warmup.
The toy shows the mechanism, including the catapult: when exceeds , the unstable direction grows, the loss jumps, and the jump pushes the parameters into a flatter region where training continues (Lewkowycz et al. 2020). A short warmup in a real network does the same thing, and if the jump is too violent it never comes back. The longer and gentler the ramp, the smaller the spike.
Worked example: the 40-step linear warmup
| step t | θ = (x, y) | η_t | λmax | η·λmax | loss |
|---|---|---|---|---|---|
| 0 | (2.00, 1.00) | 0.025 | 16.7 | 0.42 | 10.76 |
| 4 | (1.70, 0.016) | 0.125 | 16.5 | 2.06 | 2.05 |
| 8 | (1.03, 0.094) | 0.225 | 15.2 | 3.42 | 1.44 |
| 10 | (0.58, 0.59) | 0.275 | 14.0 | 3.84 | 3.29 |
| 11 | (0.09, −1.55) | 0.300 | 17.6 | 5.29 | 12.3 |
| 12 | (−2.95, 2.95) | 0.325 | 1.30 | 0.42 | 4.68 |
| 40 | (−2.00, 0.00) | 1.000 | 1.29 | 1.29 | 7.9e−7 |
- 1Steps 0–3: , so the steep direction is damped and the loss falls from 10.8 to 2.2 as .
- 2Step 4: . The direction now flips sign and grows each step, by a factor , from to by step 11. The loss climbs from its minimum of 1.44 to 12.3: an 8.5× spike.
- 3Step 11 → 12: the large produces a large -gradient () that throws to , deep in the flat region, where and . That is the catapult.
- 4From there the rate keeps growing to 1.0, which is safe because the limit is now . With the iterate drifts into the flat region before the instability has time to grow much: the worst spike is at most 1.5×, and at there is none.
Shapes and lengths
| warmup | factor w(t), t < W | notes |
|---|---|---|
| constant | (e.g. 0.1), then jumps to 1 | He et al. 2016; needs c below the initial stability limit, and the jump itself can spike (Goyal et al. 2017) |
| linear | the default almost everywhere; PyTorch LinearLR(start_factor, total_iters=W) | |
| exponential (geometric) | from | spends most of the warmup at very small rates; gentle but slow |
| exponential (saturating) | front-loaded: half the peak after ; Ma and Yarats (2021) use | |
| Transformer | as part of | in Vaswani et al. |
Typical lengths: a few hundred to a few thousand steps for transformers (BERT used 10 000, GPT-3 375 million tokens), often quoted as 1–5% of training for LLMs; about 5 epochs for large-batch ImageNet (Goyal et al.); and often none at all for small CNNs with batch norm trained by SGD at moderate rates. If you raise the peak rate or the batch size, lengthen the warmup.
1import math, torch2from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR, LambdaLR34opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)5total, warm = 100_000, 2_00067# Built-in pieces: LinearLR multiplies the base lr by start_factor -> 1.0 over warm steps8sched = SequentialLR(opt, schedulers=[9 LinearLR(opt, start_factor=1e-3, end_factor=1.0, total_iters=warm),10 CosineAnnealingLR(opt, T_max=total - warm, eta_min=3e-5),11], milestones=[warm])1213# The same idea as one lambda (factor of the base lr), plus the Transformer schedule14def warmup_cosine(t):15 if t < warm:16 return (t + 1) / warm17 p = (t - warm) / (total - warm)18 return 0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * p)) # decays to 10% of peak19sched = LambdaLR(opt, warmup_cosine)2021d_model, W = 512, 400022noam = LambdaLR(torch.optim.Adam(model.parameters(), lr=1.0, betas=(0.9, 0.98), eps=1e-9),23 lambda t: d_model ** -0.5 * min((t + 1) ** -0.5, (t + 1) * W ** -1.5))2425for step, (x, y) in enumerate(loader):26 opt.zero_grad()27 loss_fn(model(x), y).backward()28 opt.step()29 sched.step() # once per iteration: warmup is counted in stepsUse when
- +Allows a higher peak learning rate and larger batches without early divergence
- +Cheap: a small fraction of training at a reduced rate
- +Compensates for Adam's noisy second-moment estimate in the first steps
Avoid when
- −One more hyperparameter (length, and a shape) that interacts with the peak rate and batch size
- −Too long wastes steps at a low rate; too short still spikes
- −Hides rather than removes instability caused by poor initialization or normalization
Check yourself
Why does the original Transformer schedule divide by √W·√d_model only at t = W, and what is the learning rate at step 400 with W = 4000, d_model = 512?
Check yourself
In the figure, constant warmup at 10% of the peak works at peak 1.0 but never becomes smooth at peak 1.5. Why?
Remember
- ✓Warmup ramps from near 0 to the peak over steps, then hands over to the decay schedule.
- ✓Stability needs , and is often largest at initialization; warmup keeps below a limit that is still rising.
- ✓Adam's averages few samples early on; warmup (or RAdam) protects against its variance. Rule of thumb: steps.
- ✓Linear warmup is the default; constant warmup needs a known safe rate, and too short a ramp produces a loss spike (catapult) or divergence.
- ✓Transformer: , . Count warmup in optimizer steps.