# Warmup scheduling

> Start the learning rate near zero and ramp it to its peak over the first steps. Early on the loss is sharpest and Adam's second-moment estimates are least reliable, so the full rate would blow up; warmup lets training reach a region that tolerates it.

Source: https://www.mlfoundry.in/learn/warmup-scheduling  
Section: Deep Learning › Optimization Algorithms  
Level: core · 11 min read · updated 2026-10-11

**Learning-rate warmup:** multiplies the learning rate by a factor that grows from (nearly) 0 to 1 over the first $W$ steps, $\eta_t=\eta_{\text{peak}}\cdot w(t)$, after which the main schedule (constant, cosine, linear or inverse-sqrt decay) takes over. Linear warmup uses $w(t)=\min(1,\,t/W)$.

Warmup is now part of nearly every large training recipe: transformers almost never train well without it, and large-batch CNN training depends on it. It costs a few hundred to a few thousand steps at a reduced rate, and in exchange you can use a peak learning rate that would otherwise diverge in the first iterations.

Three papers made it standard. He et al. (2016) trained their 110-layer CIFAR ResNet at $\eta=0.01$ until training error fell below 80% (about 400 iterations) before switching to $0.1$, a **constant** warmup. Goyal et al. (2017), training ResNet-50 on ImageNet with batch 8192 in one hour, scaled the learning rate linearly with the batch ($\eta=0.1\cdot B/256=3.2$) and found that a **gradual** warmup, ramping linearly from $0.1$ to $3.2$ over the first 5 epochs, was needed: constant warmup still produced an error spike at the switch. Vaswani et al. (2017) built warmup into the Transformer's schedule.

## The Transformer schedule

$$
\eta_t=d_{\text{model}}^{-0.5}\cdot\min\!\big(t^{-0.5},\;t\cdot W^{-1.5}\big)
$$

- $t$: step number, starting at 1
- $W$: warmup steps; the paper used $W=4000$
- $d_{\text{model}}$: model width (512 for the base model)

The two branches meet at $t=W$: linear growth up to there, inverse-square-root decay after it. With $d_{\text{model}}=512$ and $W=4000$ the peak is $1/\sqrt{512\cdot4000}=6.99\times10^{-4}$, reached at step 4000; at step 400 the rate is a tenth of that, $6.99\times10^{-5}$, and by step 16000 it has halved to $3.49\times10^{-4}$. Modern recipes keep the linear warmup and replace the decay with cosine or WSD.

## Why the start is dangerous

**Reason 1: the landscape is sharpest at initialization.** Gradient descent on a direction with curvature $\lambda$ is stable only for $\eta<2/\lambda$. Measurements in real networks (Gilmer et al. 2022; Kalra and Barkeshli 2024) show that the largest Hessian eigenvalue $\lambda_{\max}$ is often large at initialization and falls during early training, especially in transformers and with large learning rates. A peak learning rate that is perfectly stable an hour into training can exceed $2/\lambda_{\max}$ at step 0. Warmup keeps $\eta_t$ small while $\lambda_{\max}$ comes down, and it does so without having to know either curve in advance.

**Reason 2: Adam's denominator is unreliable at first.** Adam divides by $\sqrt{\hat v_t}$, an average of squared gradients over roughly $1/(1-\beta_2)=1000$ steps when $\beta_2=0.999$. After only a handful of steps it is an average of a few samples, so its variance is large and some coordinates get a tiny $\hat v$ and a huge step. Liu et al. (2020) showed this variance is what warmup mainly compensates for in Adam, and proposed RAdam, which switches the adaptive rate on only once enough samples have accumulated. Ma and Yarats (2021) suggested a rule of thumb of linear warmup over $2/(1-\beta_2)$ steps (2000 for $\beta_2=0.999$).

**Reason 3: large batches and big models amplify both.** The linear scaling rule multiplies the rate by the batch-size ratio, so a batch of 8192 means $32\times$ the base rate on the same initial landscape. Post-LayerNorm transformers are particularly fragile early: Xiong et al. (2020) traced it to large gradients near the output layers at initialization and showed that Pre-LN transformers can train with little or no warmup.

> Figure: Gradient descent on f = ½·0.3(x + 2)² + ½·s(x)·y², where the across-valley curvature s(x) = 1 + 16·sigmoid(2x) is 16.7 at the start (x = 2) and 1.29 at the minimum (x = −2). The top panel compares η (blue) with the stability limit 2/λmax at the current point (red dashes): 0.12 at the start, 1.55 at the minimum. At peak lr 1.0 with no warmup the run diverges at step 2. Linear warmup over 40 steps survives, but η·λmax passes 2 at step 4, the y-oscillation grows and the loss spikes 8.5× (to 12.3 at step 11) before the iterate is thrown into the flat region. From 49 steps on the spike never exceeds 1.5× (at 50 it vanishes entirely). Geometric (exponential) warmup from 1% of the peak needs 21 steps; constant warmup at 10% needs only 12, but only because 0.1 sits below the initial limit 0.12: at peak 1.5 no constant warmup length is smooth.

The toy shows the mechanism, including the **catapult**: when $\eta$ exceeds $2/\lambda_{\max}$, the unstable direction grows, the loss jumps, and the jump pushes the parameters into a flatter region where training continues (Lewkowycz et al. 2020). A short warmup in a real network does the same thing, and if the jump is too violent it never comes back. The longer and gentler the ramp, the smaller the spike.

## Worked example: the 40-step linear warmup

The figure's default run (peak 1.0, $\eta_t=(t+1)/40$ during warmup); $\lambda_{\max}$ is evaluated at the iterate before step $t$. Values from the figure's code.

| step t | θ = (x, y) | η_t | λmax | η·λmax | loss |
| --- | --- | --- | --- | --- | --- |
| 0 | (2.00, 1.00) | 0.025 | 16.7 | 0.42 | 10.76 |
| 4 | (1.70, 0.016) | 0.125 | 16.5 | 2.06 | 2.05 |
| 8 | (1.03, 0.094) | 0.225 | 15.2 | 3.42 | 1.44 |
| 10 | (0.58, 0.59) | 0.275 | 14.0 | 3.84 | 3.29 |
| 11 | (0.09, −1.55) | 0.300 | 17.6 | 5.29 | 12.3 |
| 12 | (−2.95, 2.95) | 0.325 | 1.30 | 0.42 | 4.68 |
| 40 | (−2.00, 0.00) | 1.000 | 1.29 | 1.29 | 7.9e−7 |

1. Steps 0–3: $\eta\lambda_{\max}<2$, so the steep $y$ direction is damped and the loss falls from 10.8 to 2.2 as $y\to0$.
2. Step 4: $\eta\lambda_{\max}=2.06$. The $y$ direction now flips sign and grows each step, by a factor $|1-\eta\lambda|$, from $0.016$ to $1.55$ by step 11. The loss climbs from its minimum of 1.44 to 12.3: an 8.5× spike.
3. Step 11 → 12: the large $y$ produces a large $x$-gradient ($\tfrac12 s'(x)\,y^2$) that throws $x$ to $-2.95$, deep in the flat region, where $\lambda_{\max}=1.30$ and $\eta\lambda_{\max}=0.42$. That is the catapult.
4. From there the rate keeps growing to 1.0, which is safe because the limit is now $2/1.29=1.55$. With $W\ge49$ the iterate drifts into the flat region before the instability has time to grow much: the worst spike is at most 1.5×, and at $W=50$ there is none.

## Shapes and lengths

| warmup | factor w(t), t < W | notes |
| --- | --- | --- |
| constant | $c$ (e.g. 0.1), then jumps to 1 | He et al. 2016; needs c below the initial stability limit, and the jump itself can spike (Goyal et al. 2017) |
| linear | $t/W$ | the default almost everywhere; PyTorch `LinearLR(start_factor, total_iters=W)` |
| exponential (geometric) | $c^{\,1-t/W}$ from $c\ll1$ | spends most of the warmup at very small rates; gentle but slow |
| exponential (saturating) | $1-e^{-t/\tau}$ | front-loaded: half the peak after $0.7\tau$; Ma and Yarats (2021) use $\tau=1/(1-\beta_2)$ |
| Transformer | $t/W$ as part of $\min(t^{-0.5},tW^{-1.5})$ | $W=4000$ in Vaswani et al. |

Typical lengths: a few hundred to a few thousand steps for transformers (BERT used 10 000, GPT-3 375 million tokens), often quoted as 1–5% of training for LLMs; about 5 epochs for large-batch ImageNet (Goyal et al.); and often none at all for small CNNs with batch norm trained by SGD at moderate rates. If you raise the peak rate or the batch size, lengthen the warmup.

```python
import math, torch
from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR, LambdaLR

opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
total, warm = 100_000, 2_000

# Built-in pieces: LinearLR multiplies the base lr by start_factor -> 1.0 over warm steps
sched = SequentialLR(opt, schedulers=[
    LinearLR(opt, start_factor=1e-3, end_factor=1.0, total_iters=warm),
    CosineAnnealingLR(opt, T_max=total - warm, eta_min=3e-5),
], milestones=[warm])

# The same idea as one lambda (factor of the base lr), plus the Transformer schedule
def warmup_cosine(t):
    if t < warm:
        return (t + 1) / warm
    p = (t - warm) / (total - warm)
    return 0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * p))     # decays to 10% of peak
sched = LambdaLR(opt, warmup_cosine)

d_model, W = 512, 4000
noam = LambdaLR(torch.optim.Adam(model.parameters(), lr=1.0, betas=(0.9, 0.98), eps=1e-9),
                lambda t: d_model ** -0.5 * min((t + 1) ** -0.5, (t + 1) * W ** -1.5))

for step, (x, y) in enumerate(loader):
    opt.zero_grad()
    loss_fn(model(x), y).backward()
    opt.step()
    sched.step()          # once per iteration: warmup is counted in steps
```

Linear warmup followed by cosine decay, stepped per iteration

> **Stepping warmup per epoch:** A warmup of 2000 "steps" stepped once per epoch lasts 2000 epochs. Warmup is almost always specified in optimizer steps; with gradient accumulation, count optimizer steps, not micro-batches.

> **Warmup does not fix a learning rate that is too high:** It only gets you safely to the peak. If the peak exceeds $2/\lambda_{\max}$ of the region training settles in, the run still diverges, just later. In the figure, peaks above $2/1.29=1.55$ are unstable at the minimum itself, whatever the warmup.

> **Warmup and weight decay, clipping, normalization:** Gradient clipping, Pre-LN placement, smaller initializations and RAdam-style rectification all attack the same early-training instability, and each can shorten the warmup a run needs. None of them removes the need entirely at large scale, which is why recipes combine them.

**Strengths**

- Allows a higher peak learning rate and larger batches without early divergence
- Cheap: a small fraction of training at a reduced rate
- Compensates for Adam's noisy second-moment estimate in the first steps

**Limitations**

- One more hyperparameter (length, and a shape) that interacts with the peak rate and batch size
- Too long wastes steps at a low rate; too short still spikes
- Hides rather than removes instability caused by poor initialization or normalization

**Question:** Why does the original Transformer schedule divide by √W·√d_model only at t = W, and what is the learning rate at step 400 with W = 4000, d_model = 512?

- A. Both branches of the min meet at t = W; at step 400 the rate is a tenth of the peak, about 7.0e−5
- B. The peak is at t = 1; at step 400 the rate is 7.0e−4
- C. The rate is constant until W; at step 400 it is 7.0e−4
- D. The decay branch is active from the start; at step 400 it is 2.2e−3

**Answer:** A. Both branches of the min meet at t = W; at step 400 the rate is a tenth of the peak, about 7.0e−5 — $t^{-0.5}=tW^{-1.5}$ exactly when $t=W$. Before that, the linear branch $t\,W^{-1.5}d^{-0.5}$ is smaller and wins the min: at $t=400$ it is $400\cdot4000^{-1.5}\cdot512^{-0.5}=6.99\times10^{-5}$, one tenth of the peak $6.99\times10^{-4}$.

**Question:** In the figure, constant warmup at 10% of the peak works at peak 1.0 but never becomes smooth at peak 1.5. Why?

- A. At peak 1.5 the warmup rate 0.15 already exceeds the initial stability limit 2/16.7 ≈ 0.12
- B. Constant warmup only works for SGD with momentum
- C. Peak 1.5 is unstable at the minimum
- D. Constant warmup is too short at 1.5

**Answer:** A. At peak 1.5 the warmup rate 0.15 already exceeds the initial stability limit 2/16.7 ≈ 0.12 — A constant warmup needs its rate below $2/\lambda_{\max}$ at initialization. $0.1\cdot1.0=0.10<0.12$ is stable; $0.1\cdot1.5=0.15>0.12$ is not, so the warmup phase itself spikes. A gradual warmup does not need to know this threshold, which is the practical reason Goyal et al. preferred it.

**Key points**

- Warmup ramps $\eta$ from near 0 to the peak over $W$ steps, then hands over to the decay schedule.
- Stability needs $\eta<2/\lambda_{\max}$, and $\lambda_{\max}$ is often largest at initialization; warmup keeps $\eta$ below a limit that is still rising.
- Adam's $\hat v_t$ averages few samples early on; warmup (or RAdam) protects against its variance. Rule of thumb: $2/(1-\beta_2)$ steps.
- Linear warmup is the default; constant warmup needs a known safe rate, and too short a ramp produces a loss spike (catapult) or divergence.
- Transformer: $\eta=d^{-0.5}\min(t^{-0.5},tW^{-1.5})$, $W=4000$. Count warmup in optimizer steps.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
