Mixed-precision training

Run matmuls and convolutions in 16-bit floats for 2× less activation memory and several times the tensor-core throughput, but keep fp32 master weights, fp32 accumulation and fp32 for sensitive ops. fp16 also needs loss scaling against gradient underflow; bf16 does not.

14 min read1263 wordsadvanced0 viewsupdated 2026-10-11

Definition

Mixed-precision training computes most of the forward and backward pass in a 16-bit floating-point format (fp16 or bf16) while keeping the parts that need range or precision in fp32: the master copy of the weights, the optimizer update, accumulations inside matmuls and reductions, and numerically sensitive operations such as softmax, normalization and losses.

Modern accelerators are built around low precision. An NVIDIA A100 delivers 19.5 TFLOPS of ordinary fp32 arithmetic, 156 TFLOPS through tensor cores in TF32 mode, and 312 TFLOPS in fp16 or bf16. Halving the bytes per value also halves the memory traffic and the memory taken by saved activations. Training entirely in 16 bits, however, loses small weight updates and small gradients. Mixed precision gets the speed and keeps fp32 exactly where it is needed.

The recipe is from Micikevicius et al. (2018), Mixed Precision Training (NVIDIA and Baidu, ICLR 2018). They trained CNNs, RNNs, speech and translation models and GANs in fp16 to fp32 accuracy with three techniques: an fp32 master copy of the weights, loss scaling to keep gradients out of fp16's underflow zone, and fp32 accumulation of fp16 products. bfloat16, introduced by Google Brain for TPUs, trades mantissa bits for fp32's exponent range, which removes the need for loss scaling; NVIDIA GPUs support it from the A100 (2020) onward. FP8 formats (Micikevicius et al., 2022) followed on H100 GPUs.

Floating-point formats

x=(−1)s⋅2 E−bias⋅(1+M2m),bias=2e−1−1,ε=2−m,∣fl⁡(x)−x∣∣x∣≤2−(m+1)x=(-1)^{s}\cdot 2^{\,E-\text{bias}}\cdot\Big(1+\frac{M}{2^{m}}\Big),\qquad \text{bias}=2^{e-1}-1,\qquad \varepsilon=2^{-m},\qquad \frac{|\operatorname{fl}(x)-x|}{|x|}\le 2^{-(m+1)}
e, me,\ m
exponent and mantissa (fraction) bits; e sets the range, m the precision
EE
the stored exponent, 1≤E≤2e−21\le E\le 2^e-2 for normal numbers; E=0E=0 encodes zero and subnormals 21−bias⋅M/2m2^{1-\text{bias}}\cdot M/2^m
ε\varepsilon
machine epsilon: the gap between 1 and the next representable number
2−(m+1)2^{-(m+1)}
the worst relative rounding error for normal numbers under round-to-nearest
formatsign / exp / mantissamaxmin normalmin subnormalεrole
fp321 / 8 / 233.40×10383.40\times10^{38}1.18×10−381.18\times10^{-38}1.4×10−451.4\times10^{-45}1.19×10−71.19\times10^{-7}master weights, optimizer state, accumulation
tf321 / 8 / 10as fp32as fp32–9.77×10−49.77\times10^{-4}tensor-core mode for fp32 matmuls (inputs rounded to 10 bits)
fp161 / 5 / 10655046.10×10−56.10\times10^{-5}5.96×10−85.96\times10^{-8}9.77×10−49.77\times10^{-4}compute; needs loss scaling
bf161 / 8 / 73.39×10383.39\times10^{38}1.18×10−381.18\times10^{-38}9.2×10−419.2\times10^{-41}7.81×10−37.81\times10^{-3}compute; fp32's range, no loss scaling
fp8 E4M31 / 4 / 34482−62^{-6}2−92^{-9}0.1250.125forward activations and weights, with per-tensor scales
fp8 E5M21 / 5 / 2573442−142^{-14}2−162^{-16}0.250.25gradients (more range, less precision)
Formats used in training (values from the bit layouts; fp16, bf16 and fp32 checked against the figure's rounding code)

fp16 and bf16 have the same 16 bits split differently. fp16 has three more mantissa bits, so it is 8 times more precise, but its 5-bit exponent covers only about 2−242^{-24} to 2162^{16}. bf16 is fp32 with the lower 16 mantissa bits dropped: the same range, about three significant decimal digits. Deep learning tolerates imprecise values far better than values that overflow to infinity or underflow to zero, which is why bf16 became the default where hardware supports it.

Three things 16 bits break, and the fixes

  1. 1Small updates vanish. Near 1.0 the fp16 spacing is 2−10≈9.8×10−42^{-10}\approx9.8\times10^{-4}, so adding anything smaller than half of that (2−112^{-11}) leaves the number unchanged. A weight of 1.0 receiving 10,000 updates of 10−410^{-4} stays exactly 1.0 in fp16 (and in bf16), but reaches 2.0002 in fp32. Fix: keep an fp32 master copy of the weights and apply the optimizer step there; the 16-bit copy used in the forward pass is a cast of it.
  2. 2Small gradients underflow. The smallest fp16 subnormal is 2−24≈6×10−82^{-24}\approx6\times10^{-8}; values up to half of it (2−252^{-25}) round to zero, and values below 2−142^{-14} are subnormal with fewer significant bits. Activation gradients in deep nets are often that small. Fix: loss scaling (below). bf16 reaches down to 10−3810^{-38} and does not need it.
  3. 3Long sums lose precision and overflow. A dot product of length 4096 summed in fp16 accumulates rounding error, and sums of squares (variance, norms) overflow 65504 easily. Fix: accumulate in fp32. Tensor cores multiply 16-bit inputs and add into fp32 accumulators, and autocast runs reductions, softmax, normalization and losses in fp32.

Loss scaling

∇θ(S⋅L)=S⋅∇θL⟹g=1S fl⁡16 ⁣(S ∇θL), computed in fp32\nabla_\theta\big(S\cdot L\big)=S\cdot\nabla_\theta L\qquad\Longrightarrow\qquad g=\frac{1}{S}\,\operatorname{fl}_{16}\!\big(S\,\nabla_\theta L\big),\ \text{computed in fp32}
SS
the loss scale, a power of 2 (multiplying by 2k2^k only changes the exponent, so it adds no rounding error)
fl⁡16\operatorname{fl}_{16}
the fp16 rounding that every intermediate gradient goes through in backprop

By the chain rule, multiplying the loss by SS multiplies every gradient in the backward pass by SS. That shifts the whole gradient histogram up by log⁡2S\log_2 S binades, out of the underflow zone and into fp16's normal range. Before the update the gradients are divided by SS in fp32, so the optimizer sees the true values. The risk is the other end: if SS is too large, the largest gradients exceed 65504 and become inf.

fp16: 1 / 5 / 10 bits · max 65504 · min normal 6.1e−5 · min subnormal 6.0e−8 · eps 9.8e−44000 gradient magnitudes |g| ≈ 2^(−27 ± 4) × loss scale 2^0 = 1→ 0subnormalnormal2^−502^−402^−302^−242^−142^02^16flushed to 0: 68.9% · subnormal: 31.1% · normal: 0% · inf: 0%median relative error of the survivors after unscaling: 8.3e−2GradScaler: start 2^16, ×½ and skip on inf, ×2 after 50 clean steps2^102^152^202^25step → 400━ loss scale ─ overflow ceiling 65504 / max|g| │ skipped step · 3 of 400 skipped, ends at 2^19
storage format
fig.Top: a base-2 number line coloured by what fp16 does to a value (flushed to zero below 2⁻²⁵, subnormal up to 2⁻¹⁴, normal up to 65504, then inf) and 4000 illustrative gradient magnitudes centred at 2⁻²⁷. Without scaling, 68.9% of them become 0 and the other 31.1% are subnormal, so no gradient keeps full precision. A loss scale of 2¹⁶ (GradScaler's starting value) loses only 1 of the 4000 and puts 77% in the normal range; 2²⁴ gets 99.7% there. Push past 2³⁰ and the largest values overflow (5 at 2³¹). Switch to bf16: 100% survive with no scaling, but the median rounding error is 1.4·10⁻³, against 1.7·10⁻⁴ for fp16 scaled by 2²⁴. Bottom: dynamic loss scaling over 400 steps of synthetic gradients. With a growth interval of 50 the scale backs off at steps 9, 44 and 394, skips those 3 updates, and otherwise climbs to stay just under the overflow ceiling.

Dynamic loss scaling

The best scale is not known in advance and changes during training, so frameworks adjust it automatically. torch.amp.GradScaler starts at S=216S=2^{16} and, after every backward pass, checks the unscaled gradients for inf or NaN:

One step of torch.amp.GradScaler (defaults)

  1. 1Backward on S⋅LS\cdot L. scaler.unscale_ (explicit, or inside scaler.step) divides every .grad by SS in place and records whether any value is non-finite.
  2. 2If a non-finite value was found: skip optimizer.step() for this iteration and set S←S×0.5S\leftarrow S\times0.5 (backoff_factor).
  3. 3Otherwise: take the step, and after growth_interval=2000 consecutive clean steps set S←S×2S\leftarrow S\times2 (growth_factor).

The scale therefore hovers just below the largest value that does not overflow, and an occasional skipped step is normal and harmless: a few skips in thousands of steps. Many skips in a row, or SS falling toward 1, means the model itself is producing inf or NaN, which loss scaling cannot fix. In the figure, a short interval of 10 (only for illustration) causes 24 skips in 400 steps; the PyTorch default of 2000 causes 2 and leaves the scale at 2142^{14}.

Worked example: one small gradient

quantityfp32fp16fp16 with S = 2¹⁶bf16
gradient 2×10−82\times10^{-8} stored2.0000×10−82.0000\times10^{-8}0 (below 2−252^{-25})1.3103×10−31.3103\times10^{-3} stored, 1.99943×10−81.99943\times10^{-8} after unscaling2.0023×10−82.0023\times10^{-8}
relative error<10−7<10^{-7}100%2.8×10−42.8\times10^{-4}1.2×10−31.2\times10^{-3} (bound 3.9×10−33.9\times10^{-3})
1.0+10−41.0+10^{-4}1.00011.00011.01.0 (lost)–1.01.0 (lost)
1.01.0 after 10,000 updates of 10−410^{-4}2.00022.00021.01.0–1.01.0
A gradient of 2·10⁻⁸ and a weight update of 10⁻⁴ through each format (computed with the figure's IEEE rounding, which matches Math.f16round)

Two lessons. Loss scaling rescues the gradient but does nothing for the weight update, which is why the master weights stay in fp32 even with bf16. And bf16's range comes at the cost of precision: its 8 significant bits make it unsuitable for accumulating weights or optimizer statistics.

What autocast runs in which precision

torch.autocast does not change the dtype of your parameters. Inside the context, each operation casts its inputs to the precision chosen for that operation, on the fly:

autocast on CUDAoperations (abridged from the PyTorch docs)why
to fp16 / bf16matmul, linear, conv1d/2d/3d, bmm, addmm, baddbmm, prelu, RNN cellsmatrix products: tensor cores, with fp32 accumulation inside
to fp32softmax, log_softmax, layer_norm, group_norm, exp, log, pow, sum, cumsum, norm, nll_loss, mse_loss, kl_div, binary_cross_entropy_with_logits, …large dynamic range, long reductions, or catastrophic cancellation
widest inputaddcmul, addcdiv, atan2, dot, index_put, scatter_add, …mixed inputs are promoted rather than truncated
refusedbinary_cross_entropy (on probabilities)raises an error under autocast; use the _with_logits version

Element-wise operations not on either list (ReLU, add, multiply) run in whatever dtype their inputs already have, so activations between matmuls stay 16-bit and the saved activations shrink by half.

In PyTorch

python
1import torch
2
3model = model.cuda() # parameters stay float32 (the master weights)
4opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
5
6# ---- fp16: autocast + dynamic loss scaling ----
7scaler = torch.amp.GradScaler("cuda") # init_scale=2**16, growth_interval=2000
8for x, y in loader:
9 opt.zero_grad(set_to_none=True)
10 with torch.autocast(device_type="cuda", dtype=torch.float16):
11 logits = model(x) # linear/conv in fp16, softmax/norms in fp32
12 loss = torch.nn.functional.cross_entropy(logits, y) # autocast runs it in fp32
13 scaler.scale(loss).backward() # grads of S*loss, stored in fp32 .grad
14 scaler.unscale_(opt) # true gradients, needed before clipping
15 torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
16 scaler.step(opt) # skipped if any grad is inf/NaN
17 scaler.update() # halve S on overflow, double after 2000 clean steps
18
19# ---- bf16: same range as fp32, so no scaler ----
20for x, y in loader:
21 opt.zero_grad(set_to_none=True)
22 with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
23 loss = torch.nn.functional.cross_entropy(model(x), y)
24 loss.backward()
25 opt.step()
26
27print(torch.finfo(torch.float16).max, torch.finfo(torch.float16).tiny, torch.finfo(torch.bfloat16).eps)
28# 65504.0 6.103515625e-05 0.0078125
fp16 needs GradScaler; bf16 does not

Use when

  • +Much higher matmul throughput: on an A100, 312 TFLOPS in fp16/bf16 vs 156 in TF32 and 19.5 in plain fp32
  • +Half the activation memory: larger batches or longer sequences fit
  • +A few lines of code with autocast; bf16 needs no loss scaling at all
  • +Accuracy matches fp32 when master weights, accumulation and sensitive ops stay in fp32

Avoid when

  • −fp16's narrow range: needs dynamic loss scaling and can still overflow in the forward pass
  • −bf16's 8-bit mantissa: values have only about 3 significant digits
  • −Model-state memory (weights, grads, Adam) is not reduced by plain autocast
  • −Custom ops and numerically delicate code must be checked; NaN debugging gets harder

Check yourself

Why does training with bf16 autocast not need a GradScaler, while fp16 does?

Check yourself

GradScaler's scale is 2¹⁶ and the scaled gradients contain an inf on this iteration. What happens?

Remember

  • ✓fp16 = 1/5/10 bits (max 65504, ε ≈ 1e−3, underflow below 6e−8); bf16 = 1/8/7 (fp32's range, ε ≈ 8e−3); fp32 = 1/8/23.
  • ✓Micikevicius et al. (2018): fp32 master weights, loss scaling, fp32 accumulation. Updates smaller than half an ulp vanish in 16 bits.
  • ✓Loss scaling multiplies the loss by S=2kS=2^k, so every gradient moves up kk binades out of underflow; unscale in fp32 before clipping and the update.
  • ✓GradScaler: start 2¹⁶, skip the step and halve on inf/NaN, double after 2000 clean steps. bf16 needs no scaler.
  • ✓autocast runs matmuls and convolutions in 16 bits and softmax, norms, reductions and losses in fp32; parameters stay fp32.
  • ✓Benefits are speed and activation memory; FP8 (E4M3 forward, E5M2 gradients) pushes further with per-tensor scaling.
← back to Deep Learning roadmap