Definition
Mixed-precision training computes most of the forward and backward pass in a 16-bit floating-point format (fp16 or bf16) while keeping the parts that need range or precision in fp32: the master copy of the weights, the optimizer update, accumulations inside matmuls and reductions, and numerically sensitive operations such as softmax, normalization and losses.
Modern accelerators are built around low precision. An NVIDIA A100 delivers 19.5 TFLOPS of ordinary fp32 arithmetic, 156 TFLOPS through tensor cores in TF32 mode, and 312 TFLOPS in fp16 or bf16. Halving the bytes per value also halves the memory traffic and the memory taken by saved activations. Training entirely in 16 bits, however, loses small weight updates and small gradients. Mixed precision gets the speed and keeps fp32 exactly where it is needed.
The recipe is from Micikevicius et al. (2018), Mixed Precision Training (NVIDIA and Baidu, ICLR 2018). They trained CNNs, RNNs, speech and translation models and GANs in fp16 to fp32 accuracy with three techniques: an fp32 master copy of the weights, loss scaling to keep gradients out of fp16's underflow zone, and fp32 accumulation of fp16 products. bfloat16, introduced by Google Brain for TPUs, trades mantissa bits for fp32's exponent range, which removes the need for loss scaling; NVIDIA GPUs support it from the A100 (2020) onward. FP8 formats (Micikevicius et al., 2022) followed on H100 GPUs.
Floating-point formats
- exponent and mantissa (fraction) bits; e sets the range, m the precision
- the stored exponent, for normal numbers; encodes zero and subnormals
- machine epsilon: the gap between 1 and the next representable number
- the worst relative rounding error for normal numbers under round-to-nearest
| format | sign / exp / mantissa | max | min normal | min subnormal | ε | role |
|---|---|---|---|---|---|---|
| fp32 | 1 / 8 / 23 | master weights, optimizer state, accumulation | ||||
| tf32 | 1 / 8 / 10 | as fp32 | as fp32 | – | tensor-core mode for fp32 matmuls (inputs rounded to 10 bits) | |
| fp16 | 1 / 5 / 10 | 65504 | compute; needs loss scaling | |||
| bf16 | 1 / 8 / 7 | compute; fp32's range, no loss scaling | ||||
| fp8 E4M3 | 1 / 4 / 3 | 448 | forward activations and weights, with per-tensor scales | |||
| fp8 E5M2 | 1 / 5 / 2 | 57344 | gradients (more range, less precision) |
fp16 and bf16 have the same 16 bits split differently. fp16 has three more mantissa bits, so it is 8 times more precise, but its 5-bit exponent covers only about to . bf16 is fp32 with the lower 16 mantissa bits dropped: the same range, about three significant decimal digits. Deep learning tolerates imprecise values far better than values that overflow to infinity or underflow to zero, which is why bf16 became the default where hardware supports it.
Three things 16 bits break, and the fixes
- 1Small updates vanish. Near 1.0 the fp16 spacing is , so adding anything smaller than half of that () leaves the number unchanged. A weight of 1.0 receiving 10,000 updates of stays exactly 1.0 in fp16 (and in bf16), but reaches 2.0002 in fp32. Fix: keep an fp32 master copy of the weights and apply the optimizer step there; the 16-bit copy used in the forward pass is a cast of it.
- 2Small gradients underflow. The smallest fp16 subnormal is ; values up to half of it () round to zero, and values below are subnormal with fewer significant bits. Activation gradients in deep nets are often that small. Fix: loss scaling (below). bf16 reaches down to and does not need it.
- 3Long sums lose precision and overflow. A dot product of length 4096 summed in fp16 accumulates rounding error, and sums of squares (variance, norms) overflow 65504 easily. Fix: accumulate in fp32. Tensor cores multiply 16-bit inputs and add into fp32 accumulators, and autocast runs reductions, softmax, normalization and losses in fp32.
Loss scaling
- the loss scale, a power of 2 (multiplying by only changes the exponent, so it adds no rounding error)
- the fp16 rounding that every intermediate gradient goes through in backprop
By the chain rule, multiplying the loss by multiplies every gradient in the backward pass by . That shifts the whole gradient histogram up by binades, out of the underflow zone and into fp16's normal range. Before the update the gradients are divided by in fp32, so the optimizer sees the true values. The risk is the other end: if is too large, the largest gradients exceed 65504 and become inf.
Dynamic loss scaling
The best scale is not known in advance and changes during training, so frameworks adjust it automatically. torch.amp.GradScaler starts at and, after every backward pass, checks the unscaled gradients for inf or NaN:
One step of torch.amp.GradScaler (defaults)
- 1Backward on .
scaler.unscale_(explicit, or insidescaler.step) divides every.gradby in place and records whether any value is non-finite. - 2If a non-finite value was found: skip
optimizer.step()for this iteration and set (backoff_factor). - 3Otherwise: take the step, and after
growth_interval=2000consecutive clean steps set (growth_factor).
The scale therefore hovers just below the largest value that does not overflow, and an occasional skipped step is normal and harmless: a few skips in thousands of steps. Many skips in a row, or falling toward 1, means the model itself is producing inf or NaN, which loss scaling cannot fix. In the figure, a short interval of 10 (only for illustration) causes 24 skips in 400 steps; the PyTorch default of 2000 causes 2 and leaves the scale at .
Worked example: one small gradient
| quantity | fp32 | fp16 | fp16 with S = 2¹⁶ | bf16 |
|---|---|---|---|---|
| gradient stored | 0 (below ) | stored, after unscaling | ||
| relative error | 100% | (bound ) | ||
| (lost) | – | (lost) | ||
| after 10,000 updates of | – |
Two lessons. Loss scaling rescues the gradient but does nothing for the weight update, which is why the master weights stay in fp32 even with bf16. And bf16's range comes at the cost of precision: its 8 significant bits make it unsuitable for accumulating weights or optimizer statistics.
What autocast runs in which precision
torch.autocast does not change the dtype of your parameters. Inside the context, each operation casts its inputs to the precision chosen for that operation, on the fly:
| autocast on CUDA | operations (abridged from the PyTorch docs) | why |
|---|---|---|
| to fp16 / bf16 | matmul, linear, conv1d/2d/3d, bmm, addmm, baddbmm, prelu, RNN cells | matrix products: tensor cores, with fp32 accumulation inside |
| to fp32 | softmax, log_softmax, layer_norm, group_norm, exp, log, pow, sum, cumsum, norm, nll_loss, mse_loss, kl_div, binary_cross_entropy_with_logits, … | large dynamic range, long reductions, or catastrophic cancellation |
| widest input | addcmul, addcdiv, atan2, dot, index_put, scatter_add, … | mixed inputs are promoted rather than truncated |
| refused | binary_cross_entropy (on probabilities) | raises an error under autocast; use the _with_logits version |
Element-wise operations not on either list (ReLU, add, multiply) run in whatever dtype their inputs already have, so activations between matmuls stay 16-bit and the saved activations shrink by half.
In PyTorch
1import torch23model = model.cuda() # parameters stay float32 (the master weights)4opt = torch.optim.AdamW(model.parameters(), lr=3e-4)56# ---- fp16: autocast + dynamic loss scaling ----7scaler = torch.amp.GradScaler("cuda") # init_scale=2**16, growth_interval=20008for x, y in loader:9 opt.zero_grad(set_to_none=True)10 with torch.autocast(device_type="cuda", dtype=torch.float16):11 logits = model(x) # linear/conv in fp16, softmax/norms in fp3212 loss = torch.nn.functional.cross_entropy(logits, y) # autocast runs it in fp3213 scaler.scale(loss).backward() # grads of S*loss, stored in fp32 .grad14 scaler.unscale_(opt) # true gradients, needed before clipping15 torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)16 scaler.step(opt) # skipped if any grad is inf/NaN17 scaler.update() # halve S on overflow, double after 2000 clean steps1819# ---- bf16: same range as fp32, so no scaler ----20for x, y in loader:21 opt.zero_grad(set_to_none=True)22 with torch.autocast(device_type="cuda", dtype=torch.bfloat16):23 loss = torch.nn.functional.cross_entropy(model(x), y)24 loss.backward()25 opt.step()2627print(torch.finfo(torch.float16).max, torch.finfo(torch.float16).tiny, torch.finfo(torch.bfloat16).eps)28# 65504.0 6.103515625e-05 0.0078125Use when
- +Much higher matmul throughput: on an A100, 312 TFLOPS in fp16/bf16 vs 156 in TF32 and 19.5 in plain fp32
- +Half the activation memory: larger batches or longer sequences fit
- +A few lines of code with autocast; bf16 needs no loss scaling at all
- +Accuracy matches fp32 when master weights, accumulation and sensitive ops stay in fp32
Avoid when
- −fp16's narrow range: needs dynamic loss scaling and can still overflow in the forward pass
- −bf16's 8-bit mantissa: values have only about 3 significant digits
- −Model-state memory (weights, grads, Adam) is not reduced by plain autocast
- −Custom ops and numerically delicate code must be checked; NaN debugging gets harder
Check yourself
Why does training with bf16 autocast not need a GradScaler, while fp16 does?
Check yourself
GradScaler's scale is 2¹⁶ and the scaled gradients contain an inf on this iteration. What happens?
Remember
- ✓fp16 = 1/5/10 bits (max 65504, ε ≈ 1e−3, underflow below 6e−8); bf16 = 1/8/7 (fp32's range, ε ≈ 8e−3); fp32 = 1/8/23.
- ✓Micikevicius et al. (2018): fp32 master weights, loss scaling, fp32 accumulation. Updates smaller than half an ulp vanish in 16 bits.
- ✓Loss scaling multiplies the loss by , so every gradient moves up binades out of underflow; unscale in fp32 before clipping and the update.
- ✓GradScaler: start 2¹⁶, skip the step and halve on inf/NaN, double after 2000 clean steps. bf16 needs no scaler.
- ✓autocast runs matmuls and convolutions in 16 bits and softmax, norms, reductions and losses in fp32; parameters stay fp32.
- ✓Benefits are speed and activation memory; FP8 (E4M3 forward, E5M2 gradients) pushes further with per-tensor scaling.