# Mixed-precision training

> Run matmuls and convolutions in 16-bit floats for 2× less activation memory and several times the tensor-core throughput, but keep fp32 master weights, fp32 accumulation and fp32 for sensitive ops. fp16 also needs loss scaling against gradient underflow; bf16 does not.

Source: https://www.mlfoundry.in/learn/mixed-precision-training  
Section: Deep Learning › Optimization Algorithms  
Level: advanced · 14 min read · updated 2026-10-11

**Mixed-precision training:** computes most of the forward and backward pass in a 16-bit floating-point format (fp16 or bf16) while keeping the parts that need range or precision in fp32: the master copy of the weights, the optimizer update, accumulations inside matmuls and reductions, and numerically sensitive operations such as softmax, normalization and losses.

Modern accelerators are built around low precision. An NVIDIA A100 delivers 19.5 TFLOPS of ordinary fp32 arithmetic, 156 TFLOPS through tensor cores in TF32 mode, and 312 TFLOPS in fp16 or bf16. Halving the bytes per value also halves the memory traffic and the memory taken by saved activations. Training entirely in 16 bits, however, loses small weight updates and small gradients. Mixed precision gets the speed and keeps fp32 exactly where it is needed.

The recipe is from **Micikevicius et al. (2018)**, *Mixed Precision Training* (NVIDIA and Baidu, ICLR 2018). They trained CNNs, RNNs, speech and translation models and GANs in fp16 to fp32 accuracy with three techniques: an **fp32 master copy of the weights**, **loss scaling** to keep gradients out of fp16's underflow zone, and **fp32 accumulation** of fp16 products. **bfloat16**, introduced by Google Brain for TPUs, trades mantissa bits for fp32's exponent range, which removes the need for loss scaling; NVIDIA GPUs support it from the A100 (2020) onward. **FP8** formats (Micikevicius et al., 2022) followed on H100 GPUs.

## Floating-point formats

$$
x=(-1)^{s}\cdot 2^{\,E-\text{bias}}\cdot\Big(1+\frac{M}{2^{m}}\Big),\qquad \text{bias}=2^{e-1}-1,\qquad \varepsilon=2^{-m},\qquad \frac{|\operatorname{fl}(x)-x|}{|x|}\le 2^{-(m+1)}
$$

- $e,\ m$: exponent and mantissa (fraction) bits; e sets the range, m the precision
- $E$: the stored exponent, $1\le E\le 2^e-2$ for normal numbers; $E=0$ encodes zero and **subnormals** $2^{1-\text{bias}}\cdot M/2^m$
- $\varepsilon$: machine epsilon: the gap between 1 and the next representable number
- $2^{-(m+1)}$: the worst relative rounding error for normal numbers under round-to-nearest

Formats used in training (values from the bit layouts; fp16, bf16 and fp32 checked against the figure's rounding code)

| format | sign / exp / mantissa | max | min normal | min subnormal | ε | role |
| --- | --- | --- | --- | --- | --- | --- |
| fp32 | 1 / 8 / 23 | $3.40\times10^{38}$ | $1.18\times10^{-38}$ | $1.4\times10^{-45}$ | $1.19\times10^{-7}$ | master weights, optimizer state, accumulation |
| tf32 | 1 / 8 / 10 | as fp32 | as fp32 | – | $9.77\times10^{-4}$ | tensor-core mode for fp32 matmuls (inputs rounded to 10 bits) |
| fp16 | 1 / 5 / 10 | 65504 | $6.10\times10^{-5}$ | $5.96\times10^{-8}$ | $9.77\times10^{-4}$ | compute; needs loss scaling |
| bf16 | 1 / 8 / 7 | $3.39\times10^{38}$ | $1.18\times10^{-38}$ | $9.2\times10^{-41}$ | $7.81\times10^{-3}$ | compute; fp32's range, no loss scaling |
| fp8 E4M3 | 1 / 4 / 3 | 448 | $2^{-6}$ | $2^{-9}$ | $0.125$ | forward activations and weights, with per-tensor scales |
| fp8 E5M2 | 1 / 5 / 2 | 57344 | $2^{-14}$ | $2^{-16}$ | $0.25$ | gradients (more range, less precision) |

fp16 and bf16 have the same 16 bits split differently. fp16 has three more mantissa bits, so it is 8 times more precise, but its 5-bit exponent covers only about $2^{-24}$ to $2^{16}$. bf16 is fp32 with the lower 16 mantissa bits dropped: the same range, about three significant decimal digits. Deep learning tolerates imprecise values far better than values that overflow to infinity or underflow to zero, which is why bf16 became the default where hardware supports it.

## Three things 16 bits break, and the fixes

1. **Small updates vanish.** Near 1.0 the fp16 spacing is $2^{-10}\approx9.8\times10^{-4}$, so adding anything smaller than half of that ($2^{-11}$) leaves the number unchanged. A weight of 1.0 receiving 10,000 updates of $10^{-4}$ stays exactly 1.0 in fp16 (and in bf16), but reaches 2.0002 in fp32. **Fix: keep an fp32 master copy** of the weights and apply the optimizer step there; the 16-bit copy used in the forward pass is a cast of it.
2. **Small gradients underflow.** The smallest fp16 subnormal is $2^{-24}\approx6\times10^{-8}$; values up to half of it ($2^{-25}$) round to zero, and values below $2^{-14}$ are subnormal with fewer significant bits. Activation gradients in deep nets are often that small. **Fix: loss scaling** (below). bf16 reaches down to $10^{-38}$ and does not need it.
3. **Long sums lose precision and overflow.** A dot product of length 4096 summed in fp16 accumulates rounding error, and sums of squares (variance, norms) overflow 65504 easily. **Fix: accumulate in fp32.** Tensor cores multiply 16-bit inputs and add into fp32 accumulators, and autocast runs reductions, softmax, normalization and losses in fp32.

## Loss scaling

$$
\nabla_\theta\big(S\cdot L\big)=S\cdot\nabla_\theta L\qquad\Longrightarrow\qquad g=\frac{1}{S}\,\operatorname{fl}_{16}\!\big(S\,\nabla_\theta L\big),\ \text{computed in fp32}
$$

- $S$: the loss scale, a power of 2 (multiplying by $2^k$ only changes the exponent, so it adds no rounding error)
- $\operatorname{fl}_{16}$: the fp16 rounding that every intermediate gradient goes through in backprop

By the chain rule, multiplying the loss by $S$ multiplies every gradient in the backward pass by $S$. That shifts the whole gradient histogram up by $\log_2 S$ binades, out of the underflow zone and into fp16's normal range. Before the update the gradients are divided by $S$ in fp32, so the optimizer sees the true values. The risk is the other end: if $S$ is too large, the largest gradients exceed 65504 and become `inf`.

> Figure: Top: a base-2 number line coloured by what fp16 does to a value (flushed to zero below 2⁻²⁵, subnormal up to 2⁻¹⁴, normal up to 65504, then inf) and 4000 illustrative gradient magnitudes centred at 2⁻²⁷. Without scaling, 68.9% of them become 0 and the other 31.1% are subnormal, so no gradient keeps full precision. A loss scale of 2¹⁶ (GradScaler's starting value) loses only 1 of the 4000 and puts 77% in the normal range; 2²⁴ gets 99.7% there. Push past 2³⁰ and the largest values overflow (5 at 2³¹). Switch to bf16: 100% survive with no scaling, but the median rounding error is 1.4·10⁻³, against 1.7·10⁻⁴ for fp16 scaled by 2²⁴. Bottom: dynamic loss scaling over 400 steps of synthetic gradients. With a growth interval of 50 the scale backs off at steps 9, 44 and 394, skips those 3 updates, and otherwise climbs to stay just under the overflow ceiling.

## Dynamic loss scaling

The best scale is not known in advance and changes during training, so frameworks adjust it automatically. `torch.amp.GradScaler` starts at $S=2^{16}$ and, after every backward pass, checks the unscaled gradients for `inf` or `NaN`:

**One step of torch.amp.GradScaler (defaults)**

1. Backward on $S\cdot L$. `scaler.unscale_` (explicit, or inside `scaler.step`) divides every `.grad` by $S$ in place and records whether any value is non-finite.
2. If a non-finite value was found: **skip** `optimizer.step()` for this iteration and set $S\leftarrow S\times0.5$ (`backoff_factor`).
3. Otherwise: take the step, and after `growth_interval=2000` consecutive clean steps set $S\leftarrow S\times2$ (`growth_factor`).

The scale therefore hovers just below the largest value that does not overflow, and an occasional skipped step is normal and harmless: a few skips in thousands of steps. Many skips in a row, or $S$ falling toward 1, means the model itself is producing `inf` or `NaN`, which loss scaling cannot fix. In the figure, a short interval of 10 (only for illustration) causes 24 skips in 400 steps; the PyTorch default of 2000 causes 2 and leaves the scale at $2^{14}$.

## Worked example: one small gradient

A gradient of 2·10⁻⁸ and a weight update of 10⁻⁴ through each format (computed with the figure's IEEE rounding, which matches Math.f16round)

| quantity | fp32 | fp16 | fp16 with S = 2¹⁶ | bf16 |
| --- | --- | --- | --- | --- |
| gradient $2\times10^{-8}$ stored | $2.0000\times10^{-8}$ | **0** (below $2^{-25}$) | $1.3103\times10^{-3}$ stored, $1.99943\times10^{-8}$ after unscaling | $2.0023\times10^{-8}$ |
| relative error | $<10^{-7}$ | 100% | $2.8\times10^{-4}$ | $1.2\times10^{-3}$ (bound $3.9\times10^{-3}$) |
| $1.0+10^{-4}$ | $1.0001$ | $1.0$ (lost) | – | $1.0$ (lost) |
| $1.0$ after 10,000 updates of $10^{-4}$ | $2.0002$ | $1.0$ | – | $1.0$ |

Two lessons. Loss scaling rescues the gradient but does nothing for the weight update, which is why the master weights stay in fp32 even with bf16. And bf16's range comes at the cost of precision: its 8 significant bits make it unsuitable for accumulating weights or optimizer statistics.

## What autocast runs in which precision

`torch.autocast` does not change the dtype of your parameters. Inside the context, each operation casts its inputs to the precision chosen for that operation, on the fly:

| autocast on CUDA | operations (abridged from the PyTorch docs) | why |
| --- | --- | --- |
| to fp16 / bf16 | `matmul`, `linear`, `conv1d/2d/3d`, `bmm`, `addmm`, `baddbmm`, `prelu`, RNN cells | matrix products: tensor cores, with fp32 accumulation inside |
| to fp32 | `softmax`, `log_softmax`, `layer_norm`, `group_norm`, `exp`, `log`, `pow`, `sum`, `cumsum`, `norm`, `nll_loss`, `mse_loss`, `kl_div`, `binary_cross_entropy_with_logits`, … | large dynamic range, long reductions, or catastrophic cancellation |
| widest input | `addcmul`, `addcdiv`, `atan2`, `dot`, `index_put`, `scatter_add`, … | mixed inputs are promoted rather than truncated |
| refused | `binary_cross_entropy` (on probabilities) | raises an error under autocast; use the `_with_logits` version |

Element-wise operations not on either list (ReLU, add, multiply) run in whatever dtype their inputs already have, so activations between matmuls stay 16-bit and the saved activations shrink by half.

## In PyTorch

```python
import torch

model = model.cuda()                      # parameters stay float32 (the master weights)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)

# ---- fp16: autocast + dynamic loss scaling ----
scaler = torch.amp.GradScaler("cuda")     # init_scale=2**16, growth_interval=2000
for x, y in loader:
    opt.zero_grad(set_to_none=True)
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        logits = model(x)                 # linear/conv in fp16, softmax/norms in fp32
        loss = torch.nn.functional.cross_entropy(logits, y)   # autocast runs it in fp32
    scaler.scale(loss).backward()         # grads of S*loss, stored in fp32 .grad
    scaler.unscale_(opt)                  # true gradients, needed before clipping
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    scaler.step(opt)                      # skipped if any grad is inf/NaN
    scaler.update()                       # halve S on overflow, double after 2000 clean steps

# ---- bf16: same range as fp32, so no scaler ----
for x, y in loader:
    opt.zero_grad(set_to_none=True)
    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
        loss = torch.nn.functional.cross_entropy(model(x), y)
    loss.backward()
    opt.step()

print(torch.finfo(torch.float16).max, torch.finfo(torch.float16).tiny, torch.finfo(torch.bfloat16).eps)
# 65504.0 6.103515625e-05 0.0078125
```

fp16 needs GradScaler; bf16 does not

> **model.half() is not mixed precision:** Casting the whole model with `.half()` or `.to(torch.bfloat16)` and training it puts the weights, the update and the optimizer state in 16 bits: small updates disappear (the 10,000 updates above) and softmax and norms run in 16 bits. It is fine for inference. For training, keep fp32 parameters and use autocast, or use a framework that maintains explicit fp32 master weights (DeepSpeed, Megatron, FSDP's `MixedPrecision` policy).

> **Overflow in the forward pass:** Loss scaling handles gradients. Activations can overflow too: a large logit, an unnormalized attention score or a sum of squares above 65504 is `inf` in fp16, and `inf - inf` is `NaN`. Some large transformers pre-trained in bf16 produce NaNs when fine-tuned in fp16 for exactly this reason. Prefer bf16 on hardware that has it; otherwise keep such operations in fp32 (`torch.autocast(..., enabled=False)` around a block, plus `.float()` on its inputs).

> **Where the memory goes:** With Adam and mixed precision, the model state costs about **16 bytes per parameter**: 2 for 16-bit weights, 2 for 16-bit gradients, 4 for the fp32 master weights and 4 each for Adam's $m$ and $v$ (the accounting of the ZeRO paper, Rajbhandari et al., 2020). Plain autocast instead keeps fp32 weights and gradients ($4+4+8=16$ bytes as well). The savings of mixed precision are therefore mostly in **activations**, which dominate at large batch and sequence length, and in **speed**, not in the weights.

> **FP8:** H100-class GPUs add 8-bit tensor cores with twice the 16-bit throughput. The two formats of Micikevicius et al. (2022) split the job: E4M3 (max 448, more precision) for weights and activations in the forward pass, E5M2 (max 57344, more range) for gradients. Eight bits leave too little range for one global loss scale, so each tensor gets its own scaling factor, typically derived from a history of its maximum absolute value. Libraries such as NVIDIA Transformer Engine and PyTorch's `torchao` implement this; master weights and optimizer state remain in higher precision.

**Strengths**

- Much higher matmul throughput: on an A100, 312 TFLOPS in fp16/bf16 vs 156 in TF32 and 19.5 in plain fp32
- Half the activation memory: larger batches or longer sequences fit
- A few lines of code with autocast; bf16 needs no loss scaling at all
- Accuracy matches fp32 when master weights, accumulation and sensitive ops stay in fp32

**Limitations**

- fp16's narrow range: needs dynamic loss scaling and can still overflow in the forward pass
- bf16's 8-bit mantissa: values have only about 3 significant digits
- Model-state memory (weights, grads, Adam) is not reduced by plain autocast
- Custom ops and numerically delicate code must be checked; NaN debugging gets harder

**Question:** Why does training with bf16 autocast not need a GradScaler, while fp16 does?

- A. bf16 has more mantissa bits than fp16
- B. bf16 has fp32's 8-bit exponent, so small gradients do not underflow
- C. bf16 gradients are always stored in fp32
- D. bf16 rounds toward zero instead of to nearest

**Answer:** B. bf16 has fp32's 8-bit exponent, so small gradients do not underflow — Underflow is a range problem. fp16's 5-bit exponent flushes values below about $6\times10^{-8}$ to zero; bf16 shares fp32's exponent and reaches $10^{-38}$. bf16 actually has **fewer** mantissa bits (7 vs 10), so it is less precise than fp16.

**Question:** GradScaler's scale is 2¹⁶ and the scaled gradients contain an inf on this iteration. What happens?

- A. The optimizer steps with the inf values clipped to 65504
- B. The optimizer step is skipped and the scale becomes 2¹⁵
- C. Training stops with an error
- D. The scale doubles to 2¹⁷

**Answer:** B. The optimizer step is skipped and the scale becomes 2¹⁵ — `scaler.step` sees the non-finite gradients and does not call `optimizer.step()`; `scaler.update` multiplies the scale by the backoff factor 0.5. The weights are untouched, so a skipped step costs one batch of compute and nothing else.

**Key points**

- fp16 = 1/5/10 bits (max 65504, ε ≈ 1e−3, underflow below 6e−8); bf16 = 1/8/7 (fp32's range, ε ≈ 8e−3); fp32 = 1/8/23.
- Micikevicius et al. (2018): fp32 master weights, loss scaling, fp32 accumulation. Updates smaller than half an ulp vanish in 16 bits.
- Loss scaling multiplies the loss by $S=2^k$, so every gradient moves up $k$ binades out of underflow; unscale in fp32 before clipping and the update.
- GradScaler: start 2¹⁶, skip the step and halve on inf/NaN, double after 2000 clean steps. bf16 needs no scaler.
- autocast runs matmuls and convolutions in 16 bits and softmax, norms, reductions and losses in fp32; parameters stay fp32.
- Benefits are speed and activation memory; FP8 (E4M3 forward, E5M2 gradients) pushes further with per-tensor scaling.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
