# Distributed LLM training

> Mixed-precision Adam needs 16 bytes per parameter plus activations, so LLMs are trained split across GPUs: ZeRO/FSDP shards states over data-parallel replicas, Megatron tensor parallelism splits matrices inside a node, and pipelines split layers at a bubble cost of (p−1)/(m+p−1).

Source: https://www.mlfoundry.in/learn/distributed-llm-training  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 16 min read · updated 2026-10-13

**Distributed LLM training:** is the practice of training one language model on many accelerators at once by splitting its work along several axes. **Data parallelism** gives each replica different examples and averages gradients. **ZeRO** (Rajbhandari, Rasley, Ruwase and He, SC 2020) and PyTorch **FSDP** (Zhao et al., 2023) shard the optimizer state, gradients and weights across those replicas. **Tensor parallelism** (Shoeybi et al., "Megatron-LM", 2019) splits each weight matrix across the GPUs of a node. **Pipeline parallelism** (GPipe, Huang et al., 2019; PipeDream's 1F1B, Narayanan et al., 2019) places consecutive layers on different devices. **Sequence or context parallelism** splits the tokens of one sequence. Combining data, tensor and pipeline parallelism is called **3D parallelism** (Narayanan et al., SC 2021).

An 80 GB GPU holds the bf16 weights of a 40B-parameter model, but training needs far more than the weights. Adam keeps two moments per parameter, mixed precision keeps an fp32 master copy, and the backward pass needs the activations of every layer. A 70B model needs over 1.1 TB just for its training state. Even when a model fits, one GPU is far too slow: Llama 2 70B took 1,720,320 A100 GPU-hours, which is 196 years on a single device. Distributed training has to solve both problems, memory and time, without letting communication eat the speedup.

## Where the memory goes

$$
M_{\text{states}}=\underbrace{2\Psi}_{\text{bf16 weights}}+\underbrace{2\Psi}_{\text{bf16 grads}}+\underbrace{4\Psi+4\Psi+4\Psi}_{\text{fp32 master},\ m,\ v}=16\,\Psi\ \text{bytes}
$$

- $\Psi$: number of parameters
- $2\Psi+2\Psi$: the 16-bit weights used in forward and backward, and their gradients
- $12\Psi$: the optimizer state, $K=12$ in the ZeRO paper: an fp32 master copy (updates smaller than bf16's resolution would otherwise be lost) and Adam's first and second moments

$$
M_{\text{act}}\approx s\,b\,h\Big(34+\frac{5\,a\,s}{h}\Big)\ \text{bytes per layer},\qquad \xrightarrow{\ \text{TP }t\ +\ \text{seq. parallel}\ }\ s\,b\,h\Big(\frac{34}{t}+\frac{5\,a\,s}{h\,t}\Big)
$$

- $s,b,h,a$: sequence length, micro-batch size, hidden size, attention heads (Korthikanti et al., 2022, for a GPT block with 16-bit activations)
- $34\,sbh$: inputs to the linear layers, GeLU, LayerNorms and dropout masks
- $5as^2b$: the $a\times s\times s$ attention scores, softmax output and dropout mask: quadratic in $s$
- $\text{selective}$: recomputing only the attention part (or using FlashAttention) removes the $5as/h$ term, leaving $34\,sbh/t$
- $\text{full}$: full activation checkpointing keeps only each layer's input, $2sbh$, and recomputes the rest in the backward pass

For Llama 2 70B ($\Psi=6.9\times10^{10}$), $16\Psi$ is 1,104 GB. At $s=4096$, $b=1$, $h=8192$ and $a=64$, one layer's activations are $sbh(34+160)=6.5$ GB, and there are 80 layers. Most of that is the attention-score term, which is why FlashAttention matters for training memory as well as for speed. Both kinds of memory have to be divided across GPUs.

## Data parallelism and ZeRO

Plain **data parallelism** copies the whole model to each of $d$ GPUs, gives each a different slice of the global batch, and averages gradients with an all-reduce before every optimizer step. It scales throughput but saves no memory: every replica holds all $16\Psi$ bytes. ZeRO observes that this redundancy is unnecessary. Each replica only needs to *update* a $1/d$ slice of the parameters, so it only needs that slice's optimizer state.

$$
\text{ZeRO-1: } 4\Psi+\frac{12\Psi}{d}\qquad \text{ZeRO-2: } 2\Psi+\frac{14\Psi}{d}\qquad \text{ZeRO-3 / FSDP: } \frac{16\Psi}{d}
$$

- $\text{ZeRO-1}$: shard the optimizer state ($P_{os}$); after the update, all-gather the new weights
- $\text{ZeRO-2}$: also shard gradients ($P_{os+g}$): reduce-scatter instead of all-reduce, so each rank receives only its slice
- $\text{ZeRO-3}$: also shard weights ($P_{os+g+p}$): each layer's weights are all-gathered just before its forward and backward, then freed. PyTorch FSDP's FULL_SHARD is the same idea

An all-reduce is a reduce-scatter followed by an all-gather, so stages 1 and 2 send the same volume as plain data parallelism: about $2\Psi$ elements per step for each GPU. Stage 3 adds one more all-gather of the weights for the backward pass, 1.5 times the traffic. The paper's example, a 7.5B model on 64 GPUs, needs 120 GB per GPU with plain data parallelism, 31.4 GB with ZeRO-1, 16.6 GB with ZeRO-2 and 1.9 GB with ZeRO-3. ZeRO-Offload and ZeRO-Infinity go further and move optimizer state to CPU memory or NVMe.

> Figure: Left: Megatron's tensor-parallel MLP on two GPUs. The first weight matrix is split by columns, so each GPU computes its own half of the hidden layer and applies GeLU locally. The second is split by rows, so each GPU produces a partial output, and one all-reduce (g) sums them. In the backward pass the conjugate operator f all-reduces the input gradient. Right: 3D parallelism on 64 GPUs. Each node of 8 GPUs runs one tensor-parallel group over NVLink, 4 nodes form a pipeline that passes activations forward and gradients back, and 2 such pipelines are data-parallel replicas that all-reduce gradients once per step. Bottom: model-state memory per GPU for the ZeRO paper's 7.5B model on 64 GPUs: 120 GB, 31.4 GB, 16.6 GB and 1.88 GB. Hover a legend line to highlight that axis.

## Tensor parallelism (Megatron)

$$
Y=\mathrm{GeLU}(XA)\,B,\quad A=[A_1\,|\,A_2],\ B=\begin{bmatrix}B_1\\ B_2\end{bmatrix}\ \Rightarrow\ Y=\mathrm{GeLU}(XA_1)B_1+\mathrm{GeLU}(XA_2)B_2
$$

- $[A_1|A_2]$: column-parallel: GPU $i$ computes $XA_i$, a slice of the hidden units, so the elementwise GeLU needs no communication
- $[B_1;B_2]$: row-parallel: GPU $i$ multiplies its hidden slice by its rows of $B$ and gets a partial sum of $Y$
- $+$: one all-reduce of an $s\times b\times h$ tensor after the MLP, and one after attention, where the split is by heads. Two more come in the backward pass

Each GPU in a group of $t$ holds $1/t$ of every attention and MLP matrix, so weights, gradients, optimizer state and most activations shrink by $t$. The cost is four all-reduces of the full activation tensor per layer and micro-batch. Those are on the critical path, so tensor parallelism is kept inside a node, where NVLink is roughly an order of magnitude faster than the network between nodes. $t=8$ (one node) is the usual ceiling. Shoeybi et al. trained an 8.3B model on 512 V100 GPUs at 15.1 PFLOPS with 76% scaling efficiency. **Sequence parallelism** (Korthikanti et al., 2022) splits the LayerNorm and dropout regions, which tensor parallelism leaves replicated, along the sequence. It turns each all-reduce into a reduce-scatter plus an all-gather with the same total volume, and divides the remaining $10\,sbh$ of activations by $t$ as well.

## Pipeline parallelism

Pipeline parallelism splits the $L$ layers into $p$ stages on different devices. Only the activations at stage boundaries cross the network, point-to-point, so it suits slow links between nodes. The problem is idle time. With one batch, stage 2 waits for stage 1, and stage 1 waits for stage 2's backward. GPipe splits each batch into $m$ micro-batches and pipelines them, accumulating gradients until all have finished.

$$
\text{bubble fraction}=\frac{(p-1)(t_f+t_b)}{(m+p-1)(t_f+t_b)}=\frac{p-1}{m+p-1},\qquad \text{interleaved with } v \text{ chunks per device: } \frac{t_{\text{bubble}}}{t_{\text{ideal}}}=\frac{p-1}{v\,m}
$$

- $p,\ m$: pipeline stages and micro-batches per optimizer step
- $p-1$: slots each stage spends waiting at the start (fill) and end (drain)
- $v$: Narayanan et al. (2021) give each device $v$ non-adjacent chunks of layers, shrinking the bubble $v$-fold for $v$ times more point-to-point messages

GPipe runs all forwards and then all backwards, so the first stage stores activations for all $m$ micro-batches. **1F1B** (PipeDream) starts each micro-batch's backward as soon as possible and then alternates one forward with one backward. The bubble is identical, but a stage never holds more than $p$ micro-batches in flight, so $m$ can grow to shrink the bubble without growing memory. GPipe reported the bubble to be negligible once $m\ge4p$. The first stage still holds $p$ micro-batches of $L/p$ layers, which is $L$ layers' worth of activations however deep the pipeline. Pipelines therefore cut weight memory but not activation memory on the first stage.

## Sequence and context parallelism

At 128K tokens even $34\,sbh/t$ is large for a single sequence, and the batch cannot be split further. **Context parallelism** shards the sequence dimension of every layer across $c$ GPUs. Attention then needs all keys and values: ring attention (Liu, Zaharia and Abbeel, 2023) passes K/V blocks around the ring while computing blockwise attention, and other implementations all-gather the keys and values. Llama 3 405B's 128K-context stage used 16-way context parallelism. Causal masks unbalance the work, so each GPU takes one chunk from the start of the sequence and one from the end (see **Context window optimization**). Mixture-of-experts models add **expert parallelism**, placing different experts on different GPUs with all-to-all token routing (see **Mixture of experts (MoE)**).

**Published configurations**

- **Megatron-LM (2019):** 8.3B GPT-2-style model, 8-way tensor parallel, 512 V100s, 15.1 PFLOPS sustained, 76% weak-scaling efficiency
- **ZeRO (2020):** 7.5B on 64 GPUs: 120 → 31.4 → 16.6 → 1.9 GB per GPU of model states; used to train the 17B Turing-NLG
- **Megatron PTD-P (2021):** 3D parallelism with interleaved 1F1B: a 1T-parameter model at 502 PFLOP/s on 3072 A100s, 52% of peak per GPU
- **Korthikanti et al. (2022):** sequence parallelism + selective recomputation: about 5× less activation memory, and over 90% less recomputation overhead than full checkpointing
- **Llama 2 70B (2023):** 2T tokens, 1,720,320 A100-80GB GPU-hours; $6ND=8.3\times10^{23}$ FLOPs implies about 43% of the 312 TFLOPS peak
- **Llama 3 405B (2024):** 16,384 H100s; 8-way tensor × 16-way pipeline × 128-way data parallel at 8K sequence length, 38–43% bf16 MFU; context parallelism for the 128K stage
- **Interconnect:** H100 NVLink about 900 GB/s per GPU inside a node; 400 Gb/s (50 GB/s) InfiniBand per GPU between nodes

> Figure: Top: per-GPU memory for the chosen model and parallel layout, split into bf16 weights, bf16 gradients, fp32 optimizer state and activations, against an 80 GB GPU. At the defaults, Llama 2 70B on 64 GPUs (d 2 × t 8 × p 4) with ZeRO-1, selective recomputation, sequence parallelism, 4096-token sequences, micro-batch 1 and m = 8, each model shard holds 2.16B parameters. Per GPU that is 4.31 + 4.31 + 12.9 GB of states plus 11.4 GB of activations (143 MB per layer × 20 layers × 4 micro-batches in flight), 33.0 GB in total. Each GPU sends 4.31 GB of data-parallel and 75.2 GB of tensor-parallel traffic per step. With no recomputation the activations grow to 65.1 GB and the total to 86.7 GB, which does not fit. Without sequence parallelism they are 34.9 GB, and with full recomputation 1.48 GB. Switching to GPipe holds all 8 micro-batches: 22.8 GB of activations. Bottom: the simulated schedule, with forward 1 unit and backward 2. p = 4 and m = 8 take 33 units instead of 24, a 27.3% bubble, matching (p − 1)/(m + p − 1). m = 32 cuts it to 8.6% with no extra memory under 1F1B, while GPipe would need 91.3 GB of activations. Llama 2 7B on one GPU with no sharding and no recomputation needs 212 GB. ZeRO-3 over 8 GPUs brings it to 31.7 GB. Llama 2 70B with ZeRO-3 over 64 GPUs and full recomputation needs 29.1 GB per GPU but sends 407 GB per GPU per step.

## Mixed precision and activation checkpointing

Micikevicius et al. ("Mixed Precision Training", ICLR 2018) ran forward and backward in fp16 while keeping fp32 master weights for the update, and scaled the loss so that small gradients would not underflow. bf16 has fp32's 8-bit exponent, so it needs no loss scaling and is now the default for LLMs. Matrix multiplies run in bf16 with fp32 accumulation, while softmax, norms and the optimizer stay in fp32. FP8 matrix multiplies with per-tensor or finer-grained scales are the next step: DeepSeek-V3 was trained largely in FP8.

**Activation checkpointing** (Chen et al., "Training Deep Nets with Sublinear Memory Cost", 2016) stores only some activations and recomputes the rest during the backward pass. Checkpointing every layer's input costs one extra forward pass, about 33% more compute, since a forward is roughly a third of forward plus backward. **Selective recomputation** recomputes only the attention scores, which are large in memory and cheap in FLOPs. FlashAttention already behaves this way because it never stores the score matrix.

## Communication costs

$$
\text{ring all-reduce sends } \frac{2(n-1)}{n}\,|x|\ \text{per GPU},\qquad \text{DP: } |x|=2\Psi'\ \text{bytes/step},\qquad \text{TP: } 4\cdot\tfrac{L}{p}\cdot m\cdot 2sbh\ \text{bytes/step},\qquad C\approx6ND
$$

- $\Psi'=\Psi/(tp)$: parameters in one model shard; data-parallel gradients are reduced once per step and can overlap with the backward pass
- $4\cdot\frac Lp\cdot m$: four tensor-parallel all-reduces per layer per micro-batch, on the critical path
- $C\approx6ND$: training FLOPs for $N$ parameters and $D$ tokens; wall time is $C/(\#\text{GPUs}\cdot\text{peak}\cdot\text{MFU})$

The layout follows the bandwidth hierarchy. The chattiest axis, tensor parallelism, stays on NVLink. Pipeline parallelism sends only boundary activations and crosses nodes. Data parallelism sends the most bytes, but only once per step, and overlaps them with the backward pass, so it takes the outermost and slowest links. Model FLOPs utilisation (MFU) of 40–55% on thousands of GPUs is considered good. The rest goes to communication that cannot be hidden, pipeline bubbles, recomputation and stragglers.

## Worked example: Llama 2 70B on 64 GPUs

$\Psi=68.98\times10^9$, $L=80$, $h=8192$, $a=64$, sequence $s=4096$, micro-batch $b=1$, $m=8$ micro-batches per step, 80 GB GPUs in nodes of 8.

Per-GPU memory under successive choices (computed with the Python below)

| layout | model states | activations | total |
| --- | --- | --- | --- |
| plain DP, 64 replicas | 16Ψ = 1,103.6 GB | — | does not fit |
| ZeRO-3 over 64 | 16Ψ/64 = 17.2 GB | per layer 6.5 GB without recompute | does not fit |
| t 8 × p 4 × d 2, ZeRO-1, no recompute | 4Ψ′ + 12Ψ′/2 = 21.6 GB | 813.7 MB × 20 × 4 = 65.1 GB | 86.7 GB |
| same, selective, no seq. parallel | 21.6 GB | 436.2 MB × 80 = 34.9 GB | 56.5 GB |
| same, selective + seq. parallel | 21.6 GB | 142.6 MB × 80 = 11.4 GB | 33.0 GB |
| same, full recompute | 21.6 GB | 8.4 MB × 80 + 0.8 GB = 1.5 GB | 23.0 GB |

**Working it through**

1. Shard the model: $\Psi'=\Psi/(t\,p)=68.98\text{B}/32=2.156$B parameters per GPU. ZeRO-1 over $d=2$: $4\cdot2.156+12\cdot2.156/2=8.62+12.93=21.6$ GB.
2. Activations per layer with $t=8$ and sequence parallelism: $sbh=4096\cdot8192=3.36\times10^7$. No recompute: $sbh(34/8+5\cdot64\cdot4096/(8192\cdot8))=sbh(4.25+20)=813.7$ MB. The attention term is 82% of it. Selective recompute: $sbh\cdot4.25=142.6$ MB.
3. The first pipeline stage holds $80/4=20$ layers for $\min(p,m)=4$ micro-batches under 1F1B: $142.6\text{ MB}\times80=11.4$ GB. Total $21.6+11.4=33.0$ GB, which leaves room for buffers and fragmentation.
4. Bubble: $(4-1)/(8+4-1)=27.3\%$. With $m=32$ it is $3/35=8.6\%$ at the same memory, but the global batch grows to $d\cdot m\cdot b\cdot s=2\cdot32\cdot4096=262{,}144$ tokens.
5. Traffic per GPU per step at $m=8$: data parallel $2\cdot\tfrac12\cdot2\Psi'=4.31$ GB once, overlapped with the backward pass. Tensor parallel $4\cdot20\cdot8\cdot\tfrac74\cdot2sbh=75.2$ GB, on NVLink, on the critical path.
6. Time: one step processes $2\cdot8\cdot4096=65{,}536$ tokens, $6\Psi\cdot65{,}536=2.71\times10^{16}$ FLOPs. At 50% of 312 TFLOPS on 64 GPUs that is 2.7 s per step. Llama 2 70B's 2T tokens need $8.3\times10^{23}$ FLOPs, and its reported 1,720,320 GPU-hours imply 43% MFU.

```python
import math

def model_states(psi, d, zero):
    """bf16 weights + bf16 grads + fp32 (master, m, v) = 2 + 2 + 12 bytes/param; ZeRO shards over d."""
    w = 2 * psi / (d if zero >= 3 else 1)
    g = 2 * psi / (d if zero >= 2 else 1)
    o = 12 * psi / (d if zero >= 1 else 1)
    return w + g + o

def act_per_layer(s, b, h, a, t=1, recompute="selective", sp=True):
    """Korthikanti et al. 2022, bytes per transformer layer with 16-bit activations."""
    sp = sp and t > 1
    if recompute == "full":
        return 2 * s * b * h / (t if sp else 1)
    lin = 34 / t if sp else 10 + 24 / t
    attn = 5 * a * s / (h * t) if recompute == "none" else 0.0      # the softmax/dropout part
    return s * b * h * (lin + attn)

def per_gpu_gb(params, L, h, a, s, b, t, p, d, zero, recompute, sp, m):
    states = model_states(params / (t * p), d, zero)
    inflight = min(p, m)                                          # 1F1B: first stage holds p micro-batches
    act = act_per_layer(s, b, h, a, t, recompute, sp) * math.ceil(L / p) * inflight
    if recompute == "full":
        act += act_per_layer(s, b, h, a, t, "none", sp)           # one layer re-materialised at a time
    return states / 1e9, act / 1e9

def simulate(p, m, schedule="1f1b", tf=1, tb=2):
    """Run a pipeline schedule with real dependencies; return (makespan, idle fraction)."""
    def order(k):
        if schedule == "gpipe":
            return [("F", i) for i in range(m)] + [("B", i) for i in range(m)]
        warm = min(p - k - 1, m)                                  # 1F1B: warm-up forwards, then alternate
        ops, f, bk = [("F", i) for i in range(warm)], warm, 0
        while f < m:
            ops += [("F", f), ("B", bk)]; f += 1; bk += 1
        return ops + [("B", i) for i in range(bk, m)]
    lists, ptr, free, done = [order(k) for k in range(p)], [0] * p, [0] * p, {}
    while any(ptr[k] < len(lists[k]) for k in range(p)):
        for k in range(p):
            while ptr[k] < len(lists[k]):
                kind, i = lists[k][ptr[k]]
                dep = (0 if k == 0 else done.get(("F", k - 1, i))) if kind == "F" else \
                      (done.get(("F", k, i)) if k == p - 1 else done.get(("B", k + 1, i)))
                if dep is None:
                    break
                free[k] = max(free[k], dep) + (tf if kind == "F" else tb)
                done[(kind, k, i)] = free[k]; ptr[k] += 1
    span = max(free)
    return span, 1 - p * m * (tf + tb) / (p * span)

# Llama 2 70B on 64 GPUs: t = 8 (in node), p = 4, d = 2, ZeRO-1, sequence 4096, micro-batch 1, m = 8
for rc, sp in (("none", True), ("selective", False), ("selective", True), ("full", True)):
    st, act = per_gpu_gb(68.976e9, 80, 8192, 64, 4096, 1, 8, 4, 2, 1, rc, sp, 8)
    print(f"{rc:9s} SP={sp!s:5s}  states {st:5.2f} GB  activations {act:5.2f} GB  total {st + act:5.2f} GB")
print("plain DP:", 16 * 68.976e9 / 1e9, "GB per GPU;  ZeRO-3 over 64:", round(16 * 68.976e9 / 64 / 1e9, 2), "GB")
for m in (8, 32):
    span, idle = simulate(4, m)
    print(f"p=4 m={m}: makespan {span} units, idle {idle:.3f}, formula {(4 - 1) / (m + 4 - 1):.3f}")
# none      SP=True   states 21.55 GB  activations 65.10 GB  total 86.65 GB
# selective SP=False  states 21.55 GB  activations 34.90 GB  total 56.45 GB
# selective SP=True   states 21.55 GB  activations 11.41 GB  total 32.96 GB
# full      SP=True   states 21.55 GB  activations  1.48 GB  total 23.04 GB
# plain DP: 1103.616 GB per GPU;  ZeRO-3 over 64: 17.24 GB
# p=4 m=8: makespan 33 units, idle 0.273, formula 0.273
# p=4 m=32: makespan 105 units, idle 0.086, formula 0.086
```

Per-GPU memory and pipeline-bubble calculator for any 3D layout (pure Python, run; printed output shown)

```python
import functools, os, torch, torch.distributed as dist
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision, ShardingStrategy
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (
    CheckpointImpl, apply_activation_checkpointing, checkpoint_wrapper)
from transformers import AutoConfig, AutoModelForCausalLM
from transformers.models.llama.modeling_llama import LlamaDecoderLayer

dist.init_process_group("nccl")
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))

config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_config(config)                   # random init, fp32 master weights

model = FSDP(
    model,
    auto_wrap_policy=functools.partial(transformer_auto_wrap_policy,
                                       transformer_layer_cls={LlamaDecoderLayer}),   # one shard unit per block
    sharding_strategy=ShardingStrategy.FULL_SHARD,                 # ZeRO-3; SHARD_GRAD_OP is ZeRO-2
    mixed_precision=MixedPrecision(param_dtype=torch.bfloat16,     # bf16 compute and all-gathers
                                   reduce_dtype=torch.bfloat16,    # bf16 gradient reduce-scatter
                                   buffer_dtype=torch.bfloat16),
    device_id=torch.cuda.current_device(),
    use_orig_params=True,
)
apply_activation_checkpointing(                                    # keep only each block's input
    model,
    checkpoint_wrapper_fn=functools.partial(checkpoint_wrapper, checkpoint_impl=CheckpointImpl.NO_REENTRANT),
    check_fn=lambda m: isinstance(m, LlamaDecoderLayer),
)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)

def train_step(input_ids):                                         # input_ids: (micro_batch, seq) on this rank
    loss = model(input_ids=input_ids, labels=input_ids).loss       # HF shifts labels internally
    loss.backward()                                                # reduce-scatter of gradient shards
    model.clip_grad_norm_(1.0)                                     # global norm across all shards
    opt.step()                                                     # each rank updates only its 1/d slice
    opt.zero_grad(set_to_none=True)
    return loss.detach()
```

FSDP (ZeRO-3) training of a Llama-style model with bf16 mixed precision and per-layer activation checkpointing; launch with torchrun --nproc_per_node=8 (written against PyTorch 2.x and transformers; not run here)

> **Pipelines do not shrink first-stage activations:** Splitting $L$ layers over $p$ stages divides the weights by $p$. Under 1F1B, though, the first stage keeps $p$ micro-batches of $L/p$ layers in flight, which is $L$ layers' worth of activations. Doubling $p$ leaves stage 0's activation memory unchanged. Recomputation, sequence parallelism and context parallelism reduce it, and GPipe makes it worse ($m$ micro-batches). In the viz, GPipe at $m=32$ needs 91.3 GB of activations where 1F1B needs 11.4 GB.

> **Match each axis to a level of the network:** The axes differ in how often and how much they communicate. Tensor parallelism all-reduces activations four times per layer and must stay on NVLink. Pipeline parallelism sends one activation tensor per stage boundary and tolerates inter-node links. Data parallelism sends the most bytes, but once per step and overlapped with compute, so it can span the whole cluster. This is why Llama 3 405B used $t=8$ inside the node, $p=16$ across nodes and $d=128$ outermost.

> **Global batch is a hyperparameter, not a free knob:** More data-parallel replicas and more micro-batches both raise the tokens per optimizer step: $d\cdot m\cdot b\cdot s$. Past the critical batch size, extra tokens per step no longer reduce the number of steps proportionally, so scaling out stops helping convergence even when hardware utilisation looks perfect. Large runs ramp the batch up during training (GPT-3 went from 32K to 3.2M tokens) and pick $m$ to balance the bubble against this limit.

**Strengths**

- ZeRO/FSDP removes data-parallel memory redundancy at the same traffic for stages 1–2 (1.5× for stage 3)
- Tensor parallelism shrinks weights and activations inside a node without changing the batch
- Pipelines cross slow inter-node links with only point-to-point activation traffic
- Recomputation trades extra compute (up to a third for full checkpointing) for far less activation memory: 5.7× (selective) to 97× (full) per layer in the worked example

**Limitations**

- Tensor parallelism puts four all-reduces per layer on the critical path; beyond one node it collapses
- Pipeline bubbles cost (p − 1)/(m + p − 1) of the step unless the global batch is large
- ZeRO-3 all-gathers every layer twice per step and stalls when the network cannot keep up
- 3D layouts multiply failure points: at 16K GPUs, hardware failures interrupt training often

**Question:** A 10B-parameter model is trained with mixed-precision Adam and ZeRO-2 over 16 data-parallel GPUs (no tensor or pipeline parallelism). How much model-state memory does each GPU hold?

- A. 160 GB
- B. 10 GB
- C. 28.75 GB
- D. 47.5 GB

**Answer:** C. 28.75 GB — ZeRO-2 keeps full bf16 weights ($2\Psi=20$ GB) and shards gradients and optimizer state: $14\Psi/16=140/16=8.75$ GB. Total 28.75 GB. Plain data parallelism would be $16\Psi=160$ GB, ZeRO-1 $4\Psi+12\Psi/16=47.5$ GB, and ZeRO-3 $160/16=10$ GB.

**Question:** A pipeline has p = 8 stages. What is the smallest number of micro-batches m that brings the bubble (p − 1)/(m + p − 1) to 10% or less?

- A. 8
- B. 32
- C. 63
- D. 80

**Answer:** C. 63 — $7/(m+7)\le0.1\iff m+7\ge70\iff m\ge63$. At $m=8$ the bubble is $7/15=46.7\%$, and at $m=32$ it is $7/39=17.9\%$. Interleaving $v$ chunks per device would reach the same bubble with about $v$ times fewer micro-batches.

**Key points**

- Mixed-precision Adam needs $16\Psi$ bytes ($2+2+12$); activations add about $sbh(34+5as/h)$ per layer, or $34\,sbh/t$ with sequence parallelism and selective recompute.
- ZeRO shards over data parallelism: stage 1 $4\Psi+12\Psi/d$, stage 2 $2\Psi+14\Psi/d$, stage 3/FSDP $16\Psi/d$, at 1× (stages 1–2) or 1.5× (stage 3) of DP traffic.
- Megatron tensor parallelism splits A by columns and B by rows so GeLU stays local; four all-reduces per layer keep it inside a node (t ≤ 8).
- Pipeline bubble $(p-1)/(m+p-1)$; 1F1B caps in-flight micro-batches at $p$, interleaving divides the bubble by $v$, and stage 0 still holds $L$ layers of activations.
- 3D layout follows bandwidth: TP on NVLink, PP across nodes, DP outermost; context parallelism splits long sequences; 40–55% MFU is good at scale.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
