Distributed LLM training

Mixed-precision Adam needs 16 bytes per parameter plus activations, so LLMs are trained split across GPUs: ZeRO/FSDP shards states over data-parallel replicas, Megatron tensor parallelism splits matrices inside a node, and pipelines split layers at a bubble cost of (p−1)/(m+p−1).

16 min read1602 wordsadvanced0 viewsupdated 2026-10-13

Definition

Distributed LLM training is the practice of training one language model on many accelerators at once by splitting its work along several axes. Data parallelism gives each replica different examples and averages gradients. ZeRO (Rajbhandari, Rasley, Ruwase and He, SC 2020) and PyTorch FSDP (Zhao et al., 2023) shard the optimizer state, gradients and weights across those replicas. Tensor parallelism (Shoeybi et al., "Megatron-LM", 2019) splits each weight matrix across the GPUs of a node. Pipeline parallelism (GPipe, Huang et al., 2019; PipeDream's 1F1B, Narayanan et al., 2019) places consecutive layers on different devices. Sequence or context parallelism splits the tokens of one sequence. Combining data, tensor and pipeline parallelism is called 3D parallelism (Narayanan et al., SC 2021).

An 80 GB GPU holds the bf16 weights of a 40B-parameter model, but training needs far more than the weights. Adam keeps two moments per parameter, mixed precision keeps an fp32 master copy, and the backward pass needs the activations of every layer. A 70B model needs over 1.1 TB just for its training state. Even when a model fits, one GPU is far too slow: Llama 2 70B took 1,720,320 A100 GPU-hours, which is 196 years on a single device. Distributed training has to solve both problems, memory and time, without letting communication eat the speedup.

Where the memory goes

Mstates=2Ψ⏟bf16 weights+2Ψ⏟bf16 grads+4Ψ+4Ψ+4Ψ⏟fp32 master, m, v=16 Ψ bytesM_{\text{states}}=\underbrace{2\Psi}_{\text{bf16 weights}}+\underbrace{2\Psi}_{\text{bf16 grads}}+\underbrace{4\Psi+4\Psi+4\Psi}_{\text{fp32 master},\ m,\ v}=16\,\Psi\ \text{bytes}
Ψ\Psi
number of parameters
2Ψ+2Ψ2\Psi+2\Psi
the 16-bit weights used in forward and backward, and their gradients
12Ψ12\Psi
the optimizer state, K=12K=12 in the ZeRO paper: an fp32 master copy (updates smaller than bf16's resolution would otherwise be lost) and Adam's first and second moments
Mact≈s b h(34+5 a sh) bytes per layer,→ TP t + seq. parallel  s b h(34t+5 a sh t)M_{\text{act}}\approx s\,b\,h\Big(34+\frac{5\,a\,s}{h}\Big)\ \text{bytes per layer},\qquad \xrightarrow{\ \text{TP }t\ +\ \text{seq. parallel}\ }\ s\,b\,h\Big(\frac{34}{t}+\frac{5\,a\,s}{h\,t}\Big)
s,b,h,as,b,h,a
sequence length, micro-batch size, hidden size, attention heads (Korthikanti et al., 2022, for a GPT block with 16-bit activations)
34 sbh34\,sbh
inputs to the linear layers, GeLU, LayerNorms and dropout masks
5as2b5as^2b
the a×s×sa\times s\times s attention scores, softmax output and dropout mask: quadratic in ss
selective\text{selective}
recomputing only the attention part (or using FlashAttention) removes the 5as/h5as/h term, leaving 34 sbh/t34\,sbh/t
full\text{full}
full activation checkpointing keeps only each layer's input, 2sbh2sbh, and recomputes the rest in the backward pass

For Llama 2 70B (Ψ=6.9×1010\Psi=6.9\times10^{10}), 16Ψ16\Psi is 1,104 GB. At s=4096s=4096, b=1b=1, h=8192h=8192 and a=64a=64, one layer's activations are sbh(34+160)=6.5sbh(34+160)=6.5 GB, and there are 80 layers. Most of that is the attention-score term, which is why FlashAttention matters for training memory as well as for speed. Both kinds of memory have to be divided across GPUs.

Data parallelism and ZeRO

Plain data parallelism copies the whole model to each of dd GPUs, gives each a different slice of the global batch, and averages gradients with an all-reduce before every optimizer step. It scales throughput but saves no memory: every replica holds all 16Ψ16\Psi bytes. ZeRO observes that this redundancy is unnecessary. Each replica only needs to update a 1/d1/d slice of the parameters, so it only needs that slice's optimizer state.

ZeRO-1: 4Ψ+12ΨdZeRO-2: 2Ψ+14ΨdZeRO-3 / FSDP: 16Ψd\text{ZeRO-1: } 4\Psi+\frac{12\Psi}{d}\qquad \text{ZeRO-2: } 2\Psi+\frac{14\Psi}{d}\qquad \text{ZeRO-3 / FSDP: } \frac{16\Psi}{d}
ZeRO-1\text{ZeRO-1}
shard the optimizer state (PosP_{os}); after the update, all-gather the new weights
ZeRO-2\text{ZeRO-2}
also shard gradients (Pos+gP_{os+g}): reduce-scatter instead of all-reduce, so each rank receives only its slice
ZeRO-3\text{ZeRO-3}
also shard weights (Pos+g+pP_{os+g+p}): each layer's weights are all-gathered just before its forward and backward, then freed. PyTorch FSDP's FULL_SHARD is the same idea

An all-reduce is a reduce-scatter followed by an all-gather, so stages 1 and 2 send the same volume as plain data parallelism: about 2Ψ2\Psi elements per step for each GPU. Stage 3 adds one more all-gather of the weights for the backward pass, 1.5 times the traffic. The paper's example, a 7.5B model on 64 GPUs, needs 120 GB per GPU with plain data parallelism, 31.4 GB with ZeRO-1, 16.6 GB with ZeRO-2 and 1.9 GB with ZeRO-3. ZeRO-Offload and ZeRO-Infinity go further and move optimizer state to CPU memory or NVMe.

tensor parallel inside a node, pipeline across nodes, data across replicasMegatron MLP, t = 2XfGPU 0X·A₁GeLU·B₁GPU 1X·A₂GeLU·B₂ΣgA = [A₁ | A₂]: column split, GeLU localB = [B₁ ; B₂]: row split, Y = Y₁ + Y₂g: all-reduce forward, f: in backwardattention: heads split the same wayt = 8 × p = 4 × d = 2 = 64 GPUsstage 0stage 1stage 2stage 3replica 0nodenodenodenodereplica 1nodenodenodenode■ TP: all-reduce per layer, in-node NVLink→ PP: activations to next stage, p2p┆ DP: gradient all-reduce once per stepmodel states per GPU, Ψ = 7.5B on d = 64 (the ZeRO paper's example): 2Ψ + 2Ψ + 12Ψ bytesplain DPZeRO-0120.0 GBZeRO-1shard optimizer31.4 GBZeRO-2+ gradients16.6 GBZeRO-3+ weights (FSDP)1.88 GB■ bf16 weights ■ bf16 grads ■ fp32 master + Adam m, v · ZeRO-3 sends 1.5× the DP traffic
fig.Left: Megatron's tensor-parallel MLP on two GPUs. The first weight matrix is split by columns, so each GPU computes its own half of the hidden layer and applies GeLU locally. The second is split by rows, so each GPU produces a partial output, and one all-reduce (g) sums them. In the backward pass the conjugate operator f all-reduces the input gradient. Right: 3D parallelism on 64 GPUs. Each node of 8 GPUs runs one tensor-parallel group over NVLink, 4 nodes form a pipeline that passes activations forward and gradients back, and 2 such pipelines are data-parallel replicas that all-reduce gradients once per step. Bottom: model-state memory per GPU for the ZeRO paper's 7.5B model on 64 GPUs: 120 GB, 31.4 GB, 16.6 GB and 1.88 GB. Hover a legend line to highlight that axis.

Tensor parallelism (Megatron)

Y=GeLU(XA) B,A=[A1 ∣ A2], B=[B1B2] ⇒ Y=GeLU(XA1)B1+GeLU(XA2)B2Y=\mathrm{GeLU}(XA)\,B,\quad A=[A_1\,|\,A_2],\ B=\begin{bmatrix}B_1\\ B_2\end{bmatrix}\ \Rightarrow\ Y=\mathrm{GeLU}(XA_1)B_1+\mathrm{GeLU}(XA_2)B_2
[A1∣A2][A_1|A_2]
column-parallel: GPU ii computes XAiXA_i, a slice of the hidden units, so the elementwise GeLU needs no communication
[B1;B2][B_1;B_2]
row-parallel: GPU ii multiplies its hidden slice by its rows of BB and gets a partial sum of YY
++
one all-reduce of an s×b×hs\times b\times h tensor after the MLP, and one after attention, where the split is by heads. Two more come in the backward pass

Each GPU in a group of tt holds 1/t1/t of every attention and MLP matrix, so weights, gradients, optimizer state and most activations shrink by tt. The cost is four all-reduces of the full activation tensor per layer and micro-batch. Those are on the critical path, so tensor parallelism is kept inside a node, where NVLink is roughly an order of magnitude faster than the network between nodes. t=8t=8 (one node) is the usual ceiling. Shoeybi et al. trained an 8.3B model on 512 V100 GPUs at 15.1 PFLOPS with 76% scaling efficiency. Sequence parallelism (Korthikanti et al., 2022) splits the LayerNorm and dropout regions, which tensor parallelism leaves replicated, along the sequence. It turns each all-reduce into a reduce-scatter plus an all-gather with the same total volume, and divides the remaining 10 sbh10\,sbh of activations by tt as well.

Pipeline parallelism

Pipeline parallelism splits the LL layers into pp stages on different devices. Only the activations at stage boundaries cross the network, point-to-point, so it suits slow links between nodes. The problem is idle time. With one batch, stage 2 waits for stage 1, and stage 1 waits for stage 2's backward. GPipe splits each batch into mm micro-batches and pipelines them, accumulating gradients until all have finished.

bubble fraction=(p−1)(tf+tb)(m+p−1)(tf+tb)=p−1m+p−1,interleaved with v chunks per device: tbubbletideal=p−1v m\text{bubble fraction}=\frac{(p-1)(t_f+t_b)}{(m+p-1)(t_f+t_b)}=\frac{p-1}{m+p-1},\qquad \text{interleaved with } v \text{ chunks per device: } \frac{t_{\text{bubble}}}{t_{\text{ideal}}}=\frac{p-1}{v\,m}
p, mp,\ m
pipeline stages and micro-batches per optimizer step
p−1p-1
slots each stage spends waiting at the start (fill) and end (drain)
vv
Narayanan et al. (2021) give each device vv non-adjacent chunks of layers, shrinking the bubble vv-fold for vv times more point-to-point messages

GPipe runs all forwards and then all backwards, so the first stage stores activations for all mm micro-batches. 1F1B (PipeDream) starts each micro-batch's backward as soon as possible and then alternates one forward with one backward. The bubble is identical, but a stage never holds more than pp micro-batches in flight, so mm can grow to shrink the bubble without growing memory. GPipe reported the bubble to be negligible once m≥4pm\ge4p. The first stage still holds pp micro-batches of L/pL/p layers, which is LL layers' worth of activations however deep the pipeline. Pipelines therefore cut weight memory but not activation memory on the first stage.

Sequence and context parallelism

At 128K tokens even 34 sbh/t34\,sbh/t is large for a single sequence, and the batch cannot be split further. Context parallelism shards the sequence dimension of every layer across cc GPUs. Attention then needs all keys and values: ring attention (Liu, Zaharia and Abbeel, 2023) passes K/V blocks around the ring while computing blockwise attention, and other implementations all-gather the keys and values. Llama 3 405B's 128K-context stage used 16-way context parallelism. Causal masks unbalance the work, so each GPU takes one chunk from the start of the sequence and one from the end (see Context window optimization). Mixture-of-experts models add expert parallelism, placing different experts on different GPUs with all-to-all token routing (see Mixture of experts (MoE)).

Published configurations

Megatron-LM (2019)
8.3B GPT-2-style model, 8-way tensor parallel, 512 V100s, 15.1 PFLOPS sustained, 76% weak-scaling efficiency
ZeRO (2020)
7.5B on 64 GPUs: 120 → 31.4 → 16.6 → 1.9 GB per GPU of model states; used to train the 17B Turing-NLG
Megatron PTD-P (2021)
3D parallelism with interleaved 1F1B: a 1T-parameter model at 502 PFLOP/s on 3072 A100s, 52% of peak per GPU
Korthikanti et al. (2022)
sequence parallelism + selective recomputation: about 5× less activation memory, and over 90% less recomputation overhead than full checkpointing
Llama 2 70B (2023)
2T tokens, 1,720,320 A100-80GB GPU-hours; 6ND=8.3×10236ND=8.3\times10^{23} FLOPs implies about 43% of the 312 TFLOPS peak
Llama 3 405B (2024)
16,384 H100s; 8-way tensor × 16-way pipeline × 128-way data parallel at 8K sequence length, 38–43% bf16 MFU; context parallelism for the 128K stage
Interconnect
H100 NVLink about 900 GB/s per GPU inside a node; 400 Gb/s (50 GB/s) InfiniBand per GPU between nodes
16 bytes/param × Ψ/(t·p), sharded by ZeRO over d · activations per Korthikanti et al. 2022Llama 2 70B: 64 GPUs = d 2 × t 8 × p 4 · 2.16B params per model shard80 GB■ bf16 weights 4.31■ bf16 grads 4.31■ fp32 Adam 12.9■ activations 11.4per GPU: 33.0 GB · fits in 80 GB · states 21.6 GB (ZeRO-1)activations: 143 MB/layer × 20 layers × 4 micro-batches in flightsent per GPU per step: DP 4.31 GB · TP 75.2 GB (NVLink) · PP 1.07 GBpipeline: p = 4 stages, m = 8 micro-batches, 1F1B · F = 1 unit, B = 2 unitss0s1s2s31234123121122132331424344152653654655663747576774858687885678788time →step = 33 units (ideal 24)bubble (p − 1)/(m + p − 1) = 27.3% · simulated idle = 27.3%stage 0 holds 4 micro-batches of activations at peak (1F1B: min(p, m))
model
ZeRO stage
recompute
fig.Top: per-GPU memory for the chosen model and parallel layout, split into bf16 weights, bf16 gradients, fp32 optimizer state and activations, against an 80 GB GPU. At the defaults, Llama 2 70B on 64 GPUs (d 2 × t 8 × p 4) with ZeRO-1, selective recomputation, sequence parallelism, 4096-token sequences, micro-batch 1 and m = 8, each model shard holds 2.16B parameters. Per GPU that is 4.31 + 4.31 + 12.9 GB of states plus 11.4 GB of activations (143 MB per layer × 20 layers × 4 micro-batches in flight), 33.0 GB in total. Each GPU sends 4.31 GB of data-parallel and 75.2 GB of tensor-parallel traffic per step. With no recomputation the activations grow to 65.1 GB and the total to 86.7 GB, which does not fit. Without sequence parallelism they are 34.9 GB, and with full recomputation 1.48 GB. Switching to GPipe holds all 8 micro-batches: 22.8 GB of activations. Bottom: the simulated schedule, with forward 1 unit and backward 2. p = 4 and m = 8 take 33 units instead of 24, a 27.3% bubble, matching (p − 1)/(m + p − 1). m = 32 cuts it to 8.6% with no extra memory under 1F1B, while GPipe would need 91.3 GB of activations. Llama 2 7B on one GPU with no sharding and no recomputation needs 212 GB. ZeRO-3 over 8 GPUs brings it to 31.7 GB. Llama 2 70B with ZeRO-3 over 64 GPUs and full recomputation needs 29.1 GB per GPU but sends 407 GB per GPU per step.

Mixed precision and activation checkpointing

Micikevicius et al. ("Mixed Precision Training", ICLR 2018) ran forward and backward in fp16 while keeping fp32 master weights for the update, and scaled the loss so that small gradients would not underflow. bf16 has fp32's 8-bit exponent, so it needs no loss scaling and is now the default for LLMs. Matrix multiplies run in bf16 with fp32 accumulation, while softmax, norms and the optimizer stay in fp32. FP8 matrix multiplies with per-tensor or finer-grained scales are the next step: DeepSeek-V3 was trained largely in FP8.

Activation checkpointing (Chen et al., "Training Deep Nets with Sublinear Memory Cost", 2016) stores only some activations and recomputes the rest during the backward pass. Checkpointing every layer's input costs one extra forward pass, about 33% more compute, since a forward is roughly a third of forward plus backward. Selective recomputation recomputes only the attention scores, which are large in memory and cheap in FLOPs. FlashAttention already behaves this way because it never stores the score matrix.

Communication costs

ring all-reduce sends 2(n−1)n ∣x∣ per GPU,DP: ∣x∣=2Ψ′ bytes/step,TP: 4⋅Lp⋅m⋅2sbh bytes/step,C≈6ND\text{ring all-reduce sends } \frac{2(n-1)}{n}\,|x|\ \text{per GPU},\qquad \text{DP: } |x|=2\Psi'\ \text{bytes/step},\qquad \text{TP: } 4\cdot\tfrac{L}{p}\cdot m\cdot 2sbh\ \text{bytes/step},\qquad C\approx6ND
Ψ′=Ψ/(tp)\Psi'=\Psi/(tp)
parameters in one model shard; data-parallel gradients are reduced once per step and can overlap with the backward pass
4⋅Lp⋅m4\cdot\frac Lp\cdot m
four tensor-parallel all-reduces per layer per micro-batch, on the critical path
C≈6NDC\approx6ND
training FLOPs for NN parameters and DD tokens; wall time is C/(#GPUs⋅peak⋅MFU)C/(\#\text{GPUs}\cdot\text{peak}\cdot\text{MFU})

The layout follows the bandwidth hierarchy. The chattiest axis, tensor parallelism, stays on NVLink. Pipeline parallelism sends only boundary activations and crosses nodes. Data parallelism sends the most bytes, but only once per step, and overlaps them with the backward pass, so it takes the outermost and slowest links. Model FLOPs utilisation (MFU) of 40–55% on thousands of GPUs is considered good. The rest goes to communication that cannot be hidden, pipeline bubbles, recomputation and stragglers.

Worked example: Llama 2 70B on 64 GPUs

Ψ=68.98×109\Psi=68.98\times10^9, L=80L=80, h=8192h=8192, a=64a=64, sequence s=4096s=4096, micro-batch b=1b=1, m=8m=8 micro-batches per step, 80 GB GPUs in nodes of 8.

layoutmodel statesactivationstotal
plain DP, 64 replicas16Ψ = 1,103.6 GB—does not fit
ZeRO-3 over 6416Ψ/64 = 17.2 GBper layer 6.5 GB without recomputedoes not fit
t 8 × p 4 × d 2, ZeRO-1, no recompute4Ψ′ + 12Ψ′/2 = 21.6 GB813.7 MB × 20 × 4 = 65.1 GB86.7 GB
same, selective, no seq. parallel21.6 GB436.2 MB × 80 = 34.9 GB56.5 GB
same, selective + seq. parallel21.6 GB142.6 MB × 80 = 11.4 GB33.0 GB
same, full recompute21.6 GB8.4 MB × 80 + 0.8 GB = 1.5 GB23.0 GB
Per-GPU memory under successive choices (computed with the Python below)

Working it through

  1. 1Shard the model: Ψ′=Ψ/(t p)=68.98B/32=2.156\Psi'=\Psi/(t\,p)=68.98\text{B}/32=2.156B parameters per GPU. ZeRO-1 over d=2d=2: 4⋅2.156+12⋅2.156/2=8.62+12.93=21.64\cdot2.156+12\cdot2.156/2=8.62+12.93=21.6 GB.
  2. 2Activations per layer with t=8t=8 and sequence parallelism: sbh=4096⋅8192=3.36×107sbh=4096\cdot8192=3.36\times10^7. No recompute: sbh(34/8+5⋅64⋅4096/(8192⋅8))=sbh(4.25+20)=813.7sbh(34/8+5\cdot64\cdot4096/(8192\cdot8))=sbh(4.25+20)=813.7 MB. The attention term is 82% of it. Selective recompute: sbh⋅4.25=142.6sbh\cdot4.25=142.6 MB.
  3. 3The first pipeline stage holds 80/4=2080/4=20 layers for min⁡(p,m)=4\min(p,m)=4 micro-batches under 1F1B: 142.6 MB×80=11.4142.6\text{ MB}\times80=11.4 GB. Total 21.6+11.4=33.021.6+11.4=33.0 GB, which leaves room for buffers and fragmentation.
  4. 4Bubble: (4−1)/(8+4−1)=27.3%(4-1)/(8+4-1)=27.3\%. With m=32m=32 it is 3/35=8.6%3/35=8.6\% at the same memory, but the global batch grows to d⋅m⋅b⋅s=2⋅32⋅4096=262,144d\cdot m\cdot b\cdot s=2\cdot32\cdot4096=262{,}144 tokens.
  5. 5Traffic per GPU per step at m=8m=8: data parallel 2⋅12⋅2Ψ′=4.312\cdot\tfrac12\cdot2\Psi'=4.31 GB once, overlapped with the backward pass. Tensor parallel 4⋅20⋅8⋅74⋅2sbh=75.24\cdot20\cdot8\cdot\tfrac74\cdot2sbh=75.2 GB, on NVLink, on the critical path.
  6. 6Time: one step processes 2⋅8⋅4096=65,5362\cdot8\cdot4096=65{,}536 tokens, 6Ψ⋅65,536=2.71×10166\Psi\cdot65{,}536=2.71\times10^{16} FLOPs. At 50% of 312 TFLOPS on 64 GPUs that is 2.7 s per step. Llama 2 70B's 2T tokens need 8.3×10238.3\times10^{23} FLOPs, and its reported 1,720,320 GPU-hours imply 43% MFU.
python
1import math
2
3def model_states(psi, d, zero):
4 """bf16 weights + bf16 grads + fp32 (master, m, v) = 2 + 2 + 12 bytes/param; ZeRO shards over d."""
5 w = 2 * psi / (d if zero >= 3 else 1)
6 g = 2 * psi / (d if zero >= 2 else 1)
7 o = 12 * psi / (d if zero >= 1 else 1)
8 return w + g + o
9
10def act_per_layer(s, b, h, a, t=1, recompute="selective", sp=True):
11 """Korthikanti et al. 2022, bytes per transformer layer with 16-bit activations."""
12 sp = sp and t > 1
13 if recompute == "full":
14 return 2 * s * b * h / (t if sp else 1)
15 lin = 34 / t if sp else 10 + 24 / t
16 attn = 5 * a * s / (h * t) if recompute == "none" else 0.0 # the softmax/dropout part
17 return s * b * h * (lin + attn)
18
19def per_gpu_gb(params, L, h, a, s, b, t, p, d, zero, recompute, sp, m):
20 states = model_states(params / (t * p), d, zero)
21 inflight = min(p, m) # 1F1B: first stage holds p micro-batches
22 act = act_per_layer(s, b, h, a, t, recompute, sp) * math.ceil(L / p) * inflight
23 if recompute == "full":
24 act += act_per_layer(s, b, h, a, t, "none", sp) # one layer re-materialised at a time
25 return states / 1e9, act / 1e9
26
27def simulate(p, m, schedule="1f1b", tf=1, tb=2):
28 """Run a pipeline schedule with real dependencies; return (makespan, idle fraction)."""
29 def order(k):
30 if schedule == "gpipe":
31 return [("F", i) for i in range(m)] + [("B", i) for i in range(m)]
32 warm = min(p - k - 1, m) # 1F1B: warm-up forwards, then alternate
33 ops, f, bk = [("F", i) for i in range(warm)], warm, 0
34 while f < m:
35 ops += [("F", f), ("B", bk)]; f += 1; bk += 1
36 return ops + [("B", i) for i in range(bk, m)]
37 lists, ptr, free, done = [order(k) for k in range(p)], [0] * p, [0] * p, {}
38 while any(ptr[k] < len(lists[k]) for k in range(p)):
39 for k in range(p):
40 while ptr[k] < len(lists[k]):
41 kind, i = lists[k][ptr[k]]
42 dep = (0 if k == 0 else done.get(("F", k - 1, i))) if kind == "F" else \
43 (done.get(("F", k, i)) if k == p - 1 else done.get(("B", k + 1, i)))
44 if dep is None:
45 break
46 free[k] = max(free[k], dep) + (tf if kind == "F" else tb)
47 done[(kind, k, i)] = free[k]; ptr[k] += 1
48 span = max(free)
49 return span, 1 - p * m * (tf + tb) / (p * span)
50
51# Llama 2 70B on 64 GPUs: t = 8 (in node), p = 4, d = 2, ZeRO-1, sequence 4096, micro-batch 1, m = 8
52for rc, sp in (("none", True), ("selective", False), ("selective", True), ("full", True)):
53 st, act = per_gpu_gb(68.976e9, 80, 8192, 64, 4096, 1, 8, 4, 2, 1, rc, sp, 8)
54 print(f"{rc:9s} SP={sp!s:5s} states {st:5.2f} GB activations {act:5.2f} GB total {st + act:5.2f} GB")
55print("plain DP:", 16 * 68.976e9 / 1e9, "GB per GPU; ZeRO-3 over 64:", round(16 * 68.976e9 / 64 / 1e9, 2), "GB")
56for m in (8, 32):
57 span, idle = simulate(4, m)
58 print(f"p=4 m={m}: makespan {span} units, idle {idle:.3f}, formula {(4 - 1) / (m + 4 - 1):.3f}")
59# none SP=True states 21.55 GB activations 65.10 GB total 86.65 GB
60# selective SP=False states 21.55 GB activations 34.90 GB total 56.45 GB
61# selective SP=True states 21.55 GB activations 11.41 GB total 32.96 GB
62# full SP=True states 21.55 GB activations 1.48 GB total 23.04 GB
63# plain DP: 1103.616 GB per GPU; ZeRO-3 over 64: 17.24 GB
64# p=4 m=8: makespan 33 units, idle 0.273, formula 0.273
65# p=4 m=32: makespan 105 units, idle 0.086, formula 0.086
Per-GPU memory and pipeline-bubble calculator for any 3D layout (pure Python, run; printed output shown)
python
1import functools, os, torch, torch.distributed as dist
2from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision, ShardingStrategy
3from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
4from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (
5 CheckpointImpl, apply_activation_checkpointing, checkpoint_wrapper)
6from transformers import AutoConfig, AutoModelForCausalLM
7from transformers.models.llama.modeling_llama import LlamaDecoderLayer
8
9dist.init_process_group("nccl")
10torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
11
12config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
13model = AutoModelForCausalLM.from_config(config) # random init, fp32 master weights
14
15model = FSDP(
16 model,
17 auto_wrap_policy=functools.partial(transformer_auto_wrap_policy,
18 transformer_layer_cls={LlamaDecoderLayer}), # one shard unit per block
19 sharding_strategy=ShardingStrategy.FULL_SHARD, # ZeRO-3; SHARD_GRAD_OP is ZeRO-2
20 mixed_precision=MixedPrecision(param_dtype=torch.bfloat16, # bf16 compute and all-gathers
21 reduce_dtype=torch.bfloat16, # bf16 gradient reduce-scatter
22 buffer_dtype=torch.bfloat16),
23 device_id=torch.cuda.current_device(),
24 use_orig_params=True,
25)
26apply_activation_checkpointing( # keep only each block's input
27 model,
28 checkpoint_wrapper_fn=functools.partial(checkpoint_wrapper, checkpoint_impl=CheckpointImpl.NO_REENTRANT),
29 check_fn=lambda m: isinstance(m, LlamaDecoderLayer),
30)
31opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
32
33def train_step(input_ids): # input_ids: (micro_batch, seq) on this rank
34 loss = model(input_ids=input_ids, labels=input_ids).loss # HF shifts labels internally
35 loss.backward() # reduce-scatter of gradient shards
36 model.clip_grad_norm_(1.0) # global norm across all shards
37 opt.step() # each rank updates only its 1/d slice
38 opt.zero_grad(set_to_none=True)
39 return loss.detach()
FSDP (ZeRO-3) training of a Llama-style model with bf16 mixed precision and per-layer activation checkpointing; launch with torchrun --nproc_per_node=8 (written against PyTorch 2.x and transformers; not run here)

Use when

  • +ZeRO/FSDP removes data-parallel memory redundancy at the same traffic for stages 1–2 (1.5× for stage 3)
  • +Tensor parallelism shrinks weights and activations inside a node without changing the batch
  • +Pipelines cross slow inter-node links with only point-to-point activation traffic
  • +Recomputation trades extra compute (up to a third for full checkpointing) for far less activation memory: 5.7× (selective) to 97× (full) per layer in the worked example

Avoid when

  • −Tensor parallelism puts four all-reduces per layer on the critical path; beyond one node it collapses
  • −Pipeline bubbles cost (p − 1)/(m + p − 1) of the step unless the global batch is large
  • −ZeRO-3 all-gathers every layer twice per step and stalls when the network cannot keep up
  • −3D layouts multiply failure points: at 16K GPUs, hardware failures interrupt training often

Check yourself

A 10B-parameter model is trained with mixed-precision Adam and ZeRO-2 over 16 data-parallel GPUs (no tensor or pipeline parallelism). How much model-state memory does each GPU hold?

Check yourself

A pipeline has p = 8 stages. What is the smallest number of micro-batches m that brings the bubble (p − 1)/(m + p − 1) to 10% or less?

Remember

  • ✓Mixed-precision Adam needs 16Ψ16\Psi bytes (2+2+122+2+12); activations add about sbh(34+5as/h)sbh(34+5as/h) per layer, or 34 sbh/t34\,sbh/t with sequence parallelism and selective recompute.
  • ✓ZeRO shards over data parallelism: stage 1 4Ψ+12Ψ/d4\Psi+12\Psi/d, stage 2 2Ψ+14Ψ/d2\Psi+14\Psi/d, stage 3/FSDP 16Ψ/d16\Psi/d, at 1× (stages 1–2) or 1.5× (stage 3) of DP traffic.
  • ✓Megatron tensor parallelism splits A by columns and B by rows so GeLU stays local; four all-reduces per layer keep it inside a node (t ≤ 8).
  • ✓Pipeline bubble (p−1)/(m+p−1)(p-1)/(m+p-1); 1F1B caps in-flight micro-batches at pp, interleaving divides the bubble by vv, and stage 0 still holds LL layers of activations.
  • ✓3D layout follows bandwidth: TP on NVLink, PP across nodes, DP outermost; context parallelism splits long sequences; 40–55% MFU is good at scale.
← back to Deep Learning roadmap