Definition
Distributed LLM training is the practice of training one language model on many accelerators at once by splitting its work along several axes. Data parallelism gives each replica different examples and averages gradients. ZeRO (Rajbhandari, Rasley, Ruwase and He, SC 2020) and PyTorch FSDP (Zhao et al., 2023) shard the optimizer state, gradients and weights across those replicas. Tensor parallelism (Shoeybi et al., "Megatron-LM", 2019) splits each weight matrix across the GPUs of a node. Pipeline parallelism (GPipe, Huang et al., 2019; PipeDream's 1F1B, Narayanan et al., 2019) places consecutive layers on different devices. Sequence or context parallelism splits the tokens of one sequence. Combining data, tensor and pipeline parallelism is called 3D parallelism (Narayanan et al., SC 2021).
An 80 GB GPU holds the bf16 weights of a 40B-parameter model, but training needs far more than the weights. Adam keeps two moments per parameter, mixed precision keeps an fp32 master copy, and the backward pass needs the activations of every layer. A 70B model needs over 1.1 TB just for its training state. Even when a model fits, one GPU is far too slow: Llama 2 70B took 1,720,320 A100 GPU-hours, which is 196 years on a single device. Distributed training has to solve both problems, memory and time, without letting communication eat the speedup.
Where the memory goes
- number of parameters
- the 16-bit weights used in forward and backward, and their gradients
- the optimizer state, in the ZeRO paper: an fp32 master copy (updates smaller than bf16's resolution would otherwise be lost) and Adam's first and second moments
- sequence length, micro-batch size, hidden size, attention heads (Korthikanti et al., 2022, for a GPT block with 16-bit activations)
- inputs to the linear layers, GeLU, LayerNorms and dropout masks
- the attention scores, softmax output and dropout mask: quadratic in
- recomputing only the attention part (or using FlashAttention) removes the term, leaving
- full activation checkpointing keeps only each layer's input, , and recomputes the rest in the backward pass
For Llama 2 70B (), is 1,104 GB. At , , and , one layer's activations are GB, and there are 80 layers. Most of that is the attention-score term, which is why FlashAttention matters for training memory as well as for speed. Both kinds of memory have to be divided across GPUs.
Data parallelism and ZeRO
Plain data parallelism copies the whole model to each of GPUs, gives each a different slice of the global batch, and averages gradients with an all-reduce before every optimizer step. It scales throughput but saves no memory: every replica holds all bytes. ZeRO observes that this redundancy is unnecessary. Each replica only needs to update a slice of the parameters, so it only needs that slice's optimizer state.
- shard the optimizer state (); after the update, all-gather the new weights
- also shard gradients (): reduce-scatter instead of all-reduce, so each rank receives only its slice
- also shard weights (): each layer's weights are all-gathered just before its forward and backward, then freed. PyTorch FSDP's FULL_SHARD is the same idea
An all-reduce is a reduce-scatter followed by an all-gather, so stages 1 and 2 send the same volume as plain data parallelism: about elements per step for each GPU. Stage 3 adds one more all-gather of the weights for the backward pass, 1.5 times the traffic. The paper's example, a 7.5B model on 64 GPUs, needs 120 GB per GPU with plain data parallelism, 31.4 GB with ZeRO-1, 16.6 GB with ZeRO-2 and 1.9 GB with ZeRO-3. ZeRO-Offload and ZeRO-Infinity go further and move optimizer state to CPU memory or NVMe.
Tensor parallelism (Megatron)
- column-parallel: GPU computes , a slice of the hidden units, so the elementwise GeLU needs no communication
- row-parallel: GPU multiplies its hidden slice by its rows of and gets a partial sum of
- one all-reduce of an tensor after the MLP, and one after attention, where the split is by heads. Two more come in the backward pass
Each GPU in a group of holds of every attention and MLP matrix, so weights, gradients, optimizer state and most activations shrink by . The cost is four all-reduces of the full activation tensor per layer and micro-batch. Those are on the critical path, so tensor parallelism is kept inside a node, where NVLink is roughly an order of magnitude faster than the network between nodes. (one node) is the usual ceiling. Shoeybi et al. trained an 8.3B model on 512 V100 GPUs at 15.1 PFLOPS with 76% scaling efficiency. Sequence parallelism (Korthikanti et al., 2022) splits the LayerNorm and dropout regions, which tensor parallelism leaves replicated, along the sequence. It turns each all-reduce into a reduce-scatter plus an all-gather with the same total volume, and divides the remaining of activations by as well.
Pipeline parallelism
Pipeline parallelism splits the layers into stages on different devices. Only the activations at stage boundaries cross the network, point-to-point, so it suits slow links between nodes. The problem is idle time. With one batch, stage 2 waits for stage 1, and stage 1 waits for stage 2's backward. GPipe splits each batch into micro-batches and pipelines them, accumulating gradients until all have finished.
- pipeline stages and micro-batches per optimizer step
- slots each stage spends waiting at the start (fill) and end (drain)
- Narayanan et al. (2021) give each device non-adjacent chunks of layers, shrinking the bubble -fold for times more point-to-point messages
GPipe runs all forwards and then all backwards, so the first stage stores activations for all micro-batches. 1F1B (PipeDream) starts each micro-batch's backward as soon as possible and then alternates one forward with one backward. The bubble is identical, but a stage never holds more than micro-batches in flight, so can grow to shrink the bubble without growing memory. GPipe reported the bubble to be negligible once . The first stage still holds micro-batches of layers, which is layers' worth of activations however deep the pipeline. Pipelines therefore cut weight memory but not activation memory on the first stage.
Sequence and context parallelism
At 128K tokens even is large for a single sequence, and the batch cannot be split further. Context parallelism shards the sequence dimension of every layer across GPUs. Attention then needs all keys and values: ring attention (Liu, Zaharia and Abbeel, 2023) passes K/V blocks around the ring while computing blockwise attention, and other implementations all-gather the keys and values. Llama 3 405B's 128K-context stage used 16-way context parallelism. Causal masks unbalance the work, so each GPU takes one chunk from the start of the sequence and one from the end (see Context window optimization). Mixture-of-experts models add expert parallelism, placing different experts on different GPUs with all-to-all token routing (see Mixture of experts (MoE)).
Published configurations
- Megatron-LM (2019)
- 8.3B GPT-2-style model, 8-way tensor parallel, 512 V100s, 15.1 PFLOPS sustained, 76% weak-scaling efficiency
- ZeRO (2020)
- 7.5B on 64 GPUs: 120 → 31.4 → 16.6 → 1.9 GB per GPU of model states; used to train the 17B Turing-NLG
- Megatron PTD-P (2021)
- 3D parallelism with interleaved 1F1B: a 1T-parameter model at 502 PFLOP/s on 3072 A100s, 52% of peak per GPU
- Korthikanti et al. (2022)
- sequence parallelism + selective recomputation: about 5× less activation memory, and over 90% less recomputation overhead than full checkpointing
- Llama 2 70B (2023)
- 2T tokens, 1,720,320 A100-80GB GPU-hours; FLOPs implies about 43% of the 312 TFLOPS peak
- Llama 3 405B (2024)
- 16,384 H100s; 8-way tensor × 16-way pipeline × 128-way data parallel at 8K sequence length, 38–43% bf16 MFU; context parallelism for the 128K stage
- Interconnect
- H100 NVLink about 900 GB/s per GPU inside a node; 400 Gb/s (50 GB/s) InfiniBand per GPU between nodes
Mixed precision and activation checkpointing
Micikevicius et al. ("Mixed Precision Training", ICLR 2018) ran forward and backward in fp16 while keeping fp32 master weights for the update, and scaled the loss so that small gradients would not underflow. bf16 has fp32's 8-bit exponent, so it needs no loss scaling and is now the default for LLMs. Matrix multiplies run in bf16 with fp32 accumulation, while softmax, norms and the optimizer stay in fp32. FP8 matrix multiplies with per-tensor or finer-grained scales are the next step: DeepSeek-V3 was trained largely in FP8.
Activation checkpointing (Chen et al., "Training Deep Nets with Sublinear Memory Cost", 2016) stores only some activations and recomputes the rest during the backward pass. Checkpointing every layer's input costs one extra forward pass, about 33% more compute, since a forward is roughly a third of forward plus backward. Selective recomputation recomputes only the attention scores, which are large in memory and cheap in FLOPs. FlashAttention already behaves this way because it never stores the score matrix.
Communication costs
- parameters in one model shard; data-parallel gradients are reduced once per step and can overlap with the backward pass
- four tensor-parallel all-reduces per layer per micro-batch, on the critical path
- training FLOPs for parameters and tokens; wall time is
The layout follows the bandwidth hierarchy. The chattiest axis, tensor parallelism, stays on NVLink. Pipeline parallelism sends only boundary activations and crosses nodes. Data parallelism sends the most bytes, but only once per step, and overlaps them with the backward pass, so it takes the outermost and slowest links. Model FLOPs utilisation (MFU) of 40–55% on thousands of GPUs is considered good. The rest goes to communication that cannot be hidden, pipeline bubbles, recomputation and stragglers.
Worked example: Llama 2 70B on 64 GPUs
, , , , sequence , micro-batch , micro-batches per step, 80 GB GPUs in nodes of 8.
| layout | model states | activations | total |
|---|---|---|---|
| plain DP, 64 replicas | 16Ψ = 1,103.6 GB | — | does not fit |
| ZeRO-3 over 64 | 16Ψ/64 = 17.2 GB | per layer 6.5 GB without recompute | does not fit |
| t 8 × p 4 × d 2, ZeRO-1, no recompute | 4Ψ′ + 12Ψ′/2 = 21.6 GB | 813.7 MB × 20 × 4 = 65.1 GB | 86.7 GB |
| same, selective, no seq. parallel | 21.6 GB | 436.2 MB × 80 = 34.9 GB | 56.5 GB |
| same, selective + seq. parallel | 21.6 GB | 142.6 MB × 80 = 11.4 GB | 33.0 GB |
| same, full recompute | 21.6 GB | 8.4 MB × 80 + 0.8 GB = 1.5 GB | 23.0 GB |
Working it through
- 1Shard the model: B parameters per GPU. ZeRO-1 over : GB.
- 2Activations per layer with and sequence parallelism: . No recompute: MB. The attention term is 82% of it. Selective recompute: MB.
- 3The first pipeline stage holds layers for micro-batches under 1F1B: GB. Total GB, which leaves room for buffers and fragmentation.
- 4Bubble: . With it is at the same memory, but the global batch grows to tokens.
- 5Traffic per GPU per step at : data parallel GB once, overlapped with the backward pass. Tensor parallel GB, on NVLink, on the critical path.
- 6Time: one step processes tokens, FLOPs. At 50% of 312 TFLOPS on 64 GPUs that is 2.7 s per step. Llama 2 70B's 2T tokens need FLOPs, and its reported 1,720,320 GPU-hours imply 43% MFU.
1import math23def model_states(psi, d, zero):4 """bf16 weights + bf16 grads + fp32 (master, m, v) = 2 + 2 + 12 bytes/param; ZeRO shards over d."""5 w = 2 * psi / (d if zero >= 3 else 1)6 g = 2 * psi / (d if zero >= 2 else 1)7 o = 12 * psi / (d if zero >= 1 else 1)8 return w + g + o910def act_per_layer(s, b, h, a, t=1, recompute="selective", sp=True):11 """Korthikanti et al. 2022, bytes per transformer layer with 16-bit activations."""12 sp = sp and t > 113 if recompute == "full":14 return 2 * s * b * h / (t if sp else 1)15 lin = 34 / t if sp else 10 + 24 / t16 attn = 5 * a * s / (h * t) if recompute == "none" else 0.0 # the softmax/dropout part17 return s * b * h * (lin + attn)1819def per_gpu_gb(params, L, h, a, s, b, t, p, d, zero, recompute, sp, m):20 states = model_states(params / (t * p), d, zero)21 inflight = min(p, m) # 1F1B: first stage holds p micro-batches22 act = act_per_layer(s, b, h, a, t, recompute, sp) * math.ceil(L / p) * inflight23 if recompute == "full":24 act += act_per_layer(s, b, h, a, t, "none", sp) # one layer re-materialised at a time25 return states / 1e9, act / 1e92627def simulate(p, m, schedule="1f1b", tf=1, tb=2):28 """Run a pipeline schedule with real dependencies; return (makespan, idle fraction)."""29 def order(k):30 if schedule == "gpipe":31 return [("F", i) for i in range(m)] + [("B", i) for i in range(m)]32 warm = min(p - k - 1, m) # 1F1B: warm-up forwards, then alternate33 ops, f, bk = [("F", i) for i in range(warm)], warm, 034 while f < m:35 ops += [("F", f), ("B", bk)]; f += 1; bk += 136 return ops + [("B", i) for i in range(bk, m)]37 lists, ptr, free, done = [order(k) for k in range(p)], [0] * p, [0] * p, {}38 while any(ptr[k] < len(lists[k]) for k in range(p)):39 for k in range(p):40 while ptr[k] < len(lists[k]):41 kind, i = lists[k][ptr[k]]42 dep = (0 if k == 0 else done.get(("F", k - 1, i))) if kind == "F" else \43 (done.get(("F", k, i)) if k == p - 1 else done.get(("B", k + 1, i)))44 if dep is None:45 break46 free[k] = max(free[k], dep) + (tf if kind == "F" else tb)47 done[(kind, k, i)] = free[k]; ptr[k] += 148 span = max(free)49 return span, 1 - p * m * (tf + tb) / (p * span)5051# Llama 2 70B on 64 GPUs: t = 8 (in node), p = 4, d = 2, ZeRO-1, sequence 4096, micro-batch 1, m = 852for rc, sp in (("none", True), ("selective", False), ("selective", True), ("full", True)):53 st, act = per_gpu_gb(68.976e9, 80, 8192, 64, 4096, 1, 8, 4, 2, 1, rc, sp, 8)54 print(f"{rc:9s} SP={sp!s:5s} states {st:5.2f} GB activations {act:5.2f} GB total {st + act:5.2f} GB")55print("plain DP:", 16 * 68.976e9 / 1e9, "GB per GPU; ZeRO-3 over 64:", round(16 * 68.976e9 / 64 / 1e9, 2), "GB")56for m in (8, 32):57 span, idle = simulate(4, m)58 print(f"p=4 m={m}: makespan {span} units, idle {idle:.3f}, formula {(4 - 1) / (m + 4 - 1):.3f}")59# none SP=True states 21.55 GB activations 65.10 GB total 86.65 GB60# selective SP=False states 21.55 GB activations 34.90 GB total 56.45 GB61# selective SP=True states 21.55 GB activations 11.41 GB total 32.96 GB62# full SP=True states 21.55 GB activations 1.48 GB total 23.04 GB63# plain DP: 1103.616 GB per GPU; ZeRO-3 over 64: 17.24 GB64# p=4 m=8: makespan 33 units, idle 0.273, formula 0.27365# p=4 m=32: makespan 105 units, idle 0.086, formula 0.0861import functools, os, torch, torch.distributed as dist2from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision, ShardingStrategy3from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy4from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (5 CheckpointImpl, apply_activation_checkpointing, checkpoint_wrapper)6from transformers import AutoConfig, AutoModelForCausalLM7from transformers.models.llama.modeling_llama import LlamaDecoderLayer89dist.init_process_group("nccl")10torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))1112config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")13model = AutoModelForCausalLM.from_config(config) # random init, fp32 master weights1415model = FSDP(16 model,17 auto_wrap_policy=functools.partial(transformer_auto_wrap_policy,18 transformer_layer_cls={LlamaDecoderLayer}), # one shard unit per block19 sharding_strategy=ShardingStrategy.FULL_SHARD, # ZeRO-3; SHARD_GRAD_OP is ZeRO-220 mixed_precision=MixedPrecision(param_dtype=torch.bfloat16, # bf16 compute and all-gathers21 reduce_dtype=torch.bfloat16, # bf16 gradient reduce-scatter22 buffer_dtype=torch.bfloat16),23 device_id=torch.cuda.current_device(),24 use_orig_params=True,25)26apply_activation_checkpointing( # keep only each block's input27 model,28 checkpoint_wrapper_fn=functools.partial(checkpoint_wrapper, checkpoint_impl=CheckpointImpl.NO_REENTRANT),29 check_fn=lambda m: isinstance(m, LlamaDecoderLayer),30)31opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)3233def train_step(input_ids): # input_ids: (micro_batch, seq) on this rank34 loss = model(input_ids=input_ids, labels=input_ids).loss # HF shifts labels internally35 loss.backward() # reduce-scatter of gradient shards36 model.clip_grad_norm_(1.0) # global norm across all shards37 opt.step() # each rank updates only its 1/d slice38 opt.zero_grad(set_to_none=True)39 return loss.detach()Use when
- +ZeRO/FSDP removes data-parallel memory redundancy at the same traffic for stages 1–2 (1.5× for stage 3)
- +Tensor parallelism shrinks weights and activations inside a node without changing the batch
- +Pipelines cross slow inter-node links with only point-to-point activation traffic
- +Recomputation trades extra compute (up to a third for full checkpointing) for far less activation memory: 5.7× (selective) to 97× (full) per layer in the worked example
Avoid when
- −Tensor parallelism puts four all-reduces per layer on the critical path; beyond one node it collapses
- −Pipeline bubbles cost (p − 1)/(m + p − 1) of the step unless the global batch is large
- −ZeRO-3 all-gathers every layer twice per step and stalls when the network cannot keep up
- −3D layouts multiply failure points: at 16K GPUs, hardware failures interrupt training often
Check yourself
A 10B-parameter model is trained with mixed-precision Adam and ZeRO-2 over 16 data-parallel GPUs (no tensor or pipeline parallelism). How much model-state memory does each GPU hold?
Check yourself
A pipeline has p = 8 stages. What is the smallest number of micro-batches m that brings the bubble (p − 1)/(m + p − 1) to 10% or less?
Remember
- ✓Mixed-precision Adam needs bytes (); activations add about per layer, or with sequence parallelism and selective recompute.
- ✓ZeRO shards over data parallelism: stage 1 , stage 2 , stage 3/FSDP , at 1× (stages 1–2) or 1.5× (stage 3) of DP traffic.
- ✓Megatron tensor parallelism splits A by columns and B by rows so GeLU stays local; four all-reduces per layer keep it inside a node (t ≤ 8).
- ✓Pipeline bubble ; 1F1B caps in-flight micro-batches at , interleaving divides the bubble by , and stage 0 still holds layers of activations.
- ✓3D layout follows bandwidth: TP on NVLink, PP across nodes, DP outermost; context parallelism splits long sequences; 40–55% MFU is good at scale.