Definition
Gradient accumulation splits one optimizer step over forward/backward passes. Each micro-batch of examples adds its gradient into the same buffer, and optimizer.step() runs once after the -th, on an effective batch of examples.
Memory on an accelerator is dominated by activations saved for the backward pass, and they grow linearly with the batch. When the batch a recipe needs (thousands of sequences for a language model, 256 images for a ResNet schedule) does not fit, accumulation trades time for memory: passes of examples cost the same compute as one pass of , but only one micro-batch's activations are alive at a time.
The trick works because the gradient of a sum is the sum of the gradients, and PyTorch's backward() adds into .grad instead of overwriting it. It has been standard practice for years; Ott et al. (2018) used it in fairseq ("delayed updates") to train translation models with the batch of many more GPUs than they had, and Hugging Face's Trainer and Accelerate expose it as a single setting.
Why the gradients are identical
- the loss of example i
- micro-batch , with examples; the partition the big batch
- what a loss with
reduction="mean"returns for one micro-batch - the factor you must apply yourself:
(loss / K).backward()
A mean over is the average of means over , provided every micro-batch has the same size. So if each micro-batch's mean loss is divided by before backward(), the buffer ends up holding exactly the big-batch gradient, and the optimizer cannot tell the difference. Only floating-point summation order changes, at the level of in float64 or in float32.
At a glance
- Effective batch
- per device, times the number of data-parallel devices
- Memory
- activations of one micro-batch; weights, gradients and optimizer state are unchanged
- Compute
- the same FLOPs as the big batch, but sequential passes, so no faster; small micro-batches may use the GPU less efficiently
- Exact when
- the loss is a mean over equally sized micro-batches and no layer couples examples (BatchNorm does)
- Counts as
- one optimizer step: schedulers, clipping, logging and
zero_gradhappen once per micro-batches
The orange dots are worth a look too. Each micro-batch gradient is a noisy estimate of the big-batch gradient, and the big-batch gradient is their average. Accumulation does not change that average; it only computes it in pieces.
Worked example: four examples, K = 2
Model at , squared error, so . The big batch is , split into micro-batches and .
| loop | micro A adds | micro B adds | .grad at step() | equals |
|---|---|---|---|---|
| one batch of 4 | – | – | reference | |
| accumulate, loss / 2 | reference ✓ | |||
| accumulate, forgot / 2 | : lr doubled | |||
| zero_grad before each micro | , then erased | half of B only |
The micro-batch means are and ; their average is the reference .
Where accumulation stops being exact
Everything else that runs per forward pass is unaffected in expectation: dropout draws independent masks per example anyway, and data augmentation is per example.
Writing the loop
One optimizer step with accumulation
- 1
optimizer.zero_grad()once, before the first micro-batch of the step (or right after the previousstep()). - 2For each of the micro-batches: forward,
loss = loss / K,loss.backward(). Gradients add up in.grad. - 3After the -th backward only: unscale (fp16 AMP), clip the gradient norm,
optimizer.step(),scheduler.step(), thenzero_grad(). - 4Count steps in optimizer steps. An epoch of micro-batches has of them; size warmup and decay schedules with that number, not .
1import contextlib, torch2from torch.nn.parallel import DistributedDataParallel as DDP34K = 8 # accumulation steps5model = DDP(model) # effective batch = micro_bs * K * world_size6optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)7steps_per_epoch = -(-len(loader) // K) # ceil: optimizer steps, not micro-batches8scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=3e-4,9 total_steps=epochs * steps_per_epoch)1011optimizer.zero_grad(set_to_none=True)12for i, (x, y) in enumerate(loader):13 boundary = (i + 1) % K == 0 or (i + 1) == len(loader)14 # no_sync(): skip the gradient all-reduce on all but the last micro-batch15 sync = contextlib.nullcontext() if boundary else model.no_sync()16 with sync:17 with torch.autocast(device_type="cuda", dtype=torch.bfloat16):18 loss = loss_fn(model(x), y)19 (loss / K).backward() # .grad += grad(loss) / K20 if boundary:21 torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # once per step22 optimizer.step()23 scheduler.step() # once per OPTIMIZER step24 optimizer.zero_grad(set_to_none=True)25# A short final group (len(loader) not divisible by K) is still divided by K,26# so that step is smaller; drop it or divide by its real size instead.Use when
- +Train with the batch size a recipe needs on hardware that cannot hold it
- +Mathematically the same update as the big batch for mean losses without BatchNorm
- +Two extra lines in the loop; frameworks expose it as one setting
- +With no_sync, communication per optimizer step drops by a factor of K
Avoid when
- −No speed-up: K sequential passes, and tiny micro-batches can under-use the GPU
- −BatchNorm statistics come from the micro-batch, not the effective batch
- −Silent bugs: missing / K (acts like lr × K), zero_grad inside the loop, schedulers stepped per micro-batch
- −Token-averaged losses need normalization by the total token count of the window
Check yourself
You accumulate over K = 4 micro-batches but call loss.backward() without dividing by K. Using SGD, what is the effect?
Check yourself
An epoch has 1000 micro-batches and you accumulate over K = 8. How many times should a per-step learning-rate scheduler (e.g. cosine decay) be stepped per epoch?
Remember
- ✓Effective batch micro-batch (× data-parallel devices); memory is that of one micro-batch.
- ✓Divide each micro-batch loss by : then
.gradequals the big-batch gradient exactly. Forgetting it multiplies the learning rate by . - ✓
zero_gradonce per optimizer step, not per micro-batch; step the optimizer, clip, and step the scheduler only at the boundary. - ✓Exactness needs equal-size micro-batches and per-example layers: normalize token losses by the window's total tokens, and expect BatchNorm to see only the micro-batch.
- ✓Under DDP wrap all but the last micro-batch in
model.no_sync()to avoid redundant all-reduces. - ✓It reproduces a large batch; it does not make training faster or a large batch better.