Epoch, batch, mini-batch & iteration

An iteration is one update from one mini-batch; an epoch is one full pass over the data, so iterations per epoch = ⌈N / batch size⌉.

7 min read575 wordscore0 viewsupdated 2026-10-09

Definition

Epoch, mini-batch, iteration A mini-batch is a group of BB training examples whose average gradient produces one parameter update. That update is one iteration (also called a step). An epoch is one full pass in which every one of the NN training examples has been used once, which takes ⌈N/B⌉\lceil N/B\rceil iterations.

Why not use all the data for every update?

The training loss is an average over NN examples, and so is its gradient. Computing the exact gradient means a forward and backward pass over the whole dataset (for ImageNet, 1.28 million images) just to take one step.

A random subset of BB examples gives an unbiased estimate of that gradient at a fraction of the cost. The estimate is noisy, but you get N/BN/B updates for the price of one exact one, and early in training many rough steps beat one precise step.

VariantBatch sizeUpdates per epochGradient
Batch (full-batch) gradient descentB=NB=N1exact, deterministic
Stochastic gradient descent (SGD)B=1B=1NNnoisiest estimate, no parallelism
Mini-batch gradient descent1<B<N1<B<N (typically 32–1024)⌈N/B⌉\lceil N/B\rceilnoisy, and vectorizes well on a GPU
Three names for the same loop, differing only in B

In practice everyone uses mini-batches and still calls it "SGD": torch.optim.SGD simply updates with whatever batch you hand it.

The math

θt+1=θt−η g^t,g^t=1∣Bt∣∑i∈Bt∇θ ℓ(xi,yi;θt)\theta_{t+1}=\theta_t-\eta\,\hat g_t,\qquad \hat g_t=\frac{1}{|\mathcal B_t|}\sum_{i\in\mathcal B_t}\nabla_\theta\,\ell(x_i,y_i;\theta_t)
tt
iteration counter (one per update, never reset between epochs)
Bt\mathcal B_t
the mini-batch used at iteration tt; ∣Bt∣=B|\mathcal B_t|=B except possibly the last batch of an epoch
η\eta
learning rate
ℓ\ell
per-example loss
g^t\hat g_t
mini-batch gradient, an unbiased estimate of the full gradient
iterations per epoch=⌈NB⌉    or    ⌊NB⌋ with drop_last,total iterations=E⋅iterations per epoch\text{iterations per epoch}=\left\lceil \frac{N}{B}\right\rceil\;\;\text{or}\;\;\left\lfloor \frac{N}{B}\right\rfloor\text{ with drop\_last},\qquad \text{total iterations}=E\cdot\text{iterations per epoch}
NN
number of training examples
BB
batch size
EE
number of epochs

How noisy is g^t\hat g_t? If individual gradients have variance σ2\sigma^2 around the true gradient, then averaging BB of them drawn without replacement gives

Var⁡[g^]=σ2B⋅N−BN−1⇒std⁡[g^]≈σB    (B≪N)\operatorname{Var}[\hat g]=\frac{\sigma^2}{B}\cdot\frac{N-B}{N-1}\quad\Rightarrow\quad \operatorname{std}[\hat g]\approx\frac{\sigma}{\sqrt B}\;\;(B\ll N)
σ2\sigma^2
variance of a single example's gradient (summed over parameters)
N−BN−1\frac{N-B}{N-1}
finite-population correction; it is 0 at B=NB=N, where the gradient is exact

So quadrupling the batch only halves the noise while costing four times the compute per step. That square root is why huge batches give diminishing returns per example processed.

N = 60 points · colour = mini-batch in this epoch · ring = last update's batch431224312433233141121124133221324314222443131234424131323112epoch 1 order (reshuffled every epoch): 4 mini-batches, last one has 12width ∝ batch sizeloss: mini-batch (as logged) · full data · B = 60 dashed012345600.40.81.2epochs (1 epoch = 4 iterations) →
|
⌈60/16⌉ = 4 iterations per epoch · 6 epochs = 24 updates
epoch 0/6 · iteration 0/4 · global step 0/24
last mini-batch loss — · full-data loss 1.106
gradient noise std — · vs B=1: ×0.22 (≈ 1/√B)
fig.Mini-batch SGD on a 60-point regression. At B = 16 an epoch is 4 iterations and the last batch has 12 points. Press play: one batch lights up per update, and the colours reshuffle at each epoch boundary. Then compare B = 1 (60 noisy updates per epoch, loss about 0.18 after one epoch) with B = 60 (one smooth update per epoch, loss still 0.78). Finally set B = 7 and turn on drop_last, which drops 4 points every epoch.

Worked example: N = 50,000 (CIFAR-10's training set)

BIterations / epochLast batchWith drop_lastIterations in 100 epochsRelative noise
150,000150,0005,000,0001
321,563161,562156,3000.177
1283918039039,1000.088
51298336979,8000.044
50,000150,00011000
Relative noise = std of the mini-batch gradient divided by the B = 1 value, sqrt((N−B)/((N−1)B))

The B = 128 row

  1. 150,000/128=390.62550{,}000/128=390.625, so 390 full batches cover 390⋅128=49,920390\cdot128=49{,}920 examples.
  2. 2The remaining 50,000−49,920=8050{,}000-49{,}920=80 examples form a 391st, smaller batch: ⌈390.625⌉=391\lceil 390.625\rceil=391 iterations per epoch.
  3. 3With drop_last=True those 80 are skipped each epoch (a different 80 each time, because of shuffling), leaving 390 iterations.
  4. 4A 100-epoch run is 100⋅391=39,100100\cdot391=39{,}100 iterations. Learning-rate schedules and logging are usually counted in these iterations, not in epochs.

Shuffling and the last batch

Reshuffle every epoch. If the order is fixed, every epoch sees identical batches in the same sequence, and any structure in that order (for example, data sorted by class) biases the updates. A fresh permutation each epoch keeps g^t\hat g_t close to an independent random sample.

The partial batch still averages its loss, so the update has the usual scale, just more noise. It is harmless for most models. Drop it when every batch must have the same shape (some compiled or distributed setups) or when a layer misbehaves on tiny batches: BatchNorm in training mode raises an error on a batch of one sample.

Choosing B: memory versus time

Use when

  • +Larger B: better GPU utilization, fewer iterations per epoch, smoother gradient estimate, so faster wall-clock time per epoch.
  • +Activation memory grows roughly linearly in BB, so the largest batch that fits is bounded by GPU memory. Weights and optimizer state do not depend on BB.

Avoid when

  • −Per-example returns shrink: noise falls only as 1/B1/\sqrt B, so past some size, doubling BB barely reduces the number of updates needed.
  • −Very large batches usually need a retuned learning rate (a common heuristic scales η\eta in proportion to BB, with warm-up) and can generalize slightly worse without it.

If the batch you want does not fit, use gradient accumulation: run kk smaller micro-batches, add up their gradients, and call optimizer.step() once. That is one iteration with an effective batch of k×k\times the micro-batch size.

Where the counters live in PyTorch

python
1import torch
2from torch.utils.data import DataLoader, TensorDataset
3
4X, y = torch.randn(50_000, 20), torch.randint(0, 10, (50_000,))
5loader = DataLoader(TensorDataset(X, y), batch_size=128, shuffle=True, drop_last=False)
6print(len(loader)) # 391 = ceil(50000 / 128)
7
8model = torch.nn.Linear(20, 10)
9opt = torch.optim.SGD(model.parameters(), lr=0.1)
10loss_fn = torch.nn.CrossEntropyLoss()
11
12step = 0 # global iteration counter
13for epoch in range(10): # one epoch = one full pass over the loader
14 for xb, yb in loader: # new permutation each time this loop starts
15 loss = loss_fn(model(xb), yb) # mean loss over this mini-batch
16 opt.zero_grad()
17 loss.backward()
18 opt.step() # one iteration = one parameter update
19 step += 1
20 print(epoch, step) # step == (epoch + 1) * 391
len(loader) is the number of iterations per epoch; the outer loop counts epochs, the inner loop counts iterations.

Check yourself

N=10,000N=10{,}000, B=64B=64, drop_last=False, 5 epochs. How many parameter updates happen?

Check yourself

You raise the batch size from 32 to 512 (N is much larger than both). By roughly what factor does the standard deviation of the mini-batch gradient shrink?

Remember

  • ✓Iteration = one mini-batch, one update. Epoch = every training example used once.
  • ✓Iterations per epoch =⌈N/B⌉=\lceil N/B\rceil (or ⌊N/B⌋\lfloor N/B\rfloor with drop_last); for 50,000 examples at B=128B=128 that is 391.
  • ✓Batch GD is B=NB=N, SGD is B=1B=1; mini-batch sits in between and is what everyone actually uses.
  • ✓Gradient noise falls as 1/B1/\sqrt B; activation memory grows linearly in BB.
  • ✓Reshuffle every epoch, and compare runs with different B by iterations or examples seen, not by epochs.
← back to Deep Learning roadmap