Batch normalization

Normalize each channel with the mean and variance of the current mini-batch, then rescale with a learned γ and β. It allows larger learning rates and faster training, but it couples the examples in a batch, breaks down at small batch sizes and behaves differently in train and eval mode.

15 min read1588 wordscore0 viewsupdated 2026-10-11

Definition

Batch normalization (BatchNorm, BN) is a layer that standardizes each channel of its input using statistics of the current mini-batch: subtract the channel's batch mean, divide by its batch standard deviation, then apply a learned per-channel scale γ\gamma and shift β\beta. At inference it uses running averages of those statistics instead, so it becomes a fixed affine map.

Sergey Ioffe and Christian Szegedy introduced it in "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" (ICML 2015). Applied to an Inception network for ImageNet, BN matched the original model's accuracy in 14 times fewer training steps, tolerated much larger learning rates (their BN-x30 variant raised the initial rate 30-fold), let them drop dropout and cut L2 weight decay by a factor of 5, and an ensemble reached 4.9% top-5 validation error. Within a year it was in almost every convolutional network, including the Residual neural network (ResNet).

BN sits in this module because it does two jobs. Its main effect is on optimization: it keeps every layer's pre-activations at a controlled scale, so you can train deeper networks with higher learning rates. Its side effect is regularization: each example is normalized with statistics that depend on which other examples happen to share its batch, and that noise acts a little like dropout.

The forward pass

For a convolutional activation of shape (N,C,H,W)(N, C, H, W), BN treats each channel separately and pools over the batch and the spatial positions: one mean and one variance per channel, each computed from m=N⋅H⋅Wm=N\cdot H\cdot W numbers. For a fully connected layer the input is (N,C)(N, C) and m=Nm=N. Pooling over space is what makes the normalization convolutional: every position of a feature map is normalized the same way, so the layer stays translation-equivariant.

μc=1m∑n,h,wxnchw,σc2=1m∑n,h,w(xnchw−μc)2,x^nchw=xnchw−μcσc2+ϵ,ynchw=γc x^nchw+βc\mu_c=\frac1m\sum_{n,h,w}x_{nchw},\qquad \sigma_c^2=\frac1m\sum_{n,h,w}\big(x_{nchw}-\mu_c\big)^2,\qquad \hat x_{nchw}=\frac{x_{nchw}-\mu_c}{\sqrt{\sigma_c^2+\epsilon}},\qquad y_{nchw}=\gamma_c\,\hat x_{nchw}+\beta_c
mm
number of values pooled per channel: N⋅H⋅WN\cdot H\cdot W for BatchNorm2d, NN for BatchNorm1d on (N,C)(N, C)
μc, σc2\mu_c,\ \sigma_c^2
batch mean and biased batch variance (divide by mm) of channel cc
ϵ\epsilon
small constant for numerical safety, eps=1e-5 in PyTorch
γc, βc\gamma_c,\ \beta_c
learned scale and shift, 2 parameters per channel, initialised to 1 and 0

Why γ\gamma and β\beta? Forcing every channel to zero mean and unit variance would restrict what the layer can represent (a sigmoid fed unit-variance inputs stays mostly in its linear region). With γc=σc2+ϵ\gamma_c=\sqrt{\sigma_c^2+\epsilon} and βc=μc\beta_c=\mu_c the layer can undo the normalization exactly, so BN never removes capacity; it only changes the parametrisation so that the scale and offset of each channel are controlled by two dedicated parameters instead of by all the weights upstream.

BN is usually placed between the linear or convolutional layer and the nonlinearity, as in the original paper: ReLU(BN(Wx))\text{ReLU}(\text{BN}(Wx)). The layer's own bias is then redundant, since BN subtracts the mean and β\beta takes over the offset, so it is created with bias=False.

At a glance

Statistics over
(N,H,W)(N, H, W), separately for each channel cc
Learnable
γ,β∈RC\gamma, \beta \in \mathbb R^C: 2C2C parameters
Buffers
running_mean, running_var (length CC) and num_batches_tracked
Train mode
normalize with batch statistics, update the running averages
Eval mode
normalize with running statistics; a fixed per-channel affine map
Depends on batch size
yes: poor below roughly 8 examples per device for images
Typical home
CNNs (ResNet, VGG-BN, EfficientNet); rarely in transformers or RNNs

Running statistics: what eval mode uses

At test time you want a deterministic output that does not depend on the other images in the request. So during training BN keeps an exponential moving average of the batch statistics, and model.eval() switches to it. PyTorch's convention is:

μ^←(1−α) μ^+α μc,σ^2←(1−α) σ^2+α mm−1 σc2\hat\mu\leftarrow(1-\alpha)\,\hat\mu+\alpha\,\mu_c,\qquad \hat\sigma^2\leftarrow(1-\alpha)\,\hat\sigma^2+\alpha\,\frac{m}{m-1}\,\sigma_c^2
α\alpha
momentum, default 0.10.1: the weight of the new batch. Keras and TensorFlow use the opposite convention, momentum=0.99 meaning 0.990.99 on the old value
μ^, σ^2\hat\mu,\ \hat\sigma^2
running_mean and running_var, initialised to 0 and 1
mm−1\tfrac{m}{m-1}
Bessel's correction: the running variance uses the unbiased batch variance, while the normalization itself uses the biased one

With α=0.1\alpha=0.1 the average remembers about the last 1/α=101/\alpha=10 batches, and after kk batches it has moved a fraction 1−0.9k1-0.9^k of the way from its initial value: 65% after 10, 99.5% after 50. Setting momentum=None replaces the EMA by a plain cumulative average over all batches seen.

x[n, c, h, w] · N=8, C=4, H×W=3×3showing input xc0c1c2c3n0n1n2n3n4n5n6n7every row of the outlined columnpooled for c0: N·H·W = 8·9 = 72 valuesc0: μ_B 3.32 σ²_B 6.98 (true 3, 4)c1: μ_B −1.04 σ²_B 0.21 (true −1, 0.25)c2: μ_B 0.04 σ²_B 0.64 (true 0.5, 1)c3: μ_B 4.89 σ²_B 6.21 (true 6, 9)c0 mean: ● μ_B — running_mean ┄ true3c0 var: ● σ²_B — running_var ┄ true4step 80 / 80 · momentum 0.10running_mean 2.915 · running_var 3.993shown c0 values: mean 3.322, sd 2.641example n0: x̂ train −0.425 · eval −0.351RMS train−eval gap (steps > 20): 0.378σ²_B divides by m = 72; running_var uses m − 1 = 71
|
channel|H×W
fig.Mini-batches of shape (8, 4, 3, 3) streamed through one BN layer. The outlined column is everything channel c0 pools: N·H·W = 72 values. Dots are each batch's statistics, the line is the running average. After 80 batches at momentum 0.1, c0 has running_mean 2.92 and running_var 3.99 against true values 3 and 4. 'train x̂' shows each channel with mean 0 and sd 1 exactly; 'eval x̂' uses the running statistics, so c0 of the last batch comes out with mean 0.24 and sd 1.38. The bottom readout compares example n0 in the two modes: the RMS gap is 0.38 at N = 8, 0.27 at N = 16, 0.84 at N = 2 and 1.65 at N = 1. Within one image the values are correlated (70% of the variance is shared), so small batches also underestimate the variance: at N = 2, c3's running_var settles at 6.04 against a true 9. Momentum 0.5 tracks faster (running_mean 3.03 after 10 batches vs 1.84 at 0.1) but jitters three times as much. 1×1 maps with N = 1 is the case PyTorch refuses.

Worked example: one channel, four values

Take a BatchNorm1d(1) in training mode with γ=2\gamma=2, β=0.5\beta=0.5 and a batch x=(1,2,4,9)x=(1,2,4,9). Then m=4m=4, μ=4\mu=4, the biased variance is σ2=(9+4+0+25)/4=9.5\sigma^2=(9+4+0+25)/4=9.5, and σ=9.5+10−5=3.0822\sigma=\sqrt{9.5+10^{-5}}=3.0822.

xx − μx̂y = 2x̂ + 0.5∂L/∂x
1−3−0.9733−1.44670.3330
2−2−0.6489−0.7978−0.2647
400.00000.5000−0.1622
951.62223.74440.0939
Forward pass, and the backward pass for an upstream gradient that touches only the first output: g = ∂L/∂y = (1, 0, 0, 0). Checked in Node, including a finite-difference check of ∂L/∂x.

What the numbers say

  1. 1The outputs x^\hat x have mean 0 and biased variance 1 (up to ϵ\epsilon), whatever the input scale was.
  2. 2Running stats after this one step: μ^=0.9⋅0+0.1⋅4=0.4\hat\mu=0.9\cdot0+0.1\cdot4=0.4 and σ^2=0.9⋅1+0.1⋅383=2.1667\hat\sigma^2=0.9\cdot1+0.1\cdot\tfrac{38}{3}=2.1667 (unbiased variance 12.66712.667, not 9.59.5).
  3. 3Eval mode right now would map x=9x=9 to 2(9−0.4)/2.1667+0.5=12.192(9-0.4)/\sqrt{2.1667}+0.5=12.19 instead of the training-mode 3.743.74: the running averages need many batches before eval mode matches training.
  4. 4The gradient reaches all four inputs although only y1y_1 was touched, because every input moved μ\mu and σ\sigma. Treating μ,σ\mu,\sigma as constants would give γ/σ=0.6489\gamma/\sigma=0.6489 for x1x_1 and 0 elsewhere, which is wrong.
  5. 5∑i∂L/∂xi=0\sum_i\partial L/\partial x_i=0 and ∑ix^i ∂L/∂xi=0\sum_i \hat x_i\,\partial L/\partial x_i=0: shifting or scaling the whole batch cannot change BN's output, so the gradient has no component in those directions. Also ∂L/∂γ=∑gix^i=−0.9733\partial L/\partial\gamma=\sum g_i\hat x_i=-0.9733 and ∂L/∂β=∑gi=1\partial L/\partial\beta=\sum g_i=1.

The backward pass

Writing gi=∂L/∂yig_i=\partial L/\partial y_i for the values of one channel and differentiating through μ\mu and σ2\sigma^2 as well as through xix_i directly gives a compact formula:

∂L∂xi=γm σ(m gi−∑jgj−x^i∑jgjx^j),∂L∂γ=∑igix^i,∂L∂β=∑igi\frac{\partial L}{\partial x_i}=\frac{\gamma}{m\,\sigma}\Big(m\,g_i-\sum_{j}g_j-\hat x_i\sum_{j}g_j\hat x_j\Big),\qquad \frac{\partial L}{\partial\gamma}=\sum_i g_i\hat x_i,\qquad \frac{\partial L}{\partial\beta}=\sum_i g_i
σ\sigma
σc2+ϵ\sqrt{\sigma_c^2+\epsilon}, the batch standard deviation used in the forward pass
∑jgj\sum_j g_j
removes the mean of the upstream gradient: the term that flows through μ\mu
x^i∑jgjx^j\hat x_i\sum_j g_j\hat x_j
removes the component along x^\hat x: the term that flows through σ2\sigma^2

The bracket is the upstream gradient with its mean and its projection on x^\hat x removed, then scaled by γ/σ\gamma/\sigma. Two consequences matter in practice. First, the gradient of one example depends on every other example in the batch, so BN is not a per-example layer: per-example gradients (for differential privacy, say) and Gradient accumulation over micro-batches do not see the same function as a single large batch. Second, because BN(a Wx)=BN(Wx)\text{BN}(a\,Wx)=\text{BN}(Wx) for any a>0a>0, the loss is invariant to the scale of the weights feeding a BN layer, and the gradient with respect to those weights is orthogonal to them and shrinks like 1/∥W∥1/\lVert W\rVert.

Why does it help? Two explanations

Ioffe and Szegedy's motivation was internal covariate shift: as earlier layers train, the distribution of each layer's inputs keeps changing, so later layers chase a moving target. Fixing the first two moments of every layer's inputs was supposed to stabilise that.

Santurkar, Tsipras, Ilyas and Madry tested this in "How Does Batch Normalization Help Optimization?" (NeurIPS 2018). They added random, time-varying noise with non-zero mean and non-unit variance after each BN layer, deliberately creating covariate shift; the network still trained almost as well as standard BN and far better than no BN. They also found that BN did not reliably reduce covariate shift by their measures. Their explanation is that BN makes the loss landscape smoother: the loss and its gradient change more slowly as the weights move (better Lipschitz constants and β-smoothness), so gradients are more predictive and larger steps are safe.

When batch statistics break down

Every mechanism above relies on μc\mu_c and σc2\sigma_c^2 being decent estimates. With few examples per batch they are not, and three things go wrong. The noise in each example's normalized value grows, which is the RMS train/eval gap in the figure. The running variance is biased low when values within one image are correlated. And the network learns to rely on statistics that eval mode will not reproduce. Wu and He's Group Normalization paper (ECCV 2018) measured the effect: on ImageNet ResNet-50, BN's error rises steeply as the per-GPU batch shrinks, and at 2 images per GPU Group Normalization's error was 10.6% lower than BN's.

The extreme cases are degenerate. With one value per channel (N=1N=1 for BatchNorm1d, or N=1N=1 and 1×11\times1 maps), the variance is 0 and every output equals β\beta; PyTorch refuses this in training mode with "Expected more than 1 value per channel when training". With N=1N=1 and larger maps, BN normalizes an image only against itself, which is Instance normalization in disguise, and eval mode then sees statistics it was never trained with.

Where it's used

SyncBatchNorm

Under data parallelism each GPU normally computes statistics over its own shard. nn.SyncBatchNorm all-reduces the sums across processes so the statistics cover the global batch; it costs one extra communication per BN layer per step. Used for detection and segmentation, where each GPU holds only 1 or 2 images.

Group or layer normalization

Group normalization, Layer normalization and Instance normalization compute statistics per example, so they are independent of batch size and identical in train and eval. They are the standard replacements when batches are small or variable.

Batch Renormalization

Ioffe (2017) corrects the batch statistics toward the running ones with clipped factors r,dr,d that are treated as constants in the backward pass, which reduces the train/eval mismatch for small or non-i.i.d. batches.

Frozen BN for fine-tuning

When fine-tuning a pretrained CNN with tiny batches, keep the BN layers in eval mode (and often freeze γ,β\gamma,\beta) so the pretrained running statistics are used throughout.

Inference: fold BN into the convolution

In eval mode BN is y=γ(z−μ^)/σ^2+ϵ+βy=\gamma(z-\hat\mu)/\sqrt{\hat\sigma^2+\epsilon}+\beta applied to a convolution output z=w∗x+bz=w*x+b. Both steps are affine, so they collapse into one convolution with per-output-channel rescaled weights. Deployment toolchains do this automatically; the BN layer then costs nothing at inference.

wc′=γcσ^c2+ϵ wc,bc′=γc (bc−μ^c)σ^c2+ϵ+βcw'_c=\frac{\gamma_c}{\sqrt{\hat\sigma_c^2+\epsilon}}\,w_c,\qquad b'_c=\frac{\gamma_c\,(b_c-\hat\mu_c)}{\sqrt{\hat\sigma_c^2+\epsilon}}+\beta_c
wc, bcw_c,\ b_c
the filter and bias of output channel cc (bc=0b_c=0 for a bias-free conv)
wc′, bc′w'_c,\ b'_c
the fused filter and bias: one conv that equals conv followed by eval-mode BN

Example: a 1×1 conv with w=0.5w=0.5, b=0.2b=0.2, followed by BN with μ^=4\hat\mu=4, σ^2=9.5\hat\sigma^2=9.5, γ=2\gamma=2, β=0.5\beta=0.5, fuses to w′=0.3244w'=0.3244, b′=−1.9658b'=-1.9658. For input 3 both the two-layer and the fused version output −0.9924-0.9924.

python
1import copy, torch, torch.nn as nn
2
3torch.manual_seed(0)
4x = torch.randn(8, 4, 3, 3) * 2 + 3 # (N, C, H, W)
5bn = nn.BatchNorm2d(4) # eps=1e-5, momentum=0.1, affine=True
6
7# --- training mode: batch statistics over (N, H, W) ---
8y = bn(x)
9mu = x.mean(dim=(0, 2, 3))
10var = x.var(dim=(0, 2, 3), correction=0) # biased: used to normalize
11xhat = (x - mu[None, :, None, None]) / torch.sqrt(var[None, :, None, None] + bn.eps)
12print(torch.allclose(y, xhat * bn.weight[None, :, None, None] + bn.bias[None, :, None, None], atol=1e-5))
13
14# running stats: start at 0 and 1, momentum weighs the NEW batch, variance is unbiased
15m = x.shape[0] * x.shape[2] * x.shape[3]
16print(torch.allclose(bn.running_mean, 0.1 * mu))
17print(torch.allclose(bn.running_var, 0.9 * 1 + 0.1 * var * m / (m - 1)))
18
19# --- eval mode: running statistics, outputs no longer depend on the batch ---
20bn.eval()
21print(torch.allclose(bn(x[:1]), bn(x)[:1])) # True only in eval mode
22
23# BatchNorm1d with one value per channel cannot train
24try:
25 nn.BatchNorm1d(4).train()(torch.randn(1, 4))
26except ValueError as e:
27 print(e) # Expected more than 1 value per channel when training ...
28
29# --- typical block: no conv bias, BN before the nonlinearity ---
30block = nn.Sequential(nn.Conv2d(3, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU())
31
32# multi-GPU: share statistics across processes (use with DistributedDataParallel)
33sync_block = nn.SyncBatchNorm.convert_sync_batchnorm(copy.deepcopy(block))
34
35# --- inference: fold eval-mode BN into the conv ---
36from torch.nn.utils.fusion import fuse_conv_bn_eval
37block.eval()
38fused = fuse_conv_bn_eval(block[0], block[1])
39img = torch.randn(2, 3, 32, 32)
40print(torch.allclose(fused(img), block[1](block[0](img)), atol=1e-5))
BatchNorm2d by hand, the running-stat convention, eval mode, SyncBatchNorm and conv-BN fusion

Use when

  • +Allows much higher learning rates and trains deep CNNs several times faster
  • +Makes training far less sensitive to initialization and weight scale
  • +Free at inference: folds into the preceding convolution
  • +Mild regularization from batch-statistic noise; often lets you drop dropout

Avoid when

  • −Output of one example depends on the rest of its batch; not a per-example function in training
  • −Degrades at small per-device batch sizes (detection, segmentation, video, large models)
  • −Two modes: train/eval mismatch, running statistics that can go stale or be overwritten
  • −Awkward for RNNs and variable-length sequences; needs cross-device sync under data parallelism

Check yourself

A conv layer outputs a tensor of shape (16, 64, 28, 28) into nn.BatchNorm2d(64). Over how many values is each mean computed, and how many mean values are there?

Check yourself

Your BN model reaches 92% accuracy when evaluated in train mode on the validation set, but 61% after model.eval(). Which is the most likely cause?

Remember

  • ✓BN standardizes each channel with batch statistics over (N,H,W)(N,H,W), then applies a learned γc,βc\gamma_c,\beta_c; it can represent the identity, so no capacity is lost.
  • ✓Running stats: μ^←0.9μ^+0.1μB\hat\mu\leftarrow0.9\hat\mu+0.1\mu_B in PyTorch, with the unbiased variance; eval mode uses them and becomes a fixed affine map that folds into the preceding conv.
  • ✓Backward: ∂L/∂x=γmσ(mg−∑g−x^∑gx^)\partial L/\partial x=\tfrac{\gamma}{m\sigma}\big(mg-\sum g-\hat x\sum g\hat x\big); gradients couple all examples in the batch.
  • ✓It works mostly by smoothing optimization (Santurkar et al., 2018) rather than by removing internal covariate shift.
  • ✓Small per-device batches break it: use SyncBatchNorm, or Group, Layer or Instance normalization.
  • ✓Call model.eval() for inference, avoid dropout before BN, and remember weight decay on BN-fed weights acts as a learning-rate change.
← back to Deep Learning roadmap