Definition
Batch normalization (BatchNorm, BN) is a layer that standardizes each channel of its input using statistics of the current mini-batch: subtract the channel's batch mean, divide by its batch standard deviation, then apply a learned per-channel scale and shift . At inference it uses running averages of those statistics instead, so it becomes a fixed affine map.
Sergey Ioffe and Christian Szegedy introduced it in "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" (ICML 2015). Applied to an Inception network for ImageNet, BN matched the original model's accuracy in 14 times fewer training steps, tolerated much larger learning rates (their BN-x30 variant raised the initial rate 30-fold), let them drop dropout and cut L2 weight decay by a factor of 5, and an ensemble reached 4.9% top-5 validation error. Within a year it was in almost every convolutional network, including the Residual neural network (ResNet).
BN sits in this module because it does two jobs. Its main effect is on optimization: it keeps every layer's pre-activations at a controlled scale, so you can train deeper networks with higher learning rates. Its side effect is regularization: each example is normalized with statistics that depend on which other examples happen to share its batch, and that noise acts a little like dropout.
The forward pass
For a convolutional activation of shape , BN treats each channel separately and pools over the batch and the spatial positions: one mean and one variance per channel, each computed from numbers. For a fully connected layer the input is and . Pooling over space is what makes the normalization convolutional: every position of a feature map is normalized the same way, so the layer stays translation-equivariant.
- number of values pooled per channel: for
BatchNorm2d, forBatchNorm1don - batch mean and biased batch variance (divide by ) of channel
- small constant for numerical safety,
eps=1e-5in PyTorch - learned scale and shift, 2 parameters per channel, initialised to 1 and 0
Why and ? Forcing every channel to zero mean and unit variance would restrict what the layer can represent (a sigmoid fed unit-variance inputs stays mostly in its linear region). With and the layer can undo the normalization exactly, so BN never removes capacity; it only changes the parametrisation so that the scale and offset of each channel are controlled by two dedicated parameters instead of by all the weights upstream.
BN is usually placed between the linear or convolutional layer and the nonlinearity, as in the original paper: . The layer's own bias is then redundant, since BN subtracts the mean and takes over the offset, so it is created with bias=False.
At a glance
- Statistics over
- , separately for each channel
- Learnable
- : parameters
- Buffers
running_mean,running_var(length ) andnum_batches_tracked- Train mode
- normalize with batch statistics, update the running averages
- Eval mode
- normalize with running statistics; a fixed per-channel affine map
- Depends on batch size
- yes: poor below roughly 8 examples per device for images
- Typical home
- CNNs (ResNet, VGG-BN, EfficientNet); rarely in transformers or RNNs
Running statistics: what eval mode uses
At test time you want a deterministic output that does not depend on the other images in the request. So during training BN keeps an exponential moving average of the batch statistics, and model.eval() switches to it. PyTorch's convention is:
momentum, default : the weight of the new batch. Keras and TensorFlow use the opposite convention,momentum=0.99meaning on the old valuerunning_meanandrunning_var, initialised to 0 and 1- Bessel's correction: the running variance uses the unbiased batch variance, while the normalization itself uses the biased one
With the average remembers about the last batches, and after batches it has moved a fraction of the way from its initial value: 65% after 10, 99.5% after 50. Setting momentum=None replaces the EMA by a plain cumulative average over all batches seen.
Worked example: one channel, four values
Take a BatchNorm1d(1) in training mode with , and a batch . Then , , the biased variance is , and .
| x | x − μ | x̂ | y = 2x̂ + 0.5 | ∂L/∂x |
|---|---|---|---|---|
| 1 | −3 | −0.9733 | −1.4467 | 0.3330 |
| 2 | −2 | −0.6489 | −0.7978 | −0.2647 |
| 4 | 0 | 0.0000 | 0.5000 | −0.1622 |
| 9 | 5 | 1.6222 | 3.7444 | 0.0939 |
What the numbers say
- 1The outputs have mean 0 and biased variance 1 (up to ), whatever the input scale was.
- 2Running stats after this one step: and (unbiased variance , not ).
- 3Eval mode right now would map to instead of the training-mode : the running averages need many batches before eval mode matches training.
- 4The gradient reaches all four inputs although only was touched, because every input moved and . Treating as constants would give for and 0 elsewhere, which is wrong.
- 5 and : shifting or scaling the whole batch cannot change BN's output, so the gradient has no component in those directions. Also and .
The backward pass
Writing for the values of one channel and differentiating through and as well as through directly gives a compact formula:
- , the batch standard deviation used in the forward pass
- removes the mean of the upstream gradient: the term that flows through
- removes the component along : the term that flows through
The bracket is the upstream gradient with its mean and its projection on removed, then scaled by . Two consequences matter in practice. First, the gradient of one example depends on every other example in the batch, so BN is not a per-example layer: per-example gradients (for differential privacy, say) and Gradient accumulation over micro-batches do not see the same function as a single large batch. Second, because for any , the loss is invariant to the scale of the weights feeding a BN layer, and the gradient with respect to those weights is orthogonal to them and shrinks like .
Why does it help? Two explanations
Ioffe and Szegedy's motivation was internal covariate shift: as earlier layers train, the distribution of each layer's inputs keeps changing, so later layers chase a moving target. Fixing the first two moments of every layer's inputs was supposed to stabilise that.
Santurkar, Tsipras, Ilyas and Madry tested this in "How Does Batch Normalization Help Optimization?" (NeurIPS 2018). They added random, time-varying noise with non-zero mean and non-unit variance after each BN layer, deliberately creating covariate shift; the network still trained almost as well as standard BN and far better than no BN. They also found that BN did not reliably reduce covariate shift by their measures. Their explanation is that BN makes the loss landscape smoother: the loss and its gradient change more slowly as the weights move (better Lipschitz constants and β-smoothness), so gradients are more predictive and larger steps are safe.
When batch statistics break down
Every mechanism above relies on and being decent estimates. With few examples per batch they are not, and three things go wrong. The noise in each example's normalized value grows, which is the RMS train/eval gap in the figure. The running variance is biased low when values within one image are correlated. And the network learns to rely on statistics that eval mode will not reproduce. Wu and He's Group Normalization paper (ECCV 2018) measured the effect: on ImageNet ResNet-50, BN's error rises steeply as the per-GPU batch shrinks, and at 2 images per GPU Group Normalization's error was 10.6% lower than BN's.
The extreme cases are degenerate. With one value per channel ( for BatchNorm1d, or and maps), the variance is 0 and every output equals ; PyTorch refuses this in training mode with "Expected more than 1 value per channel when training". With and larger maps, BN normalizes an image only against itself, which is Instance normalization in disguise, and eval mode then sees statistics it was never trained with.
Where it's used
SyncBatchNorm
Under data parallelism each GPU normally computes statistics over its own shard. nn.SyncBatchNorm all-reduces the sums across processes so the statistics cover the global batch; it costs one extra communication per BN layer per step. Used for detection and segmentation, where each GPU holds only 1 or 2 images.
Group or layer normalization
Group normalization, Layer normalization and Instance normalization compute statistics per example, so they are independent of batch size and identical in train and eval. They are the standard replacements when batches are small or variable.
Batch Renormalization
Ioffe (2017) corrects the batch statistics toward the running ones with clipped factors that are treated as constants in the backward pass, which reduces the train/eval mismatch for small or non-i.i.d. batches.
Frozen BN for fine-tuning
When fine-tuning a pretrained CNN with tiny batches, keep the BN layers in eval mode (and often freeze ) so the pretrained running statistics are used throughout.
Inference: fold BN into the convolution
In eval mode BN is applied to a convolution output . Both steps are affine, so they collapse into one convolution with per-output-channel rescaled weights. Deployment toolchains do this automatically; the BN layer then costs nothing at inference.
- the filter and bias of output channel ( for a bias-free conv)
- the fused filter and bias: one conv that equals conv followed by eval-mode BN
Example: a 1×1 conv with , , followed by BN with , , , , fuses to , . For input 3 both the two-layer and the fused version output .
1import copy, torch, torch.nn as nn23torch.manual_seed(0)4x = torch.randn(8, 4, 3, 3) * 2 + 3 # (N, C, H, W)5bn = nn.BatchNorm2d(4) # eps=1e-5, momentum=0.1, affine=True67# --- training mode: batch statistics over (N, H, W) ---8y = bn(x)9mu = x.mean(dim=(0, 2, 3))10var = x.var(dim=(0, 2, 3), correction=0) # biased: used to normalize11xhat = (x - mu[None, :, None, None]) / torch.sqrt(var[None, :, None, None] + bn.eps)12print(torch.allclose(y, xhat * bn.weight[None, :, None, None] + bn.bias[None, :, None, None], atol=1e-5))1314# running stats: start at 0 and 1, momentum weighs the NEW batch, variance is unbiased15m = x.shape[0] * x.shape[2] * x.shape[3]16print(torch.allclose(bn.running_mean, 0.1 * mu))17print(torch.allclose(bn.running_var, 0.9 * 1 + 0.1 * var * m / (m - 1)))1819# --- eval mode: running statistics, outputs no longer depend on the batch ---20bn.eval()21print(torch.allclose(bn(x[:1]), bn(x)[:1])) # True only in eval mode2223# BatchNorm1d with one value per channel cannot train24try:25 nn.BatchNorm1d(4).train()(torch.randn(1, 4))26except ValueError as e:27 print(e) # Expected more than 1 value per channel when training ...2829# --- typical block: no conv bias, BN before the nonlinearity ---30block = nn.Sequential(nn.Conv2d(3, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU())3132# multi-GPU: share statistics across processes (use with DistributedDataParallel)33sync_block = nn.SyncBatchNorm.convert_sync_batchnorm(copy.deepcopy(block))3435# --- inference: fold eval-mode BN into the conv ---36from torch.nn.utils.fusion import fuse_conv_bn_eval37block.eval()38fused = fuse_conv_bn_eval(block[0], block[1])39img = torch.randn(2, 3, 32, 32)40print(torch.allclose(fused(img), block[1](block[0](img)), atol=1e-5))Use when
- +Allows much higher learning rates and trains deep CNNs several times faster
- +Makes training far less sensitive to initialization and weight scale
- +Free at inference: folds into the preceding convolution
- +Mild regularization from batch-statistic noise; often lets you drop dropout
Avoid when
- −Output of one example depends on the rest of its batch; not a per-example function in training
- −Degrades at small per-device batch sizes (detection, segmentation, video, large models)
- −Two modes: train/eval mismatch, running statistics that can go stale or be overwritten
- −Awkward for RNNs and variable-length sequences; needs cross-device sync under data parallelism
Check yourself
A conv layer outputs a tensor of shape (16, 64, 28, 28) into nn.BatchNorm2d(64). Over how many values is each mean computed, and how many mean values are there?
Check yourself
Your BN model reaches 92% accuracy when evaluated in train mode on the validation set, but 61% after model.eval(). Which is the most likely cause?
Remember
- ✓BN standardizes each channel with batch statistics over , then applies a learned ; it can represent the identity, so no capacity is lost.
- ✓Running stats: in PyTorch, with the unbiased variance; eval mode uses them and becomes a fixed affine map that folds into the preceding conv.
- ✓Backward: ; gradients couple all examples in the batch.
- ✓It works mostly by smoothing optimization (Santurkar et al., 2018) rather than by removing internal covariate shift.
- ✓Small per-device batches break it: use SyncBatchNorm, or Group, Layer or Instance normalization.
- ✓Call model.eval() for inference, avoid dropout before BN, and remember weight decay on BN-fed weights acts as a learning-rate change.