Group normalization

Split the channels into G groups and normalize each group of each sample over its channels and positions. It never looks at other samples, so it works the same at batch size 2 as at 64; G = 1 gives layer norm and G = C gives instance norm.

11 min read888 wordscore0 viewsupdated 2026-10-11

Definition

Group normalization (GN) divides the CC channels of a layer into GG groups of C/GC/G consecutive channels. For each sample and each group it computes one mean and one variance over all channels of the group and all spatial positions, normalizes with them, and then applies a learned per-channel scale γc\gamma_c and shift βc\beta_c.

Batch normalization estimates its statistics from the batch. That works well when each GPU holds 32 or more images, and breaks down when it holds 1 or 2. Object detection, instance segmentation and video models are in that regime because their inputs are large. Their per-batch means and variances are then noisy, the network trains on noisy normalization, and at test time it is evaluated with running averages it never saw during training. GN removes the batch from the computation entirely.

Wu and He (2018), Group Normalization (ECCV), took the idea from classic hand-designed features: SIFT and HOG normalize groups of histogram bins rather than single bins or the whole descriptor. Channels of a conv layer are similarly related (filters for the same orientation at different frequencies, say), so normalizing them together is a reasonable middle ground. On ResNet-50 for ImageNet with 32 images per GPU they reported 24.1% top-1 error for GN against 23.6% for BN, 25.3% for Layer normalization and 28.4% for Instance normalization. At 2 images per GPU, GN's error was 10.6 points lower than BN's, because GN's accuracy does not depend on the batch size at all.

Formula

Sn,g={(c,h,w)  :  ⌊cC/G⌋=g},μng=1m∑Sn,gxnchw,σng2=1m∑Sn,g(xnchw−μng)2\mathcal S_{n,g}=\Big\{(c,h,w)\;:\;\Big\lfloor\tfrac{c}{C/G}\Big\rfloor=g\Big\},\qquad \mu_{ng}=\frac{1}{m}\sum_{\mathcal S_{n,g}}x_{nchw},\qquad \sigma^2_{ng}=\frac{1}{m}\sum_{\mathcal S_{n,g}}\big(x_{nchw}-\mu_{ng}\big)^2
GG
number of groups; CC must be divisible by GG. Wu & He's default is G=32G=32
Sn,g\mathcal S_{n,g}
the values that share statistics: sample nn, the C/GC/G channels of group gg, all positions
mm
∣Sn,g∣=(C/G)⋅H⋅W|\mathcal S_{n,g}|=(C/G)\cdot H\cdot W
ynchw=γc xnchw−μn,g(c)σn,g(c)2+ε+βcy_{nchw}=\gamma_c\,\frac{x_{nchw}-\mu_{n,g(c)}}{\sqrt{\sigma^2_{n,g(c)}+\varepsilon}}+\beta_c
g(c)g(c)
the group that channel cc belongs to
γc, βc\gamma_c,\ \beta_c
learned per channel, not per group: 2C2C parameters in total
ε\varepsilon
10−510^{-5} by default in PyTorch

The gradient is the standard normalization backward pass taken over each set Sn,g\mathcal S_{n,g}: with x^=(x−μ)/σ^\hat x=(x-\mu)/\hat\sigma and gi=γc ∂L/∂yig_i=\gamma_c\,\partial L/\partial y_i,   ∂L/∂xi=σ^−1(gi−g‾−x^i gx^‾)\;\partial L/\partial x_i=\hat\sigma^{-1}\big(g_i-\overline{g}-\hat x_i\,\overline{g\hat x}\big), where the bars average over the mm members of the set. Within a group the gradient sums to zero and is orthogonal to x^\hat x, so nothing upstream can change a group's mean or overall scale through this layer.

One family, four members

All four normalizers apply the same formula. They differ only in which activations share a mean and variance. Lay the activations of a conv layer out as an N×CN\times C grid of cells, each cell holding the H×WH\times W values of one (sample, channel) pair:

normalizerstatistics overvalues per statisticdepends on batch?
Batch norm(N,H,W)(N,H,W) for each ccN⋅H⋅WN\cdot H\cdot Wyes
Layer norm(C,H,W)(C,H,W) for each nnC⋅H⋅WC\cdot H\cdot Wno
Instance norm(H,W)(H,W) for each (n,c)(n,c)H⋅WH\cdot Wno
Group norm(C/G,H,W)(C/G,H,W) for each (n,g)(n,g)(C/G)⋅H⋅W(C/G)\cdot H\cdot Wno
Which cells share statistics, and how many values go into each mean (C channels, H×W positions, batch N)

With G=1G=1 the single group is all channels, which is layer norm over (C,H,W)(C,H,W). With G=CG=C each group is one channel, which is instance norm. GN interpolates: layer norm assumes all channels deserve one scale, instance norm assumes none are related, and GN assumes channels come in related bundles of C/GC/G.

At a glance

Statistics over
C/GC/G channels × H×W\times\,H\times W positions, per sample
Batch dependence
none: identical in training and eval, no running statistics
Parameters
2C2C (γc\gamma_c, βc\beta_c), same as batch norm
Default G
32 (or a fixed C/GC/G such as 16 channels per group)
Special cases
G=1G=1: layer norm over (C,H,W)(C,H,W); G=CG=C: instance norm
Used in
detection and segmentation (Mask R-CNN with GN), diffusion U-Nets, BiT (with weight standardization)
Paper
Wu & He (2018), ECCV
one cell = one (sample, channel), its H·W = 9 valuesgroup norm, G = 4 (4 ch/group)n0n1n2n3channels c = 0 … 15 →shared by 4 cells × 9 = 36 valuesμ = 0.107 σ = 0.602x̂[n0, c5, centre] = 0.003uses only this samplesame in train and evalsample 0, channel 0, centre pixel:x̂ in 200 random batches of size B = 2−1012● BN train ┆ BN eval 0.561 ┃ GN G=4 −0.263BN spread 0.538 · |train − eval| 0.501GN spread 0 · |train − eval| 0BN spread of x̂ vs batch size0.0010.5420.4240.3080.19160.1132batch size B (B = 1: BN train = instance norm)click a cell to see its statistics set · 64 samples in the dataset
groups GG = 1 → LN · G = C → IN
fig.Left: 4 samples × 16 channels, each cell one 3×3 feature map. Click a cell to see which cells share its statistics: a column for batch norm, a row for layer norm, the cell alone for instance norm, a row segment of 16/G channels for group norm. With G = 4 one statistic pools 4 cells, 36 values. Right: the batch-normalized value of one fixed pixel of sample 0, over 200 random batches drawn from 64 samples. Its spread across batches is 0.538 at batch size 2, 0.416 at 4, 0.303 at 8, 0.186 at 16 and 0.113 at 32, and its mean distance from the eval-mode value 0.561 is 0.501 at batch size 2. At batch size 1 the spread is 0 because BN then equals instance norm, but the distance from eval mode is still 0.502. Group norm gives one fixed value for every batch.

The right-hand panel is the whole argument for GN in one picture. Under BN, the same input produces a different output depending on which other images happen to share its batch. That noise shrinks only like 1/N1/\sqrt{N}, and the gap between training-mode and eval-mode outputs is just as large. GN's output depends on the sample alone, so there is nothing to average and no train/test discrepancy.

Worked example: G = 1, 2 and 4

One sample, C=4C=4 channels, two positions each: c0=(1,3)c_0=(1,3), c1=(5,7)c_1=(5,7), c2=(2,2)c_2=(2,2), c3=(4,8)c_3=(4,8). Take γ=1\gamma=1, β=0\beta=0, ε=10−5\varepsilon=10^{-5}. Values checked in Node:

Ggroupsμ, σ per groupoutput (c₀ | c₁ | c₂ | c₃)
1 (layer norm){c₀, c₁, c₂, c₃}4, 2.345(−1.279, −0.426) | (0.426, 1.279) | (−0.853, −0.853) | (0, 1.706)
2{c₀, c₁}, {c₂, c₃}4, 2.236 and 4, 2.449(−1.342, −0.447) | (0.447, 1.342) | (−0.816, −0.816) | (0, 1.633)
4 (instance norm)each channel alone(2,1), (6,1), (2,0), (6,2)(−1, 1) | (−1, 1) | (0, 0) | (−1, 1)
The same 8 activations under three group counts

Two things show up. First, with more groups each statistic is computed from fewer values: 8, 4 and 2 here. Second, at G=CG=C the constant channel c2c_2 has σ=0\sigma=0 and is mapped to exactly 0, and every two-value channel becomes (−1,1)(-1,1) regardless of its content. Grouping keeps relative information between channels: in the G=2G=2 row, c1c_1 still sits above c0c_0.

python
1import torch, torch.nn as nn
2import torchvision
3
4x = torch.randn(2, 64, 28, 28) # batch of 2: fine for GN
5
6gn = nn.GroupNorm(num_groups=32, num_channels=64) # eps=1e-5, affine=True
7y = gn(x)
8
9# By hand: reshape to (N, G, C/G * H * W) and normalize the last axis
10N, C, H, W = x.shape
11G = 32
12xg = x.view(N, G, -1)
13mu, var = xg.mean(-1, keepdim=True), xg.var(-1, keepdim=True, unbiased=False)
14y_ref = ((xg - mu) / torch.sqrt(var + 1e-5)).view(N, C, H, W)
15print(torch.allclose(y, y_ref, atol=1e-5)) # True (gamma=1, beta=0 at init)
16
17# Same output for each sample whether it is alone or in a batch: no batch dependence
18print(torch.allclose(gn(x[:1]), y[:1], atol=1e-6)) # True
19
20# Special cases
21ln_like = nn.GroupNorm(1, 64) # layer norm over (C, H, W), but gamma/beta per channel
22in_like = nn.GroupNorm(64, 64) # instance norm with affine=True
23
24# Replace every BatchNorm2d of a ResNet-50 with GroupNorm(32, C)
25model = torchvision.models.resnet50(norm_layer=lambda c: nn.GroupNorm(32, c))
26
27# nn.GroupNorm(32, 48) raises: num_channels must be divisible by num_groups
nn.GroupNorm, its special cases, and swapping it into a torchvision ResNet

Use when

  • +Accuracy independent of batch size: same behaviour at 1, 2 or 64 samples per device
  • +No running statistics: training and evaluation compute exactly the same function
  • +No cross-device synchronization needed, unlike SyncBatchNorm
  • +Keeps relative scale between channels inside a group, unlike instance norm

Avoid when

  • −Slightly worse than BN on large-batch image classification (24.1% vs 23.6% top-1 error on ResNet-50)
  • −Cannot be folded into the convolution at inference: a real per-input cost
  • −Adds a hyperparameter GG and the divisibility constraint C mod G=0C\bmod G=0
  • −Loses BN's mild regularizing noise, which large-batch classifiers sometimes benefit from

Check yourself

A layer has C = 64 channels and 14×14 feature maps. With nn.GroupNorm(32, 64), how many values go into each mean, and how many means are computed for a batch of 8?

Check yourself

Why does replacing batch norm with group norm remove the train/test discrepancy?

Remember

  • ✓GN normalizes each sample over groups of C/GC/G channels and all positions, then applies per-channel γc,βc\gamma_c,\beta_c.
  • ✓G=1G=1 is layer norm over (C,H,W)(C,H,W), G=CG=C is instance norm; G=32G=32 is the usual default.
  • ✓It is independent of batch size and identical in training and eval: the fix for small-batch detection, segmentation and video.
  • ✓BN's per-sample output is noisy in small batches (spread 0.54 at batch size 2 vs 0.11 at 32 in the figure); GN's is not.
  • ✓Cost: it cannot be folded into the conv at inference, and CC must be divisible by GG.
← back to Deep Learning roadmap