Definition
Group normalization (GN) divides the channels of a layer into groups of consecutive channels. For each sample and each group it computes one mean and one variance over all channels of the group and all spatial positions, normalizes with them, and then applies a learned per-channel scale and shift .
Batch normalization estimates its statistics from the batch. That works well when each GPU holds 32 or more images, and breaks down when it holds 1 or 2. Object detection, instance segmentation and video models are in that regime because their inputs are large. Their per-batch means and variances are then noisy, the network trains on noisy normalization, and at test time it is evaluated with running averages it never saw during training. GN removes the batch from the computation entirely.
Wu and He (2018), Group Normalization (ECCV), took the idea from classic hand-designed features: SIFT and HOG normalize groups of histogram bins rather than single bins or the whole descriptor. Channels of a conv layer are similarly related (filters for the same orientation at different frequencies, say), so normalizing them together is a reasonable middle ground. On ResNet-50 for ImageNet with 32 images per GPU they reported 24.1% top-1 error for GN against 23.6% for BN, 25.3% for Layer normalization and 28.4% for Instance normalization. At 2 images per GPU, GN's error was 10.6 points lower than BN's, because GN's accuracy does not depend on the batch size at all.
Formula
- number of groups; must be divisible by . Wu & He's default is
- the values that share statistics: sample , the channels of group , all positions
- the group that channel belongs to
- learned per channel, not per group: parameters in total
- by default in PyTorch
The gradient is the standard normalization backward pass taken over each set : with and , , where the bars average over the members of the set. Within a group the gradient sums to zero and is orthogonal to , so nothing upstream can change a group's mean or overall scale through this layer.
One family, four members
All four normalizers apply the same formula. They differ only in which activations share a mean and variance. Lay the activations of a conv layer out as an grid of cells, each cell holding the values of one (sample, channel) pair:
| normalizer | statistics over | values per statistic | depends on batch? |
|---|---|---|---|
| Batch norm | for each | yes | |
| Layer norm | for each | no | |
| Instance norm | for each | no | |
| Group norm | for each | no |
With the single group is all channels, which is layer norm over . With each group is one channel, which is instance norm. GN interpolates: layer norm assumes all channels deserve one scale, instance norm assumes none are related, and GN assumes channels come in related bundles of .
At a glance
- Statistics over
- channels positions, per sample
- Batch dependence
- none: identical in training and eval, no running statistics
- Parameters
- (, ), same as batch norm
- Default G
- 32 (or a fixed such as 16 channels per group)
- Special cases
- : layer norm over ; : instance norm
- Used in
- detection and segmentation (Mask R-CNN with GN), diffusion U-Nets, BiT (with weight standardization)
- Paper
- Wu & He (2018), ECCV
The right-hand panel is the whole argument for GN in one picture. Under BN, the same input produces a different output depending on which other images happen to share its batch. That noise shrinks only like , and the gap between training-mode and eval-mode outputs is just as large. GN's output depends on the sample alone, so there is nothing to average and no train/test discrepancy.
Worked example: G = 1, 2 and 4
One sample, channels, two positions each: , , , . Take , , . Values checked in Node:
| G | groups | μ, σ per group | output (c₀ | c₁ | c₂ | c₃) |
|---|---|---|---|
| 1 (layer norm) | {c₀, c₁, c₂, c₃} | 4, 2.345 | (−1.279, −0.426) | (0.426, 1.279) | (−0.853, −0.853) | (0, 1.706) |
| 2 | {c₀, c₁}, {c₂, c₃} | 4, 2.236 and 4, 2.449 | (−1.342, −0.447) | (0.447, 1.342) | (−0.816, −0.816) | (0, 1.633) |
| 4 (instance norm) | each channel alone | (2,1), (6,1), (2,0), (6,2) | (−1, 1) | (−1, 1) | (0, 0) | (−1, 1) |
Two things show up. First, with more groups each statistic is computed from fewer values: 8, 4 and 2 here. Second, at the constant channel has and is mapped to exactly 0, and every two-value channel becomes regardless of its content. Grouping keeps relative information between channels: in the row, still sits above .
1import torch, torch.nn as nn2import torchvision34x = torch.randn(2, 64, 28, 28) # batch of 2: fine for GN56gn = nn.GroupNorm(num_groups=32, num_channels=64) # eps=1e-5, affine=True7y = gn(x)89# By hand: reshape to (N, G, C/G * H * W) and normalize the last axis10N, C, H, W = x.shape11G = 3212xg = x.view(N, G, -1)13mu, var = xg.mean(-1, keepdim=True), xg.var(-1, keepdim=True, unbiased=False)14y_ref = ((xg - mu) / torch.sqrt(var + 1e-5)).view(N, C, H, W)15print(torch.allclose(y, y_ref, atol=1e-5)) # True (gamma=1, beta=0 at init)1617# Same output for each sample whether it is alone or in a batch: no batch dependence18print(torch.allclose(gn(x[:1]), y[:1], atol=1e-6)) # True1920# Special cases21ln_like = nn.GroupNorm(1, 64) # layer norm over (C, H, W), but gamma/beta per channel22in_like = nn.GroupNorm(64, 64) # instance norm with affine=True2324# Replace every BatchNorm2d of a ResNet-50 with GroupNorm(32, C)25model = torchvision.models.resnet50(norm_layer=lambda c: nn.GroupNorm(32, c))2627# nn.GroupNorm(32, 48) raises: num_channels must be divisible by num_groupsUse when
- +Accuracy independent of batch size: same behaviour at 1, 2 or 64 samples per device
- +No running statistics: training and evaluation compute exactly the same function
- +No cross-device synchronization needed, unlike SyncBatchNorm
- +Keeps relative scale between channels inside a group, unlike instance norm
Avoid when
- −Slightly worse than BN on large-batch image classification (24.1% vs 23.6% top-1 error on ResNet-50)
- −Cannot be folded into the convolution at inference: a real per-input cost
- −Adds a hyperparameter and the divisibility constraint
- −Loses BN's mild regularizing noise, which large-batch classifiers sometimes benefit from
Check yourself
A layer has C = 64 channels and 14×14 feature maps. With nn.GroupNorm(32, 64), how many values go into each mean, and how many means are computed for a batch of 8?
Check yourself
Why does replacing batch norm with group norm remove the train/test discrepancy?
Remember
- ✓GN normalizes each sample over groups of channels and all positions, then applies per-channel .
- ✓ is layer norm over , is instance norm; is the usual default.
- ✓It is independent of batch size and identical in training and eval: the fix for small-batch detection, segmentation and video.
- ✓BN's per-sample output is noisy in small batches (spread 0.54 at batch size 2 vs 0.11 at 32 in the figure); GN's is not.
- ✓Cost: it cannot be folded into the conv at inference, and must be divisible by .