Definition
Instance normalization (IN) normalizes every channel of every sample separately. For sample and channel it computes the mean and variance over the spatial positions of that one feature map, subtracts the mean and divides by the standard deviation. No statistic is shared across samples or across channels.
Batch normalization pools each channel over the whole batch. Instance normalization keeps the same per-channel formula but shrinks the pool to a single image. The consequence is an invariance: if a feature map is multiplied by a positive constant and shifted, its instance-normalized output does not change. In an image-generation network the per-channel mean and spread of a feature map mostly encode global contrast, colour cast and texture strength, which together look like "style". IN deletes them at every layer.
The idea comes from fast neural style transfer. Gatys, Ecker and Bethge (2016) stylized an image by slow per-image optimization. Johnson, Alahi and Fei-Fei (2016) and Ulyanov et al. (2016, texture networks) trained feed-forward generators to do it in one pass, but with batch norm their outputs were visibly worse than the optimization result. Ulyanov, Vedaldi and Lempitsky (2016), Instance Normalization: The Missing Ingredient for Fast Stylization, replaced batch norm with per-instance normalization, applied it at test time as well, and closed most of that gap. They argued that the result of stylization should not depend on the contrast of the content image, so the network should be built to ignore it.
Formula and gradient
- activation of sample , channel , position
- mean and biased variance of one feature map: values, one sample, one channel
- optional per-channel scale and shift; PyTorch's
nn.InstanceNorm2dleaves them out by default (affine=False) - small constant ( in PyTorch) that keeps the division finite for flat maps
The backward pass is the usual normalization gradient, taken over the values of one map. Write with and :
- , the number of values in one feature map
- removes the part of the gradient that would shift the whole map; a shift cannot change the output
- removes the part that would rescale the map; a rescaling cannot change the output either
The two subtracted terms are the invariance seen from the backward side. The input gradient of each map sums to zero and is orthogonal to , so gradient descent never tries to change the map's mean or contrast through this layer. Those directions do not exist for anything downstream.
At a glance
- Statistics over
- for each : one mean and variance per sample per channel
- Invariant to
- with , per map (exactly when )
- Batch dependence
- none: each sample is normalized alone, the same in training and at test time
- Parameters
- none by default; with
affine=True - Special case of
- Group normalization with groups (one channel per group)
- Typical use
- style transfer, image-to-image translation (CycleGAN), as the base of AdaIN in StyleGAN
- Paper
- Ulyanov, Vedaldi & Lempitsky (2016)
Contrast and brightness disappear
Take one feature map and change it to with . Its mean becomes and its standard deviation , so . The output is identical. Batch norm has no such property: its mean and variance mix all samples, so a brighter image produces a brighter normalized map, and it also shifts the normalized maps of the other images in the same batch.
The figure shows the trade at the heart of IN. It removes per-image intensity statistics completely and makes every sample independent of the batch. But it removes them whether or not they mattered: a dark cat and a bright cat look the same after IN, and so do a faint edge and a strong edge in the same channel. For recognition that information is often useful, which is why IN is rare in classifiers.
Worked example: one 2×2 map
Take a single channel with values (a 2×2 map, flattened). The mean is 4 and the biased variance is , so . Every number below was checked in Node with .
| input map | μ | σ | output |
|---|---|---|---|
| (1, 3, 5, 7) | 4 | 2.236 | (−1.3416, −0.4472, 0.4472, 1.3416) |
| = (12, 16, 20, 24) | 18 | 4.472 | (−1.3416, −0.4472, 0.4472, 1.3416) |
| AdaIN onto style (0.3, 0.7, 0.3, 0.7) | 0.5 | 0.2 | (0.2317, 0.4106, 0.5894, 0.7683) |
Backward through the first map, upstream gradient ∂L/∂y = (1, 0, 0, 0), γ = 1
- 1. Mean of : . Mean of : .
- 2.
- 3Divide by : , matching a finite-difference check.
- 4The entries sum to 0, and : the gradient can neither shift nor rescale the map.
AdaIN: replace the statistics instead of removing them
If channel means and standard deviations carry style, then a style can be imposed by normalizing the content features and putting in another image's statistics. Huang and Belongie (2017), Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, did exactly this with VGG features of a content image and a style image . AdaIN has no learned parameters, so a single network transfers any style, including styles never seen during training.
- instance statistics of the content features (removed)
- instance statistics of the style features, used in place of and
Two neighbours use the same slot. Conditional instance normalization (Dumoulin, Shlens and Kudlur, 2017) learns one pair per style, so one network holds many styles. StyleGAN (Karras, Laine and Aila, 2019) computes the scale and shift of every AdaIN layer from a learned latent code , which is how it controls coarse and fine style separately. StyleGAN2 (2020) later swapped AdaIN for weight demodulation, a normalization folded into the convolution weights, to remove the "droplet" artefacts that the per-map normalization caused.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.randn(8, 64, 32, 32) # (N, C, H, W)45inorm = nn.InstanceNorm2d(64) # defaults: eps=1e-5, momentum=0.1,6 # affine=False, track_running_stats=False7y = inorm(x)89# The same thing by hand: statistics over (H, W) only, biased variance10mu = x.mean(dim=(2, 3), keepdim=True)11var = x.var(dim=(2, 3), keepdim=True, unbiased=False)12print(torch.allclose(y, (x - mu) / torch.sqrt(var + 1e-5), atol=1e-5)) # True1314# Invariance to per-map contrast and brightness (a > 0)15a = torch.rand(8, 64, 1, 1) + 0.116b = torch.randn(8, 64, 1, 1)17print(torch.allclose(inorm(a * x + b), y, atol=1e-4)) # True1819# InstanceNorm with affine params == GroupNorm with one channel per group20gn = nn.GroupNorm(num_groups=64, num_channels=64)21print(torch.allclose(gn(x), nn.InstanceNorm2d(64, affine=True)(x), atol=1e-5)) # True2223def adain(content, style, eps=1e-5):24 """Huang & Belongie (2017): content statistics out, style statistics in."""25 c_mu, c_std = content.mean((2, 3), keepdim=True), content.std((2, 3), keepdim=True, unbiased=False)26 s_mu, s_std = style.mean((2, 3), keepdim=True), style.std((2, 3), keepdim=True, unbiased=False)27 return s_std * (content - c_mu) / (c_std + eps) + s_mu2829# A 1x1 feature map has no spatial variance: in training mode this raises30# "Expected more than 1 spatial element when training".31# F.instance_norm(torch.randn(8, 64, 1, 1))Use when
- +Removes per-image contrast and brightness exactly, at every layer
- +Independent of the batch: works with batch size 1 and behaves the same in training and at test time
- +Gives AdaIN and conditional IN a natural slot for injecting style statistics
- +Cheap: one mean and variance per map, no running statistics to maintain
Avoid when
- −Discards information that classifiers often need (global intensity, contrast between images)
- −Undefined or degenerate for 1×1 maps and vector inputs; noisy for very small maps
- −Ignores relations between channels: each one is standardized alone, unlike Layer normalization or Group normalization
- −Can cause artefacts in generators that need to carry magnitude information (StyleGAN2's reason to drop AdaIN)
Check yourself
In training mode, a feature map in one image is multiplied by 3 and shifted by +2. What happens to the outputs of an instance-norm layer and of a batch-norm layer (batch size 8)?
Check yourself
Why does AdaIN let one trained decoder render arbitrary, unseen styles?
Remember
- ✓IN normalizes each feature map over its own values: .
- ✓It is invariant to () per map, so it erases contrast and brightness, which act as "style" in generators.
- ✓No batch statistics: identical in training and evaluation, works at batch size 1. It is Group normalization with G = C.
- ✓AdaIN replaces the removed statistics with a style image's: ; StyleGAN computes them from a latent code.
- ✓PyTorch defaults:
affine=False,track_running_stats=False. It fails on maps and is a poor default for classifiers.