Instance normalization

Normalize each channel of each sample over its own spatial positions. The output no longer depends on that map's contrast or brightness, which is exactly the style information a style-transfer network should drop, and it never looks at the rest of the batch.

11 min read1025 wordsadvanced0 viewsupdated 2026-10-11

Definition

Instance normalization (IN) normalizes every channel of every sample separately. For sample nn and channel cc it computes the mean and variance over the H×WH\times W spatial positions of that one feature map, subtracts the mean and divides by the standard deviation. No statistic is shared across samples or across channels.

Batch normalization pools each channel over the whole batch. Instance normalization keeps the same per-channel formula but shrinks the pool to a single image. The consequence is an invariance: if a feature map is multiplied by a positive constant and shifted, its instance-normalized output does not change. In an image-generation network the per-channel mean and spread of a feature map mostly encode global contrast, colour cast and texture strength, which together look like "style". IN deletes them at every layer.

The idea comes from fast neural style transfer. Gatys, Ecker and Bethge (2016) stylized an image by slow per-image optimization. Johnson, Alahi and Fei-Fei (2016) and Ulyanov et al. (2016, texture networks) trained feed-forward generators to do it in one pass, but with batch norm their outputs were visibly worse than the optimization result. Ulyanov, Vedaldi and Lempitsky (2016), Instance Normalization: The Missing Ingredient for Fast Stylization, replaced batch norm with per-instance normalization, applied it at test time as well, and closed most of that gap. They argued that the result of stylization should not depend on the contrast of the content image, so the network should be built to ignore it.

Formula and gradient

μnc=1HW∑h,wxnchw,σnc2=1HW∑h,w(xnchw−μnc)2,ynchw=γc xnchw−μncσnc2+ε+βc\mu_{nc}=\frac{1}{HW}\sum_{h,w}x_{nchw},\qquad \sigma^2_{nc}=\frac{1}{HW}\sum_{h,w}\big(x_{nchw}-\mu_{nc}\big)^2,\qquad y_{nchw}=\gamma_c\,\frac{x_{nchw}-\mu_{nc}}{\sqrt{\sigma^2_{nc}+\varepsilon}}+\beta_c
xnchwx_{nchw}
activation of sample nn, channel cc, position (h,w)(h,w)
μnc, σnc2\mu_{nc},\ \sigma^2_{nc}
mean and biased variance of one feature map: HWHW values, one sample, one channel
γc, βc\gamma_c,\ \beta_c
optional per-channel scale and shift; PyTorch's nn.InstanceNorm2d leaves them out by default (affine=False)
ε\varepsilon
small constant (10−510^{-5} in PyTorch) that keeps the division finite for flat maps

The backward pass is the usual normalization gradient, taken over the m=HWm=HW values of one map. Write x^=(x−μ)/σ^\hat x=(x-\mu)/\hat\sigma with σ^=σ2+ε\hat\sigma=\sqrt{\sigma^2+\varepsilon} and g=γ ∂L/∂yg=\gamma\,\partial L/\partial y:

∂L∂xi=1σ^(gi−1m∑jgj−x^i 1m∑jgjx^j),∂L∂γc=∑n,h,w∂L∂y x^,∂L∂βc=∑n,h,w∂L∂y\frac{\partial L}{\partial x_i}=\frac{1}{\hat\sigma}\Big(g_i-\frac1m\sum_j g_j-\hat x_i\,\frac1m\sum_j g_j\hat x_j\Big),\qquad \frac{\partial L}{\partial \gamma_c}=\sum_{n,h,w}\frac{\partial L}{\partial y}\,\hat x,\qquad \frac{\partial L}{\partial \beta_c}=\sum_{n,h,w}\frac{\partial L}{\partial y}
mm
HWHW, the number of values in one feature map
−1m∑g-\tfrac1m\sum g
removes the part of the gradient that would shift the whole map; a shift cannot change the output
−x^i1m∑gx^-\hat x_i\tfrac1m\sum g\hat x
removes the part that would rescale the map; a rescaling cannot change the output either

The two subtracted terms are the invariance seen from the backward side. The input gradient of each map sums to zero and is orthogonal to x^\hat x, so gradient descent never tries to change the map's mean or contrast through this layer. Those directions do not exist for anything downstream.

At a glance

Statistics over
(H,W)(H,W) for each (n,c)(n,c): one mean and variance per sample per channel
Invariant to
xnc↦a xnc+bx_{nc}\mapsto a\,x_{nc}+b with a>0a>0, per map (exactly when ε=0\varepsilon=0)
Batch dependence
none: each sample is normalized alone, the same in training and at test time
Parameters
none by default; 2C2C with affine=True
Special case of
Group normalization with G=CG=C groups (one channel per group)
Typical use
style transfer, image-to-image translation (CycleGAN), as the base of AdaIN in StyleGAN
Paper
Ulyanov, Vedaldi & Lempitsky (2016)

Contrast and brightness disappear

Take one feature map and change it to x′=a x+bx'=a\,x+b with a>0a>0. Its mean becomes aμ+ba\mu+b and its standard deviation aσa\sigma, so (x′−μ′)/σ′=(ax+b−aμ−b)/(aσ)=(x−μ)/σ(x'-\mu')/\sigma'=(ax+b-a\mu-b)/(a\sigma)=(x-\mu)/\sigma. The output is identical. Batch norm has no such property: its mean and variance mix all samples, so a brighter image produces a brighter normalized map, and it also shifts the normalized maps of the other images in the same batch.

input xₙ = aₙ·contentₙ + bₙ (one channel, 8×8, N = 4)diska 1.00 b 0.00crossa 0.40 b 0.60stripesa 2.00 b −0.50rampa 1.20 b 1.50instance norm: (μ, σ) per sample over its own H·W = 64 valuesΔ 0Δ 0Δ 0Δ 0sampleμ inσ inμ outσ outdisk0.490.510.001.00cross0.780.200.001.00stripes0.541.010.001.00ramp2.100.400.001.00Δ = max |change in output| vsthe default stylesother samples moved: 0cross moved: 0a, b removed: output = content onlyε = 1e−5, biased variancetop: ■ high · bottom: ■ < 0 ■ > 0 (±2.5) · click a map to edit its style
fig.Four 8×8 maps, each a pattern times a contrast a plus a brightness b. With batch norm (one μ, σ over all 256 values) the outputs keep their styles: their standard deviations are 0.57, 0.22, 1.13 and 0.45. Instance norm gives every map mean 0 and std 1.000. Set the cross to a = 3.16, b = −1: under instance norm no output moves (Δ < 0.001), under batch norm the cross moves by 1.381 and the other three by up to 0.573. Push a down to 0.0032 and σ² = 2.6e−6 falls below ε = 1e−5: the output std shrinks to 0.457. AdaIN with the stripes as style gives all four maps mean 0.542 and std 1.006.

The figure shows the trade at the heart of IN. It removes per-image intensity statistics completely and makes every sample independent of the batch. But it removes them whether or not they mattered: a dark cat and a bright cat look the same after IN, and so do a faint edge and a strong edge in the same channel. For recognition that information is often useful, which is why IN is rare in classifiers.

Worked example: one 2×2 map

Take a single channel with values x=(1,3,5,7)x=(1,3,5,7) (a 2×2 map, flattened). The mean is 4 and the biased variance is (9+1+1+9)/4=5(9+1+1+9)/4=5, so σ^=5+10−5=2.2361\hat\sigma=\sqrt{5+10^{-5}}=2.2361. Every number below was checked in Node with ε=10−5\varepsilon=10^{-5}.

input mapμσoutput
(1, 3, 5, 7)42.236(−1.3416, −0.4472, 0.4472, 1.3416)
2x+102x+10 = (12, 16, 20, 24)184.472(−1.3416, −0.4472, 0.4472, 1.3416)
AdaIN onto style (0.3, 0.7, 0.3, 0.7)0.50.2(0.2317, 0.4106, 0.5894, 0.7683)
Instance norm of a 2×2 map, of a contrast-and-brightness-changed copy, and AdaIN onto a target with μ = 0.5, σ = 0.2

Backward through the first map, upstream gradient ∂L/∂y = (1, 0, 0, 0), γ = 1

  1. 1x^=(−1.3416,−0.4472,0.4472,1.3416)\hat x=(-1.3416,-0.4472,0.4472,1.3416). Mean of gg: 0.250.25. Mean of gx^g\hat x: −1.3416/4=−0.3354-1.3416/4=-0.3354.
  2. 2g−0.25−x^⋅(−0.3354)=(0.75−0.45,  −0.25−0.15,  −0.25+0.15,  −0.25+0.45)=(0.3, −0.4, −0.1, 0.2)g-0.25-\hat x\cdot(-0.3354)=(0.75-0.45,\;-0.25-0.15,\;-0.25+0.15,\;-0.25+0.45)=(0.3,\,-0.4,\,-0.1,\,0.2).
  3. 3Divide by σ^=2.2361\hat\sigma=2.2361: ∂L/∂x=(0.1342, −0.1789, −0.0447, 0.0894)\partial L/\partial x=(0.1342,\,-0.1789,\,-0.0447,\,0.0894), matching a finite-difference check.
  4. 4The entries sum to 0, and ∑ix^i ∂L/∂xi=0\sum_i \hat x_i\,\partial L/\partial x_i=0: the gradient can neither shift nor rescale the map.

AdaIN: replace the statistics instead of removing them

If channel means and standard deviations carry style, then a style can be imposed by normalizing the content features and putting in another image's statistics. Huang and Belongie (2017), Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, did exactly this with VGG features of a content image xx and a style image ss. AdaIN has no learned parameters, so a single network transfers any style, including styles never seen during training.

AdaIN(x,s)=σnc(s) xnc−μnc(x)σnc(x)+μnc(s)\mathrm{AdaIN}(x,s)=\sigma_{nc}(s)\,\frac{x_{nc}-\mu_{nc}(x)}{\sigma_{nc}(x)}+\mu_{nc}(s)
μnc(x), σnc(x)\mu_{nc}(x),\ \sigma_{nc}(x)
instance statistics of the content features (removed)
μnc(s), σnc(s)\mu_{nc}(s),\ \sigma_{nc}(s)
instance statistics of the style features, used in place of βc\beta_c and γc\gamma_c

Two neighbours use the same slot. Conditional instance normalization (Dumoulin, Shlens and Kudlur, 2017) learns one (γ,β)(\gamma,\beta) pair per style, so one network holds many styles. StyleGAN (Karras, Laine and Aila, 2019) computes the scale and shift of every AdaIN layer from a learned latent code ww, which is how it controls coarse and fine style separately. StyleGAN2 (2020) later swapped AdaIN for weight demodulation, a normalization folded into the convolution weights, to remove the "droplet" artefacts that the per-map normalization caused.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.randn(8, 64, 32, 32) # (N, C, H, W)
4
5inorm = nn.InstanceNorm2d(64) # defaults: eps=1e-5, momentum=0.1,
6 # affine=False, track_running_stats=False
7y = inorm(x)
8
9# The same thing by hand: statistics over (H, W) only, biased variance
10mu = x.mean(dim=(2, 3), keepdim=True)
11var = x.var(dim=(2, 3), keepdim=True, unbiased=False)
12print(torch.allclose(y, (x - mu) / torch.sqrt(var + 1e-5), atol=1e-5)) # True
13
14# Invariance to per-map contrast and brightness (a > 0)
15a = torch.rand(8, 64, 1, 1) + 0.1
16b = torch.randn(8, 64, 1, 1)
17print(torch.allclose(inorm(a * x + b), y, atol=1e-4)) # True
18
19# InstanceNorm with affine params == GroupNorm with one channel per group
20gn = nn.GroupNorm(num_groups=64, num_channels=64)
21print(torch.allclose(gn(x), nn.InstanceNorm2d(64, affine=True)(x), atol=1e-5)) # True
22
23def adain(content, style, eps=1e-5):
24 """Huang & Belongie (2017): content statistics out, style statistics in."""
25 c_mu, c_std = content.mean((2, 3), keepdim=True), content.std((2, 3), keepdim=True, unbiased=False)
26 s_mu, s_std = style.mean((2, 3), keepdim=True), style.std((2, 3), keepdim=True, unbiased=False)
27 return s_std * (content - c_mu) / (c_std + eps) + s_mu
28
29# A 1x1 feature map has no spatial variance: in training mode this raises
30# "Expected more than 1 spatial element when training".
31# F.instance_norm(torch.randn(8, 64, 1, 1))
nn.InstanceNorm2d, its defaults, and AdaIN in a few lines

Use when

  • +Removes per-image contrast and brightness exactly, at every layer
  • +Independent of the batch: works with batch size 1 and behaves the same in training and at test time
  • +Gives AdaIN and conditional IN a natural slot for injecting style statistics
  • +Cheap: one mean and variance per map, no running statistics to maintain

Avoid when

  • −Discards information that classifiers often need (global intensity, contrast between images)
  • −Undefined or degenerate for 1×1 maps and vector inputs; noisy for very small maps
  • −Ignores relations between channels: each one is standardized alone, unlike Layer normalization or Group normalization
  • −Can cause artefacts in generators that need to carry magnitude information (StyleGAN2's reason to drop AdaIN)

Check yourself

In training mode, a feature map in one image is multiplied by 3 and shifted by +2. What happens to the outputs of an instance-norm layer and of a batch-norm layer (batch size 8)?

Check yourself

Why does AdaIN let one trained decoder render arbitrary, unseen styles?

Remember

  • ✓IN normalizes each (n,c)(n,c) feature map over its own H×WH\times W values: y=γc(x−μnc)/σnc2+ε+βcy=\gamma_c(x-\mu_{nc})/\sqrt{\sigma^2_{nc}+\varepsilon}+\beta_c.
  • ✓It is invariant to x↦ax+bx\mapsto ax+b (a>0a>0) per map, so it erases contrast and brightness, which act as "style" in generators.
  • ✓No batch statistics: identical in training and evaluation, works at batch size 1. It is Group normalization with G = C.
  • ✓AdaIN replaces the removed statistics with a style image's: σ(s) x^+μ(s)\sigma(s)\,\hat x+\mu(s); StyleGAN computes them from a latent code.
  • ✓PyTorch defaults: affine=False, track_running_stats=False. It fails on 1×11\times1 maps and is a poor default for classifiers.
← back to Deep Learning roadmap