StyleGAN

StyleGAN (Karras, Laine, Aila, CVPR 2019) maps z through an 8-layer MLP to an intermediate latent w. It injects w as per-layer AdaIN styles into a synthesis network that starts from a learned constant, adds per-pixel noise for stochastic detail, and supports style mixing and truncation in W.

15 min read1237 wordsadvanced0 viewsupdated 2026-10-12

Definition

StyleGAN is the style-based GAN generator of Karras, Laine and Aila, "A Style-Based Generator Architecture for Generative Adversarial Networks" (CVPR 2019). It never feeds the latent zz directly into the image network. An 8-layer mapping network first turns z∈R512z\in\mathbb R^{512} into an intermediate latent w∈Ww\in\mathcal W. A learned affine map then turns ww into a style (ys,yb)(y_s,y_b) for every layer of the synthesis network, applied by adaptive instance normalisation (AdaIN). The synthesis network starts from a learned constant 4×4×5124\times4\times512 tensor, and per-pixel noise is added at every layer to produce stochastic detail.

By 2018 GANs could produce convincing 1024×10241024\times1024 faces. Progressive GAN (Karras et al., ICLR 2018) grew generator and discriminator together from 424^2 to 102421024^2. The generator itself was still a black box: zz entered at the first layer, and nothing controlled which layer decided pose, identity, or freckles. Tero Karras, Samuli Laine and Timo Aila (NVIDIA) redesigned only the generator, borrowing from style transfer. In Huang and Belongie's AdaIN (2017), the "style" of an image is the per-channel mean and standard deviation of its feature maps. StyleGAN lets ww set those statistics separately at every resolution. The discriminator, loss and training schedule are kept from Progressive GAN. The paper also introduced the FFHQ dataset and two measures of latent-space disentanglement, perceptual path length and linear separability. Its successors, StyleGAN2 (2020) and StyleGAN3 (2021), defined the state of the art for unconditional image synthesis for several years.

Mapping network and AdaIN

w=f(z),f:R512→R512 (8 FC layers),y=(ys, yb)=Aℓ(w),AdaIN(xi,y)=ys,i xi−μ(xi)σ(xi)+yb,iw=f(z),\quad f:\mathbb R^{512}\to\mathbb R^{512}\ \text{(8 FC layers)},\qquad y=(y_s,\,y_b)=A_\ell(w),\qquad \mathrm{AdaIN}(x_i,y)=y_{s,i}\,\frac{x_i-\mu(x_i)}{\sigma(x_i)}+y_{b,i}
z∼N(0,I)z\sim\mathcal N(0,I)
input latent, normalised before the first FC layer
ff
mapping network, 8 fully connected layers of width 512 with LeakyReLU; its learning rate is 100×100\times lower than the rest
AℓA_\ell
learned affine transform for layer ℓ\ell that "specialises" ww into one scale and one bias per channel
xix_i
feature map ii of the current layer; μ,σ\mu,\sigma are its spatial mean and standard deviation
AdaIN\mathrm{AdaIN}
normalise each feature map, then impose the style's scale and bias; the style controls this layer only, until the next AdaIN overwrites the statistics

Why an intermediate latent? zz has a fixed prior, so the generator must warp a Gaussian onto the data. Suppose some attribute combination is rare in the data, such as long-haired men in the paper's example. In Z\mathcal Z that region must be squeezed, and the mapping from zz to image becomes curved and entangled. W\mathcal W has no fixed distribution. The learned ff can "unwarp" it so that factors of variation become more linear. The paper argues this is easier for the generator and measures it.

Synthesis network: constant input, styles, noise

The 102421024^2 synthesis network has 18 layers, two per resolution from 424^2 to 102421024^2, each followed by AdaIN. The first takes a learned constant 4×4×5124\times4\times512 tensor instead of zz. Removing the traditional input did not hurt FID. The image is controlled entirely by the styles and the noise. Upsampling uses bilinear filtering. A final 1×11\times1 convolution (toRGB) produces the image. Before each AdaIN, a single-channel image of uncorrelated Gaussian noise is added to the feature maps, scaled by learned per-channel factors (the "B" blocks in the paper's Figure 1). The noise controls only stochastic detail: the exact placement of hairs, freckles, stubble and skin pores. Identity and pose are unaffected. Because every layer has fresh noise, the network does not need to generate pseudo-randomness from earlier activations. Coarse-layer noise yields large-scale curls, and fine-layer noise yields fine strands.

The synthesis network has 26.2M trainable parameters, against 23.1M for the traditional generator. The style mechanism is cheap.

Style mixing

Each style is used by one layer and then normalised away, so styles are localised. This allows style mixing: map two latents z1,z2z_1,z_2 to w1,w2w_1,w_2, and feed w1w_1 to the layers before a random crossover point and w2w_2 to the rest. The paper's Figure 3 shows the effect. Coarse styles (424^2–828^2) from source B carry pose, general hair style, face shape and eyeglasses. Middle styles (16216^2–32232^2) carry smaller facial features, hair style, and eyes open or closed. Fine styles (64264^2–102421024^2) carry mainly the colour scheme and microstructure. As a regulariser, a given percentage of training images are generated with two latents. The default is 90%. This stops the network from assuming that adjacent styles are correlated.

Truncation in W

wˉ=Ez∼p(z)[f(z)],w′=wˉ+ψ (w−wˉ),ψ<1\bar w=\mathbb E_{z\sim p(z)}\big[f(z)\big],\qquad w'=\bar w+\psi\,(w-\bar w),\qquad \psi<1
wˉ\bar w
centre of mass of W\mathcal W, estimated as a running average of f(z)f(z) during training
ψ\psi
truncation strength: ψ=1\psi=1 no change, ψ=0\psi=0 every sample becomes the "mean face", ψ<0\psi<0 gives faces with opposite attributes
w′w'
can be applied only to some layers; the paper's sample figures use ψ = 0.7 on the 4²–32² layers

Low-density regions of the data are poorly represented, so samples drawn from them look worse. Truncation trades variety for average quality by shrinking samples toward the dense centre. BigGAN (see BigGAN) truncates in Z\mathcal Z by resampling large ∣z∣|z|. StyleGAN truncates in W\mathcal W, where the learned centre is meaningful. All FIDs in the paper are computed without truncation.

Measuring disentanglement: perceptual path length

lW=E[1ϵ2 d(g(lerp(f(z1),f(z2);t)), g(lerp(f(z1),f(z2);t+ϵ)))],t∼U(0,1), ϵ=10−4l_{\mathcal W}=\mathbb E\Big[\frac1{\epsilon^2}\,d\big(g(\mathrm{lerp}(f(z_1),f(z_2);t)),\ g(\mathrm{lerp}(f(z_1),f(z_2);t+\epsilon))\big)\Big],\qquad t\sim U(0,1),\ \epsilon=10^{-4}
gg
the synthesis network
dd
LPIPS perceptual distance (VGG16 features) between the two images, cropped to the face
lerp\mathrm{lerp}
linear interpolation in W\mathcal W; for Z\mathcal Z the paper uses spherical interpolation (slerp) of zz
ll
small path length means images change smoothly along latent lines; sharp changes betray entanglement

A second measure, linear separability, trains 40 linear SVMs on ww (or zz) to predict CelebA attributes labelled by auxiliary classifiers. It reports exp⁡(∑iH(Yi∣Xi))\exp\big(\sum_i H(Y_i\mid X_i)\big), the conditional entropy of the classifier's label given the SVM's side of the hyperplane. Lower values mean more linear factors. With the full 8-layer mapping network the style-based generator reached path length 234.0 in W\mathcal W and separability 3.79. The traditional generator scored 412.0 and 10.78.

Setup in the paper

FFHQ
Flickr-Faces-HQ: 70,000 face images at 102421024^2, aligned and cropped, with more variation in age, ethnicity, background and accessories than CelebA-HQ
Latents
z,w∈R512z,w\in\mathbb R^{512}; 18 copies of ww (one per layer) at 102421024^2
Mapping
8 FC layers, learning rate multiplied by 0.010.01
Losses
WGAN-GP on CelebA-HQ; non-saturating loss with R1R_1 regularisation (γ=10\gamma=10) on FFHQ
Training
progressive growing as in Progressive GAN; about one week on 8 Tesla V100 GPUs
Regularisation
style mixing on 90% of training images
Evaluation
FID on 50,000 random images, no truncation
configurationCelebA-HQFFHQ
A baseline Progressive GAN7.798.04
B + tuning (incl. bilinear up/down)6.115.25
C + mapping network and styles5.344.85
D + remove traditional input5.074.88
E + noise inputs5.064.42
F + mixing regularisation5.174.40
FID (lower is better) as the generator changes step by step (Karras et al. 2019, Table 1)
z ~ N(0, I) → mapping f → w → 18 × (A: w ↦ (y_s, y_b)) → AdaIN in every layertoy W = f(Z) · shade p_W · ● truncated w'AB✕ w̄ = E[f(z)] = (0.5, 0)const 4×4×512 input, then style per layer:4²×512B0B1coarse8²×512B2B316²×512A4A5middle32²×512A6A764²×256A8A9fine128²×128A10A11256²×64A12A13512²×32A14A151024²×16A16A17+ per-pixel noise, scaledper channel, every layerψ = 0.70: mean ‖w' − w̄‖ = 0.993 = |ψ| × 1.418 (diversity shrinks linearly)mean log p_W(w') = −2.477 (at ψ = 1: −2.907); in rare regions (log p < -4): 7 of 160rare regions of W are learned worst; shrinking toward w̄ trades diversity for qualityw_A = (1.437, 0.978), w_B = (0.662, 0.382) after truncation; B feeds layers 0–3from B (coarse): pose, general hair style, face shape, eyeglasseseverything else comes from A (including colours, which live in the fine layers)mixing regularisation (two latents in 90% of training images) decorrelates adjacent styles
style mixing

Crossover c: layers 0…c−1 take w_A, layers c…17 take w_B. The W panel is a 2-D toy (f(z) = (z₁ + z₂²/2, z₂ + 0.4 sin 2z₁)); the real W is 512-dimensional.

fig.The left panel is a 2-D stand-in for W: 160 samples z ~ N(0, I) pushed through a fixed warp f(z) = (z₁ + z₂²/2, z₂ + 0.4 sin 2z₁). The shading is a kernel density estimate of p_W from 500 more samples. The mean is w̄ = E[f(z)] = (0.5, 0), whereas f(0) = (0, 0): the centre of Z is not the centre of W, so truncation must use w̄. Truncation w' = w̄ + ψ(w − w̄) scales the mean distance to w̄ exactly linearly: 1.418 at ψ = 1, 0.993 at ψ = 0.7, 0.709 at ψ = 0.5. The mean log-density of the samples rises from −2.907 to −2.477 and −2.279, and at ψ = 0 every sample sits at w̄ (−2.077). The number of samples in rare regions (log p < −4) falls from 22 of 160 to 7 and then 2. These are the regions a generator learns worst, so this is the diversity-for-quality trade. Negative ψ reflects samples through w̄. The right panel lists the 18 style inputs of the 1024² synthesis network with feature maps per resolution (512 up to 32², then 256, 128, 64, 32, 16). Style-mixing presets send w_B to the coarse (layers 0–3, 4²–8²), middle (4–7, 16²–32²) or fine (8–17, 64²–1024²) layers and list what the paper reports those layers control. The crossover slider gives the training-time mixing rule: layers before c take w_A, the rest take w_B.

Worked example: one AdaIN, one truncation, one demodulation

pixelx(x − μ)/σAdaIN output
12−0.4472−1.1708
240.44720.1708
361.34161.5125
40−1.3416−2.5125
AdaIN on one 2×2 feature map with style (y_s, y_b) = (1.5, −0.5) (computed in Python)

What each step does to the numbers

  1. 1Statistics: μ=3\mu=3, σ=(1+1+9+9)/4=5=2.2361\sigma=\sqrt{(1+1+9+9)/4}=\sqrt5=2.2361 (population standard deviation, as in instance norm).
  2. 2After AdaIN the map has mean yb=−0.5y_b=-0.5 and standard deviation ys=1.5y_s=1.5, whatever its statistics were before. This is why one layer's style cannot leak into the next: the next AdaIN overwrites both statistics.
  3. 3Truncation with wˉ=(0.2,−0.1)\bar w=(0.2,-0.1), w=(1.4,0.5)w=(1.4,0.5), ψ=0.7\psi=0.7: w′=(0.2+0.7⋅1.2, −0.1+0.7⋅0.6)=(1.04, 0.32)w'=(0.2+0.7\cdot1.2,\ -0.1+0.7\cdot0.6)=(1.04,\ 0.32).
  4. 4StyleGAN2 demodulation for one output channel with weights wik=(0.5−0.51.00.5)w_{ik}=\begin{pmatrix}0.5&-0.5\\1.0&0.5\end{pmatrix} (input channel ii, tap kk) and style s=(2,0.5)s=(2,0.5): modulated wik′=siwik=(1−10.50.25)w'_{ik}=s_iw_{ik}=\begin{pmatrix}1&-1\\0.5&0.25\end{pmatrix}.
  5. 5∑w′2=2.3125=1.5207\sqrt{\sum w'^2}=\sqrt{2.3125}=1.5207, so w′′=(0.6576−0.65760.32880.1644)w''=\begin{pmatrix}0.6576&-0.6576\\0.3288&0.1644\end{pmatrix}. Its squared norm is 1, so unit-variance i.i.d. inputs give a unit-variance output. This is the same normalisation as AdaIN, but done on the weights from expected statistics rather than on the activations.

StyleGAN2: weight demodulation

StyleGAN images often show blob-shaped "water droplet" artifacts. Karras et al. ("Analyzing and Improving the Image Quality of StyleGAN", CVPR 2020) traced them to AdaIN. Normalising each feature map separately destroys information carried in the relative magnitudes of features. The generator learns to sneak that information past the normalisation with a strong, localised spike that dominates the statistics. StyleGAN2 removes the normalisation of activations. The style modulates the convolution weights instead, and demodulation rescales each output channel to unit expected norm:

wijk′=si⋅wijk,wijk′′=wijk′∑i,kwijk′2+ϵw'_{ijk}=s_i\cdot w_{ijk},\qquad w''_{ijk}=\frac{w'_{ijk}}{\sqrt{\sum_{i,k}{w'_{ijk}}^2+\epsilon}}
wijkw_{ijk}
convolution weight from input channel ii to output channel jj at spatial tap kk
sis_i
style scale for input channel ii, from the affine map of ww
w′′w''
demodulated weight: assuming i.i.d. unit-variance inputs, every output channel has unit standard deviation

StyleGAN2 also moves the noise and bias outside the style block and modulates only the standard deviation. It adds a path-length regulariser that makes a fixed step in W\mathcal W produce a fixed-size change in the image, and replaces progressive growing with skip-connection or residual architectures. On FFHQ at 102421024^2, FID fell from 4.40 to 2.84.

python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3class Mapping(nn.Module): # f: Z → W, 8 fully connected layers, 512 → 512
4 def __init__(self, d=512, n=8):
5 super().__init__()
6 L = []
7 for _ in range(n):
8 L += [nn.Linear(d, d), nn.LeakyReLU(0.2)]
9 self.net = nn.Sequential(*L)
10 def forward(self, z):
11 z = z * torch.rsqrt(z.pow(2).mean(1, keepdim=True) + 1e-8) # normalise z first
12 return self.net(z)
13
14class StyleLayer(nn.Module): # 3×3 conv → noise → LeakyReLU → AdaIN with style from w
15 def __init__(self, cin, cout, d=512):
16 super().__init__()
17 self.conv = nn.Conv2d(cin, cout, 3, padding=1)
18 self.noise_scale = nn.Parameter(torch.zeros(1, cout, 1, 1)) # "B": learned per-channel scale
19 self.affine = nn.Linear(d, 2 * cout) # "A": w ↦ (y_s, y_b)
20 nn.init.ones_(self.affine.bias[:cout]) # start with y_s ≈ 1
21 def forward(self, x, w, conv=True):
22 if conv:
23 x = self.conv(x)
24 n = torch.randn(x.size(0), 1, x.size(2), x.size(3), device=x.device) # one noise image, all channels
25 x = F.leaky_relu(x + self.noise_scale * n, 0.2)
26 ys, yb = self.affine(w).chunk(2, 1)
27 return ys[..., None, None] * F.instance_norm(x) + yb[..., None, None] # AdaIN
28
29class Synthesis(nn.Module):
30 def __init__(self, chans=(512, 512, 256, 128), d=512): # 4², 8², 16², 32² for the demo
31 super().__init__()
32 self.const = nn.Parameter(torch.ones(1, chans[0], 4, 4)) # learned constant input
33 self.layers = nn.ModuleList()
34 for i, c in enumerate(chans):
35 cin = chans[max(i - 1, 0)]
36 self.layers.append(StyleLayer(cin, c, d)) # first layer of 4² skips its conv
37 self.layers.append(StyleLayer(c, c, d))
38 self.to_rgb = nn.Conv2d(chans[-1], 3, 1)
39 def forward(self, ws): # ws: (N, n_layers, d), one w per layer
40 x = self.const.expand(ws.size(0), -1, -1, -1)
41 for i, layer in enumerate(self.layers):
42 if i % 2 == 0 and i > 0:
43 x = F.interpolate(x, scale_factor=2, mode="bilinear", align_corners=False)
44 x = layer(x, ws[:, i], conv=(i > 0))
45 return self.to_rgb(x)
46
47mapping, synth = Mapping(), Synthesis()
48L = len(synth.layers) # 8 here; 18 at 1024²
49z1, z2 = torch.randn(4, 512), torch.randn(4, 512)
50w1, w2 = mapping(z1), mapping(z2)
51
52# truncation in W: pull towards the running mean of w (estimated from many samples)
53with torch.no_grad():
54 w_avg = mapping(torch.randn(10000, 512)).mean(0)
55psi = 0.7
56w1_t = w_avg + psi * (w1 - w_avg)
57
58# style mixing: layers [0, c) from w1, [c, L) from w2
59c = 4
60ws = torch.stack([w1_t] * c + [w2] * (L - c), 1) # (4, 8, 512)
61img = synth(ws)
62print(ws.shape, img.shape) # torch.Size([4, 8, 512]) torch.Size([4, 3, 32, 32])
63print(f"mapping {sum(p.numel() for p in mapping.parameters()) / 1e6:.2f}M parameters") # 2.10M
64
65# StyleGAN2: modulate and demodulate the conv weights instead of AdaIN on activations
66def modulated_conv(x, weight, s, eps=1e-8): # x (N, cin, H, W), weight (cout, cin, k, k), s (N, cin)
67 N, cin, H, W = x.shape
68 wgt = weight[None] * s[:, None, :, None, None] # w'_ijk = s_i · w_ijk
69 wgt = wgt * torch.rsqrt(wgt.pow(2).sum((2, 3, 4), keepdim=True) + eps) # w'' = w' / ‖w'_j‖
70 out = F.conv2d(x.reshape(1, N * cin, H, W), wgt.reshape(-1, cin, *weight.shape[2:]),
71 padding=weight.shape[-1] // 2, groups=N) # one weight set per sample
72 return out.reshape(N, -1, H, W)
73
74y = modulated_conv(torch.randn(4, 64, 16, 16), torch.randn(32, 64, 3, 3), torch.rand(4, 64) * 3)
75print(y.shape, round(y.std().item(), 1)) # torch.Size([4, 32, 16, 16]) 1.0
Mapping network, AdaIN style layers with noise, truncation in W, style mixing, and a StyleGAN2 modulated convolution (32² demo; run with PyTorch 2 on CPU)

Use when

  • +Scale-specific control: coarse, middle and fine attributes through per-layer styles
  • +More disentangled latent space W\mathcal W (shorter perceptual paths, better linear separability)
  • +Noise inputs separate stochastic detail from identity
  • +Better FID than Progressive GAN on CelebA-HQ and FFHQ at almost the same parameter count

Avoid when

  • −AdaIN causes water-droplet artifacts (fixed in StyleGAN2 by weight demodulation)
  • −Progressive growing causes phase artifacts, such as teeth that do not follow pose; StyleGAN2 dropped it
  • −Truncation improves typical quality at the cost of diversity
  • −Expensive to train at 1024², and strongest on aligned, single-category data such as faces

Check yourself

Which StyleGAN component is responsible for the exact placement of individual hairs and freckles?

Check yourself

You generate a face from w_A, but feed w_B to layers 0–3 (4² and 8²). What should change?

Remember

  • ✓StyleGAN generator: w=f(z)w=f(z) (8-layer MLP, 512→512512\to512), then per layer AdaIN(xi,y)=ys,ixi−μ(xi)σ(xi)+yb,i\mathrm{AdaIN}(x_i,y)=y_{s,i}\frac{x_i-\mu(x_i)}{\sigma(x_i)}+y_{b,i} with (ys,yb)=Aℓ(w)(y_s,y_b)=A_\ell(w), starting from a learned constant 4×4×5124\times4\times512.
  • ✓Truncation in W\mathcal W: w′=wˉ+ψ(w−wˉ)w'=\bar w+\psi(w-\bar w), ψ<1\psi<1, trades diversity for quality; style mixing feeds w1w_1/w2w_2 to layers before/after a crossover (regulariser on 90% of images).
  • ✓Per-pixel noise at every layer controls stochastic detail (hair, freckles) without changing identity.
  • ✓18 style inputs at 1024²: coarse 4²–8² (pose, face shape), middle 16²–32² (features, hair style), fine 64²–1024² (colour, microstructure).
  • ✓FFHQ FID 8.04 (Progressive GAN) → 4.40 (StyleGAN); StyleGAN2's weight demodulation removes droplet artifacts and reaches 2.84.
← back to Deep Learning roadmap