Definition
StyleGAN is the style-based GAN generator of Karras, Laine and Aila, "A Style-Based Generator Architecture for Generative Adversarial Networks" (CVPR 2019). It never feeds the latent directly into the image network. An 8-layer mapping network first turns into an intermediate latent . A learned affine map then turns into a style for every layer of the synthesis network, applied by adaptive instance normalisation (AdaIN). The synthesis network starts from a learned constant tensor, and per-pixel noise is added at every layer to produce stochastic detail.
By 2018 GANs could produce convincing faces. Progressive GAN (Karras et al., ICLR 2018) grew generator and discriminator together from to . The generator itself was still a black box: entered at the first layer, and nothing controlled which layer decided pose, identity, or freckles. Tero Karras, Samuli Laine and Timo Aila (NVIDIA) redesigned only the generator, borrowing from style transfer. In Huang and Belongie's AdaIN (2017), the "style" of an image is the per-channel mean and standard deviation of its feature maps. StyleGAN lets set those statistics separately at every resolution. The discriminator, loss and training schedule are kept from Progressive GAN. The paper also introduced the FFHQ dataset and two measures of latent-space disentanglement, perceptual path length and linear separability. Its successors, StyleGAN2 (2020) and StyleGAN3 (2021), defined the state of the art for unconditional image synthesis for several years.
Mapping network and AdaIN
- input latent, normalised before the first FC layer
- mapping network, 8 fully connected layers of width 512 with LeakyReLU; its learning rate is lower than the rest
- learned affine transform for layer that "specialises" into one scale and one bias per channel
- feature map of the current layer; are its spatial mean and standard deviation
- normalise each feature map, then impose the style's scale and bias; the style controls this layer only, until the next AdaIN overwrites the statistics
Why an intermediate latent? has a fixed prior, so the generator must warp a Gaussian onto the data. Suppose some attribute combination is rare in the data, such as long-haired men in the paper's example. In that region must be squeezed, and the mapping from to image becomes curved and entangled. has no fixed distribution. The learned can "unwarp" it so that factors of variation become more linear. The paper argues this is easier for the generator and measures it.
Synthesis network: constant input, styles, noise
The synthesis network has 18 layers, two per resolution from to , each followed by AdaIN. The first takes a learned constant tensor instead of . Removing the traditional input did not hurt FID. The image is controlled entirely by the styles and the noise. Upsampling uses bilinear filtering. A final convolution (toRGB) produces the image. Before each AdaIN, a single-channel image of uncorrelated Gaussian noise is added to the feature maps, scaled by learned per-channel factors (the "B" blocks in the paper's Figure 1). The noise controls only stochastic detail: the exact placement of hairs, freckles, stubble and skin pores. Identity and pose are unaffected. Because every layer has fresh noise, the network does not need to generate pseudo-randomness from earlier activations. Coarse-layer noise yields large-scale curls, and fine-layer noise yields fine strands.
The synthesis network has 26.2M trainable parameters, against 23.1M for the traditional generator. The style mechanism is cheap.
Style mixing
Each style is used by one layer and then normalised away, so styles are localised. This allows style mixing: map two latents to , and feed to the layers before a random crossover point and to the rest. The paper's Figure 3 shows the effect. Coarse styles (–) from source B carry pose, general hair style, face shape and eyeglasses. Middle styles (–) carry smaller facial features, hair style, and eyes open or closed. Fine styles (–) carry mainly the colour scheme and microstructure. As a regulariser, a given percentage of training images are generated with two latents. The default is 90%. This stops the network from assuming that adjacent styles are correlated.
Truncation in W
- centre of mass of , estimated as a running average of during training
- truncation strength: no change, every sample becomes the "mean face", gives faces with opposite attributes
- can be applied only to some layers; the paper's sample figures use ψ = 0.7 on the 4²–32² layers
Low-density regions of the data are poorly represented, so samples drawn from them look worse. Truncation trades variety for average quality by shrinking samples toward the dense centre. BigGAN (see BigGAN) truncates in by resampling large . StyleGAN truncates in , where the learned centre is meaningful. All FIDs in the paper are computed without truncation.
Measuring disentanglement: perceptual path length
- the synthesis network
- LPIPS perceptual distance (VGG16 features) between the two images, cropped to the face
- linear interpolation in ; for the paper uses spherical interpolation (slerp) of
- small path length means images change smoothly along latent lines; sharp changes betray entanglement
A second measure, linear separability, trains 40 linear SVMs on (or ) to predict CelebA attributes labelled by auxiliary classifiers. It reports , the conditional entropy of the classifier's label given the SVM's side of the hyperplane. Lower values mean more linear factors. With the full 8-layer mapping network the style-based generator reached path length 234.0 in and separability 3.79. The traditional generator scored 412.0 and 10.78.
Setup in the paper
- FFHQ
- Flickr-Faces-HQ: 70,000 face images at , aligned and cropped, with more variation in age, ethnicity, background and accessories than CelebA-HQ
- Latents
- ; 18 copies of (one per layer) at
- Mapping
- 8 FC layers, learning rate multiplied by
- Losses
- WGAN-GP on CelebA-HQ; non-saturating loss with regularisation () on FFHQ
- Training
- progressive growing as in Progressive GAN; about one week on 8 Tesla V100 GPUs
- Regularisation
- style mixing on 90% of training images
- Evaluation
- FID on 50,000 random images, no truncation
| configuration | CelebA-HQ | FFHQ |
|---|---|---|
| A baseline Progressive GAN | 7.79 | 8.04 |
| B + tuning (incl. bilinear up/down) | 6.11 | 5.25 |
| C + mapping network and styles | 5.34 | 4.85 |
| D + remove traditional input | 5.07 | 4.88 |
| E + noise inputs | 5.06 | 4.42 |
| F + mixing regularisation | 5.17 | 4.40 |
Crossover c: layers 0…c−1 take w_A, layers c…17 take w_B. The W panel is a 2-D toy (f(z) = (z₁ + z₂²/2, z₂ + 0.4 sin 2z₁)); the real W is 512-dimensional.
Worked example: one AdaIN, one truncation, one demodulation
| pixel | x | (x − μ)/σ | AdaIN output |
|---|---|---|---|
| 1 | 2 | −0.4472 | −1.1708 |
| 2 | 4 | 0.4472 | 0.1708 |
| 3 | 6 | 1.3416 | 1.5125 |
| 4 | 0 | −1.3416 | −2.5125 |
What each step does to the numbers
- 1Statistics: , (population standard deviation, as in instance norm).
- 2After AdaIN the map has mean and standard deviation , whatever its statistics were before. This is why one layer's style cannot leak into the next: the next AdaIN overwrites both statistics.
- 3Truncation with , , : .
- 4StyleGAN2 demodulation for one output channel with weights (input channel , tap ) and style : modulated .
- 5, so . Its squared norm is 1, so unit-variance i.i.d. inputs give a unit-variance output. This is the same normalisation as AdaIN, but done on the weights from expected statistics rather than on the activations.
StyleGAN2: weight demodulation
StyleGAN images often show blob-shaped "water droplet" artifacts. Karras et al. ("Analyzing and Improving the Image Quality of StyleGAN", CVPR 2020) traced them to AdaIN. Normalising each feature map separately destroys information carried in the relative magnitudes of features. The generator learns to sneak that information past the normalisation with a strong, localised spike that dominates the statistics. StyleGAN2 removes the normalisation of activations. The style modulates the convolution weights instead, and demodulation rescales each output channel to unit expected norm:
- convolution weight from input channel to output channel at spatial tap
- style scale for input channel , from the affine map of
- demodulated weight: assuming i.i.d. unit-variance inputs, every output channel has unit standard deviation
StyleGAN2 also moves the noise and bias outside the style block and modulates only the standard deviation. It adds a path-length regulariser that makes a fixed step in produce a fixed-size change in the image, and replaces progressive growing with skip-connection or residual architectures. On FFHQ at , FID fell from 4.40 to 2.84.
1import math, torch, torch.nn as nn, torch.nn.functional as F23class Mapping(nn.Module): # f: Z → W, 8 fully connected layers, 512 → 5124 def __init__(self, d=512, n=8):5 super().__init__()6 L = []7 for _ in range(n):8 L += [nn.Linear(d, d), nn.LeakyReLU(0.2)]9 self.net = nn.Sequential(*L)10 def forward(self, z):11 z = z * torch.rsqrt(z.pow(2).mean(1, keepdim=True) + 1e-8) # normalise z first12 return self.net(z)1314class StyleLayer(nn.Module): # 3×3 conv → noise → LeakyReLU → AdaIN with style from w15 def __init__(self, cin, cout, d=512):16 super().__init__()17 self.conv = nn.Conv2d(cin, cout, 3, padding=1)18 self.noise_scale = nn.Parameter(torch.zeros(1, cout, 1, 1)) # "B": learned per-channel scale19 self.affine = nn.Linear(d, 2 * cout) # "A": w ↦ (y_s, y_b)20 nn.init.ones_(self.affine.bias[:cout]) # start with y_s ≈ 121 def forward(self, x, w, conv=True):22 if conv:23 x = self.conv(x)24 n = torch.randn(x.size(0), 1, x.size(2), x.size(3), device=x.device) # one noise image, all channels25 x = F.leaky_relu(x + self.noise_scale * n, 0.2)26 ys, yb = self.affine(w).chunk(2, 1)27 return ys[..., None, None] * F.instance_norm(x) + yb[..., None, None] # AdaIN2829class Synthesis(nn.Module):30 def __init__(self, chans=(512, 512, 256, 128), d=512): # 4², 8², 16², 32² for the demo31 super().__init__()32 self.const = nn.Parameter(torch.ones(1, chans[0], 4, 4)) # learned constant input33 self.layers = nn.ModuleList()34 for i, c in enumerate(chans):35 cin = chans[max(i - 1, 0)]36 self.layers.append(StyleLayer(cin, c, d)) # first layer of 4² skips its conv37 self.layers.append(StyleLayer(c, c, d))38 self.to_rgb = nn.Conv2d(chans[-1], 3, 1)39 def forward(self, ws): # ws: (N, n_layers, d), one w per layer40 x = self.const.expand(ws.size(0), -1, -1, -1)41 for i, layer in enumerate(self.layers):42 if i % 2 == 0 and i > 0:43 x = F.interpolate(x, scale_factor=2, mode="bilinear", align_corners=False)44 x = layer(x, ws[:, i], conv=(i > 0))45 return self.to_rgb(x)4647mapping, synth = Mapping(), Synthesis()48L = len(synth.layers) # 8 here; 18 at 1024²49z1, z2 = torch.randn(4, 512), torch.randn(4, 512)50w1, w2 = mapping(z1), mapping(z2)5152# truncation in W: pull towards the running mean of w (estimated from many samples)53with torch.no_grad():54 w_avg = mapping(torch.randn(10000, 512)).mean(0)55psi = 0.756w1_t = w_avg + psi * (w1 - w_avg)5758# style mixing: layers [0, c) from w1, [c, L) from w259c = 460ws = torch.stack([w1_t] * c + [w2] * (L - c), 1) # (4, 8, 512)61img = synth(ws)62print(ws.shape, img.shape) # torch.Size([4, 8, 512]) torch.Size([4, 3, 32, 32])63print(f"mapping {sum(p.numel() for p in mapping.parameters()) / 1e6:.2f}M parameters") # 2.10M6465# StyleGAN2: modulate and demodulate the conv weights instead of AdaIN on activations66def modulated_conv(x, weight, s, eps=1e-8): # x (N, cin, H, W), weight (cout, cin, k, k), s (N, cin)67 N, cin, H, W = x.shape68 wgt = weight[None] * s[:, None, :, None, None] # w'_ijk = s_i · w_ijk69 wgt = wgt * torch.rsqrt(wgt.pow(2).sum((2, 3, 4), keepdim=True) + eps) # w'' = w' / ‖w'_j‖70 out = F.conv2d(x.reshape(1, N * cin, H, W), wgt.reshape(-1, cin, *weight.shape[2:]),71 padding=weight.shape[-1] // 2, groups=N) # one weight set per sample72 return out.reshape(N, -1, H, W)7374y = modulated_conv(torch.randn(4, 64, 16, 16), torch.randn(32, 64, 3, 3), torch.rand(4, 64) * 3)75print(y.shape, round(y.std().item(), 1)) # torch.Size([4, 32, 16, 16]) 1.0Use when
- +Scale-specific control: coarse, middle and fine attributes through per-layer styles
- +More disentangled latent space (shorter perceptual paths, better linear separability)
- +Noise inputs separate stochastic detail from identity
- +Better FID than Progressive GAN on CelebA-HQ and FFHQ at almost the same parameter count
Avoid when
- −AdaIN causes water-droplet artifacts (fixed in StyleGAN2 by weight demodulation)
- −Progressive growing causes phase artifacts, such as teeth that do not follow pose; StyleGAN2 dropped it
- −Truncation improves typical quality at the cost of diversity
- −Expensive to train at 1024², and strongest on aligned, single-category data such as faces
Check yourself
Which StyleGAN component is responsible for the exact placement of individual hairs and freckles?
Check yourself
You generate a face from w_A, but feed w_B to layers 0–3 (4² and 8²). What should change?
Remember
- ✓StyleGAN generator: (8-layer MLP, ), then per layer with , starting from a learned constant .
- ✓Truncation in : , , trades diversity for quality; style mixing feeds / to layers before/after a crossover (regulariser on 90% of images).
- ✓Per-pixel noise at every layer controls stochastic detail (hair, freckles) without changing identity.
- ✓18 style inputs at 1024²: coarse 4²–8² (pose, face shape), middle 16²–32² (features, hair style), fine 64²–1024² (colour, microstructure).
- ✓FFHQ FID 8.04 (Progressive GAN) → 4.40 (StyleGAN); StyleGAN2's weight demodulation removes droplet artifacts and reaches 2.84.