# StyleGAN

> StyleGAN (Karras, Laine, Aila, CVPR 2019) maps z through an 8-layer MLP to an intermediate latent w. It injects w as per-layer AdaIN styles into a synthesis network that starts from a learned constant, adds per-pixel noise for stochastic detail, and supports style mixing and truncation in W.

Source: https://www.mlfoundry.in/learn/stylegan  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**StyleGAN:** is the style-based GAN generator of Karras, Laine and Aila, "A Style-Based Generator Architecture for Generative Adversarial Networks" (CVPR 2019). It never feeds the latent $z$ directly into the image network. An 8-layer **mapping network** first turns $z\in\mathbb R^{512}$ into an intermediate latent $w\in\mathcal W$. A learned affine map then turns $w$ into a **style** $(y_s,y_b)$ for every layer of the **synthesis network**, applied by adaptive instance normalisation (AdaIN). The synthesis network starts from a **learned constant** $4\times4\times512$ tensor, and per-pixel **noise** is added at every layer to produce stochastic detail.

By 2018 GANs could produce convincing $1024\times1024$ faces. Progressive GAN (Karras et al., ICLR 2018) grew generator and discriminator together from $4^2$ to $1024^2$. The generator itself was still a black box: $z$ entered at the first layer, and nothing controlled *which* layer decided pose, identity, or freckles. Tero Karras, Samuli Laine and Timo Aila (NVIDIA) redesigned only the generator, borrowing from style transfer. In Huang and Belongie's AdaIN (2017), the "style" of an image is the per-channel mean and standard deviation of its feature maps. StyleGAN lets $w$ set those statistics separately at every resolution. The discriminator, loss and training schedule are kept from Progressive GAN. The paper also introduced the **FFHQ** dataset and two measures of latent-space disentanglement, **perceptual path length** and **linear separability**. Its successors, StyleGAN2 (2020) and StyleGAN3 (2021), defined the state of the art for unconditional image synthesis for several years.

## Mapping network and AdaIN

$$
w=f(z),\quad f:\mathbb R^{512}\to\mathbb R^{512}\ \text{(8 FC layers)},\qquad y=(y_s,\,y_b)=A_\ell(w),\qquad \mathrm{AdaIN}(x_i,y)=y_{s,i}\,\frac{x_i-\mu(x_i)}{\sigma(x_i)}+y_{b,i}
$$

- $z\sim\mathcal N(0,I)$: input latent, normalised before the first FC layer
- $f$: mapping network, 8 fully connected layers of width 512 with LeakyReLU; its learning rate is $100\times$ lower than the rest
- $A_\ell$: learned affine transform for layer $\ell$ that "specialises" $w$ into one scale and one bias per channel
- $x_i$: feature map $i$ of the current layer; $\mu,\sigma$ are its spatial mean and standard deviation
- $\mathrm{AdaIN}$: normalise each feature map, then impose the style's scale and bias; the style controls this layer only, until the next AdaIN overwrites the statistics

Why an intermediate latent? $z$ has a fixed prior, so the generator must warp a Gaussian onto the data. Suppose some attribute combination is rare in the data, such as long-haired men in the paper's example. In $\mathcal Z$ that region must be squeezed, and the mapping from $z$ to image becomes curved and entangled. $\mathcal W$ has no fixed distribution. The learned $f$ can "unwarp" it so that factors of variation become more linear. The paper argues this is easier for the generator and measures it.

## Synthesis network: constant input, styles, noise

The $1024^2$ synthesis network has **18 layers**, two per resolution from $4^2$ to $1024^2$, each followed by AdaIN. The first takes a learned constant $4\times4\times512$ tensor instead of $z$. Removing the traditional input did not hurt FID. The image is controlled entirely by the styles and the noise. Upsampling uses bilinear filtering. A final $1\times1$ convolution (toRGB) produces the image. Before each AdaIN, a single-channel image of uncorrelated Gaussian noise is added to the feature maps, scaled by learned per-channel factors (the "B" blocks in the paper's Figure 1). The noise controls only **stochastic** detail: the exact placement of hairs, freckles, stubble and skin pores. Identity and pose are unaffected. Because every layer has fresh noise, the network does not need to generate pseudo-randomness from earlier activations. Coarse-layer noise yields large-scale curls, and fine-layer noise yields fine strands.

The synthesis network has 26.2M trainable parameters, against 23.1M for the traditional generator. The style mechanism is cheap.

## Style mixing

Each style is used by one layer and then normalised away, so styles are localised. This allows **style mixing**: map two latents $z_1,z_2$ to $w_1,w_2$, and feed $w_1$ to the layers before a random crossover point and $w_2$ to the rest. The paper's Figure 3 shows the effect. Coarse styles ($4^2$–$8^2$) from source B carry pose, general hair style, face shape and eyeglasses. Middle styles ($16^2$–$32^2$) carry smaller facial features, hair style, and eyes open or closed. Fine styles ($64^2$–$1024^2$) carry mainly the colour scheme and microstructure. As a **regulariser**, a given percentage of training images are generated with two latents. The default is 90%. This stops the network from assuming that adjacent styles are correlated.

## Truncation in W

$$
\bar w=\mathbb E_{z\sim p(z)}\big[f(z)\big],\qquad w'=\bar w+\psi\,(w-\bar w),\qquad \psi<1
$$

- $\bar w$: centre of mass of $\mathcal W$, estimated as a running average of $f(z)$ during training
- $\psi$: truncation strength: $\psi=1$ no change, $\psi=0$ every sample becomes the "mean face", $\psi<0$ gives faces with opposite attributes
- $w'$: can be applied only to some layers; the paper's sample figures use ψ = 0.7 on the 4²–32² layers

Low-density regions of the data are poorly represented, so samples drawn from them look worse. Truncation trades variety for average quality by shrinking samples toward the dense centre. BigGAN (see **BigGAN**) truncates in $\mathcal Z$ by resampling large $|z|$. StyleGAN truncates in $\mathcal W$, where the learned centre is meaningful. All FIDs in the paper are computed **without** truncation.

## Measuring disentanglement: perceptual path length

$$
l_{\mathcal W}=\mathbb E\Big[\frac1{\epsilon^2}\,d\big(g(\mathrm{lerp}(f(z_1),f(z_2);t)),\ g(\mathrm{lerp}(f(z_1),f(z_2);t+\epsilon))\big)\Big],\qquad t\sim U(0,1),\ \epsilon=10^{-4}
$$

- $g$: the synthesis network
- $d$: LPIPS perceptual distance (VGG16 features) between the two images, cropped to the face
- $\mathrm{lerp}$: linear interpolation in $\mathcal W$; for $\mathcal Z$ the paper uses spherical interpolation (slerp) of $z$
- $l$: small path length means images change smoothly along latent lines; sharp changes betray entanglement

A second measure, **linear separability**, trains 40 linear SVMs on $w$ (or $z$) to predict CelebA attributes labelled by auxiliary classifiers. It reports $\exp\big(\sum_i H(Y_i\mid X_i)\big)$, the conditional entropy of the classifier's label given the SVM's side of the hyperplane. Lower values mean more linear factors. With the full 8-layer mapping network the style-based generator reached path length 234.0 in $\mathcal W$ and separability 3.79. The traditional generator scored 412.0 and 10.78.

**Setup in the paper**

- **FFHQ:** Flickr-Faces-HQ: 70,000 face images at $1024^2$, aligned and cropped, with more variation in age, ethnicity, background and accessories than CelebA-HQ
- **Latents:** $z,w\in\mathbb R^{512}$; 18 copies of $w$ (one per layer) at $1024^2$
- **Mapping:** 8 FC layers, learning rate multiplied by $0.01$
- **Losses:** WGAN-GP on CelebA-HQ; non-saturating loss with $R_1$ regularisation ($\gamma=10$) on FFHQ
- **Training:** progressive growing as in Progressive GAN; about one week on 8 Tesla V100 GPUs
- **Regularisation:** style mixing on 90% of training images
- **Evaluation:** FID on 50,000 random images, no truncation

FID (lower is better) as the generator changes step by step (Karras et al. 2019, Table 1)

| configuration | CelebA-HQ | FFHQ |
| --- | --- | --- |
| A  baseline Progressive GAN | 7.79 | 8.04 |
| B  + tuning (incl. bilinear up/down) | 6.11 | 5.25 |
| C  + mapping network and styles | 5.34 | 4.85 |
| D  + remove traditional input | 5.07 | 4.88 |
| E  + noise inputs | 5.06 | 4.42 |
| F  + mixing regularisation | 5.17 | 4.40 |

> Figure: The left panel is a 2-D stand-in for W: 160 samples z ~ N(0, I) pushed through a fixed warp f(z) = (z₁ + z₂²/2, z₂ + 0.4 sin 2z₁). The shading is a kernel density estimate of p_W from 500 more samples. The mean is w̄ = E[f(z)] = (0.5, 0), whereas f(0) = (0, 0): the centre of Z is not the centre of W, so truncation must use w̄. Truncation w' = w̄ + ψ(w − w̄) scales the mean distance to w̄ exactly linearly: 1.418 at ψ = 1, 0.993 at ψ = 0.7, 0.709 at ψ = 0.5. The mean log-density of the samples rises from −2.907 to −2.477 and −2.279, and at ψ = 0 every sample sits at w̄ (−2.077). The number of samples in rare regions (log p < −4) falls from 22 of 160 to 7 and then 2. These are the regions a generator learns worst, so this is the diversity-for-quality trade. Negative ψ reflects samples through w̄. The right panel lists the 18 style inputs of the 1024² synthesis network with feature maps per resolution (512 up to 32², then 256, 128, 64, 32, 16). Style-mixing presets send w_B to the coarse (layers 0–3, 4²–8²), middle (4–7, 16²–32²) or fine (8–17, 64²–1024²) layers and list what the paper reports those layers control. The crossover slider gives the training-time mixing rule: layers before c take w_A, the rest take w_B.

## Worked example: one AdaIN, one truncation, one demodulation

AdaIN on one 2×2 feature map with style (y_s, y_b) = (1.5, −0.5) (computed in Python)

| pixel | x | (x − μ)/σ | AdaIN output |
| --- | --- | --- | --- |
| 1 | 2 | −0.4472 | −1.1708 |
| 2 | 4 | 0.4472 | 0.1708 |
| 3 | 6 | 1.3416 | 1.5125 |
| 4 | 0 | −1.3416 | −2.5125 |

**What each step does to the numbers**

1. Statistics: $\mu=3$, $\sigma=\sqrt{(1+1+9+9)/4}=\sqrt5=2.2361$ (population standard deviation, as in instance norm).
2. After AdaIN the map has mean $y_b=-0.5$ and standard deviation $y_s=1.5$, whatever its statistics were before. This is why one layer's style cannot leak into the next: the next AdaIN overwrites both statistics.
3. Truncation with $\bar w=(0.2,-0.1)$, $w=(1.4,0.5)$, $\psi=0.7$: $w'=(0.2+0.7\cdot1.2,\ -0.1+0.7\cdot0.6)=(1.04,\ 0.32)$.
4. StyleGAN2 demodulation for one output channel with weights $w_{ik}=\begin{pmatrix}0.5&-0.5\\1.0&0.5\end{pmatrix}$ (input channel $i$, tap $k$) and style $s=(2,0.5)$: modulated $w'_{ik}=s_iw_{ik}=\begin{pmatrix}1&-1\\0.5&0.25\end{pmatrix}$.
5. $\sqrt{\sum w'^2}=\sqrt{2.3125}=1.5207$, so $w''=\begin{pmatrix}0.6576&-0.6576\\0.3288&0.1644\end{pmatrix}$. Its squared norm is 1, so unit-variance i.i.d. inputs give a unit-variance output. This is the same normalisation as AdaIN, but done on the weights from expected statistics rather than on the activations.

## StyleGAN2: weight demodulation

StyleGAN images often show blob-shaped **"water droplet"** artifacts. Karras et al. ("Analyzing and Improving the Image Quality of StyleGAN", CVPR 2020) traced them to AdaIN. Normalising each feature map separately destroys information carried in the relative magnitudes of features. The generator learns to sneak that information past the normalisation with a strong, localised spike that dominates the statistics. StyleGAN2 removes the normalisation of activations. The style **modulates** the convolution weights instead, and **demodulation** rescales each output channel to unit expected norm:

$$
w'_{ijk}=s_i\cdot w_{ijk},\qquad w''_{ijk}=\frac{w'_{ijk}}{\sqrt{\sum_{i,k}{w'_{ijk}}^2+\epsilon}}
$$

- $w_{ijk}$: convolution weight from input channel $i$ to output channel $j$ at spatial tap $k$
- $s_i$: style scale for input channel $i$, from the affine map of $w$
- $w''$: demodulated weight: assuming i.i.d. unit-variance inputs, every output channel has unit standard deviation

StyleGAN2 also moves the noise and bias outside the style block and modulates only the standard deviation. It adds a path-length regulariser that makes a fixed step in $\mathcal W$ produce a fixed-size change in the image, and replaces progressive growing with skip-connection or residual architectures. On FFHQ at $1024^2$, FID fell from 4.40 to 2.84.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

class Mapping(nn.Module):                 # f: Z → W, 8 fully connected layers, 512 → 512
    def __init__(self, d=512, n=8):
        super().__init__()
        L = []
        for _ in range(n):
            L += [nn.Linear(d, d), nn.LeakyReLU(0.2)]
        self.net = nn.Sequential(*L)
    def forward(self, z):
        z = z * torch.rsqrt(z.pow(2).mean(1, keepdim=True) + 1e-8)   # normalise z first
        return self.net(z)

class StyleLayer(nn.Module):              # 3×3 conv → noise → LeakyReLU → AdaIN with style from w
    def __init__(self, cin, cout, d=512):
        super().__init__()
        self.conv = nn.Conv2d(cin, cout, 3, padding=1)
        self.noise_scale = nn.Parameter(torch.zeros(1, cout, 1, 1))  # "B": learned per-channel scale
        self.affine = nn.Linear(d, 2 * cout)                          # "A": w ↦ (y_s, y_b)
        nn.init.ones_(self.affine.bias[:cout])                        # start with y_s ≈ 1
    def forward(self, x, w, conv=True):
        if conv:
            x = self.conv(x)
        n = torch.randn(x.size(0), 1, x.size(2), x.size(3), device=x.device)  # one noise image, all channels
        x = F.leaky_relu(x + self.noise_scale * n, 0.2)
        ys, yb = self.affine(w).chunk(2, 1)
        return ys[..., None, None] * F.instance_norm(x) + yb[..., None, None]  # AdaIN

class Synthesis(nn.Module):
    def __init__(self, chans=(512, 512, 256, 128), d=512):            # 4², 8², 16², 32² for the demo
        super().__init__()
        self.const = nn.Parameter(torch.ones(1, chans[0], 4, 4))      # learned constant input
        self.layers = nn.ModuleList()
        for i, c in enumerate(chans):
            cin = chans[max(i - 1, 0)]
            self.layers.append(StyleLayer(cin, c, d))                  # first layer of 4² skips its conv
            self.layers.append(StyleLayer(c, c, d))
        self.to_rgb = nn.Conv2d(chans[-1], 3, 1)
    def forward(self, ws):                # ws: (N, n_layers, d), one w per layer
        x = self.const.expand(ws.size(0), -1, -1, -1)
        for i, layer in enumerate(self.layers):
            if i % 2 == 0 and i > 0:
                x = F.interpolate(x, scale_factor=2, mode="bilinear", align_corners=False)
            x = layer(x, ws[:, i], conv=(i > 0))
        return self.to_rgb(x)

mapping, synth = Mapping(), Synthesis()
L = len(synth.layers)                                       # 8 here; 18 at 1024²
z1, z2 = torch.randn(4, 512), torch.randn(4, 512)
w1, w2 = mapping(z1), mapping(z2)

# truncation in W: pull towards the running mean of w (estimated from many samples)
with torch.no_grad():
    w_avg = mapping(torch.randn(10000, 512)).mean(0)
psi = 0.7
w1_t = w_avg + psi * (w1 - w_avg)

# style mixing: layers [0, c) from w1, [c, L) from w2
c = 4
ws = torch.stack([w1_t] * c + [w2] * (L - c), 1)            # (4, 8, 512)
img = synth(ws)
print(ws.shape, img.shape)                                  # torch.Size([4, 8, 512]) torch.Size([4, 3, 32, 32])
print(f"mapping {sum(p.numel() for p in mapping.parameters()) / 1e6:.2f}M parameters")  # 2.10M

# StyleGAN2: modulate and demodulate the conv weights instead of AdaIN on activations
def modulated_conv(x, weight, s, eps=1e-8):  # x (N, cin, H, W), weight (cout, cin, k, k), s (N, cin)
    N, cin, H, W = x.shape
    wgt = weight[None] * s[:, None, :, None, None]                       # w'_ijk = s_i · w_ijk
    wgt = wgt * torch.rsqrt(wgt.pow(2).sum((2, 3, 4), keepdim=True) + eps)  # w'' = w' / ‖w'_j‖
    out = F.conv2d(x.reshape(1, N * cin, H, W), wgt.reshape(-1, cin, *weight.shape[2:]),
                   padding=weight.shape[-1] // 2, groups=N)               # one weight set per sample
    return out.reshape(N, -1, H, W)

y = modulated_conv(torch.randn(4, 64, 16, 16), torch.randn(32, 64, 3, 3), torch.rand(4, 64) * 3)
print(y.shape, round(y.std().item(), 1))                    # torch.Size([4, 32, 16, 16]) 1.0
```

Mapping network, AdaIN style layers with noise, truncation in W, style mixing, and a StyleGAN2 modulated convolution (32² demo; run with PyTorch 2 on CPU)

> **Truncate around the mean of W, not at z = 0:** $f(0)$ is not $\bar w$, because a nonlinear map does not preserve means. In the figure $f(0)=(0,0)$ while $\bar w=(0.5,0)$. Shrinking $z$ toward 0 and then mapping is a different, uncontrolled operation. Use the running average $\bar w$ that StyleGAN implementations track during training, apply truncation only where you want it (often the coarse layers), and report FID without truncation.

> **Styles are local because AdaIN forgets:** Each AdaIN overwrites the mean and scale of every feature map, so a style can affect only the convolutions between its AdaIN and the next one. That is what makes coarse/middle/fine mixing work. It also explains StyleGAN2's droplet artifacts: to pass information that normalisation would erase, the generator learned a localised spike that dominates the statistics.

> **W versus W+:** Sampling uses one $w$ copied to all 18 layers, and mixing uses two. GAN-inversion and editing methods often optimise a separate $w$ for every layer, a space usually called $\mathcal W+$ ($18\times512$ at $1024^2$). It reconstructs real photos far better but leaves the region that the generator was trained on, so edits can be less reliable.

**Strengths**

- Scale-specific control: coarse, middle and fine attributes through per-layer styles
- More disentangled latent space $\mathcal W$ (shorter perceptual paths, better linear separability)
- Noise inputs separate stochastic detail from identity
- Better FID than Progressive GAN on CelebA-HQ and FFHQ at almost the same parameter count

**Limitations**

- AdaIN causes water-droplet artifacts (fixed in StyleGAN2 by weight demodulation)
- Progressive growing causes phase artifacts, such as teeth that do not follow pose; StyleGAN2 dropped it
- Truncation improves typical quality at the cost of diversity
- Expensive to train at 1024², and strongest on aligned, single-category data such as faces

**Question:** Which StyleGAN component is responsible for the exact placement of individual hairs and freckles?

- A. The mapping network
- B. The coarse styles at 4²–8²
- C. The per-layer noise inputs, scaled by learned per-channel factors
- D. The truncation trick

**Answer:** C. The per-layer noise inputs, scaled by learned per-channel factors — Fresh Gaussian noise is added at every layer. Resampling it changes only stochastic detail (hair placement, freckles, pores) and leaves identity and pose unchanged. Coarse styles control pose and face shape, and truncation moves $w$ toward $\bar w$.

**Question:** You generate a face from w_A, but feed w_B to layers 0–3 (4² and 8²). What should change?

- A. Only colour scheme and skin microstructure
- B. Pose, general hair style, face shape and eyeglasses take B's; finer features and colours stay A's
- C. Nothing, because styles are averaged
- D. Only the background, because coarse layers handle it

**Answer:** B. Pose, general hair style, face shape and eyeglasses take B's; finer features and colours stay A's — In the paper's style-mixing figure, coarse styles ($4^2$–$8^2$) from source B transfer high-level aspects such as pose, general hair style, face shape and eyeglasses. The other layers, including the fine ones that set the colour scheme, still use $w_A$.

**Key points**

- StyleGAN generator: $w=f(z)$ (8-layer MLP, $512\to512$), then per layer $\mathrm{AdaIN}(x_i,y)=y_{s,i}\frac{x_i-\mu(x_i)}{\sigma(x_i)}+y_{b,i}$ with $(y_s,y_b)=A_\ell(w)$, starting from a learned constant $4\times4\times512$.
- Truncation in $\mathcal W$: $w'=\bar w+\psi(w-\bar w)$, $\psi<1$, trades diversity for quality; style mixing feeds $w_1$/$w_2$ to layers before/after a crossover (regulariser on 90% of images).
- Per-pixel noise at every layer controls stochastic detail (hair, freckles) without changing identity.
- 18 style inputs at 1024²: coarse 4²–8² (pose, face shape), middle 16²–32² (features, hair style), fine 64²–1024² (colour, microstructure).
- FFHQ FID 8.04 (Progressive GAN) → 4.40 (StyleGAN); StyleGAN2's weight demodulation removes droplet artifacts and reaches 2.84.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
