Conditional GAN (cGAN)

A conditional GAN gives side information y (a label, text or an image) to both networks and plays the GAN game on pairs (x, y), so G models p(x|y). y can enter by concatenation, an auxiliary classifier (AC-GAN), a projection term yᵀVφ(x), or conditional batch norm.

14 min read960 wordsadvanced0 viewsupdated 2026-10-12

Definition

Conditional GAN (cGAN) A cGAN is a Generative adversarial network (GAN) in which both players receive extra information yy (Mirza & Osindero, "Conditional Generative Adversarial Nets", arXiv 2014). The generator maps (z,y)(z,y) to a sample G(z,y)G(z,y), which defines pg(x∣y)p_g(x\mid y). The discriminator scores a pair, D(x,y)D(x,y): is xx a real sample for this yy? The value function is the GAN objective with every term conditioned on yy:

min⁡Gmax⁡D E(x,y)∼pdata[log⁡D(x,y)]+Ey∼p(y), z∼p(z)[log⁡(1−D(G(z,y),y))]\min_G\max_D\ \mathbb E_{(x,y)\sim p_{\text{data}}}\big[\log D(x,y)\big]+\mathbb E_{y\sim p(y),\,z\sim p(z)}\big[\log\big(1-D(G(z,y),y)\big)\big]
yy
class label, attribute vector, text embedding or image; drawn from the data's p(y)p(y) for the fake term too
G(z,y)G(z,y)
a sample for condition yy; zz supplies the variation that yy does not determine
D(x,y)D(x,y)
probability that (x,y)(x,y) is a real pair; a real image with the wrong label should be rejected

Goodfellow et al. (2014) listed conditioning as the first extension of their framework. Mehdi Mirza and Simon Osindero tried it on MNIST digits and on image tagging. On MNIST, the one-hot label went into both networks, and the generator produced the requested digit on demand, which an unconditional GAN cannot do. Conditioning on images rather than labels later gave image-to-image translation (Pix2Pix). Conditioning on text gave text-to-image GANs (Reed et al., 2016). Class conditioning at scale gave BigGAN. The question that drove the design work after Mirza and Osindero was how to give yy to the discriminator.

What the optimal discriminator sees

The analysis of the unconditional game carries over, applied to the joint distribution. Let the fakes be pairs (G(z,y),y)(G(z,y),y) with y∼p(y)y\sim p(y), so their joint density is q(x,y)=p(y) q(x∣y)q(x,y)=p(y)\,q(x\mid y). For fixed GG:

D∗(x,y)=p(x,y)p(x,y)+q(x,y),V(D∗,G)=−log⁡4+2 JS(p(x,y) ∥ q(x,y)),log⁡D∗1−D∗=log⁡p(x)q(x)⏟unconditional+log⁡p(y∣x)q(y∣x)⏟class termD^*(x,y)=\frac{p(x,y)}{p(x,y)+q(x,y)},\qquad V(D^*,G)=-\log4+2\,\mathrm{JS}\big(p(x,y)\,\|\,q(x,y)\big),\qquad \log\frac{D^*}{1-D^*}=\underbrace{\log\frac{p(x)}{q(x)}}_{\text{unconditional}}+\underbrace{\log\frac{p(y\mid x)}{q(y\mid x)}}_{\text{class term}}
p(x,y)p(x,y)
real joint distribution; q(x,y)q(x,y) the generator's, with the same marginal p(y)p(y)
JS(p(x,y)∥q(x,y))\mathrm{JS}(p(x,y)\|q(x,y))
zero only if q(x∣y)=p(x∣y)q(x\mid y)=p(x\mid y) for every yy; a generator that ignores yy is penalised even if its images look right
split\text{split}
Bayes' rule with q(y)=p(y)q(y)=p(y); the first term is all an unconditional discriminator can estimate, the second needs yy

The split is the key to the later designs. An unconditional DD only learns log⁡p(x)/q(x)\log p(x)/q(x). A generator that draws realistic images but ignores its label has q(x)=p(x)q(x)=p(x), and such a DD is exactly indifferent to it. The figure shows this case: the joint JS is 0.2681 nats while the marginal JS is 0. To catch label errors, DD must estimate the class term log⁡p(y∣x)/q(y∣x)\log p(y\mid x)/q(y\mid x), which is a comparison of two classifiers.

Four ways to condition

methodgeneratordiscriminatornotes
Concatenation (Mirza & Osindero 2014)concat [z; embed(y)] at the inputconcat [x; y] at the input or a hidden layersimplest; for images y is tiled as extra channels; D can learn to ignore y
AC-GAN (Odena, Olah & Shlens, ICML 2017)concat or one-hot in zunconditional real/fake head + auxiliary classifier head p^(y∣x)\hat p(y\mid x)both players also minimise class cross-entropy; sharp, but less diverse within a class
Projection (Miyato & Koyama, ICLR 2018)conditional batch normf(x,y)=ψ(ϕ(x))+embed(y)⊤ϕ(x)f(x,y)=\psi(\phi(x))+\mathrm{embed}(y)^\top\phi(x)the form of the optimal logit under a log-linear class model; used by SNGAN, SAGAN and BigGAN
Conditional normalisation (de Vries et al. 2017; Dumoulin et al. 2017)γ(y)⊙h^+β(y)\gamma(y)\odot\hat h+\beta(y) in every norm layer(any of the above)y modulates every layer instead of only the input; BigGAN feeds a shared class embedding plus a chunk of z
How y enters the networks

AC-GAN. The discriminator returns a source probability P(S∣x)P(S\mid x) and a class distribution P(C∣x)P(C\mid x). With LS=E[log⁡P(S=real∣xreal)]+E[log⁡P(S=fake∣xfake)]L_S=\mathbb E[\log P(S=\text{real}\mid x_{\text{real}})]+\mathbb E[\log P(S=\text{fake}\mid x_{\text{fake}})] and LC=E[log⁡P(C=c∣xreal)]+E[log⁡P(C=c∣xfake)]L_C=\mathbb E[\log P(C=c\mid x_{\text{real}})]+\mathbb E[\log P(C=c\mid x_{\text{fake}})], DD maximises LS+LCL_S+L_C and GG maximises LC−LSL_C-L_S. The auxiliary classifier makes GG produce samples that are easy to classify. That raised image quality on ImageNet: the authors trained 100 AC-GANs of 10 classes each at 128×128. It also biases q(x∣y)q(x\mid y) toward the confident interior of each class and away from class boundaries, which reduces intra-class diversity (Gong et al., 2019, TAC-GAN).

Projection discriminator. Miyato and Koyama start from the optimal logit above. They model both p(y∣x)p(y\mid x) and q(y∣x)q(y\mid x) as softmax classifiers on a shared feature ϕ(x)\phi(x). The class term then becomes y⊤(Vp−Vq)ϕ(x)y^\top(V^p-V^q)\phi(x) plus a function of xx alone, which merges into log⁡p(x)/q(x)\log p(x)/q(x). So the right discriminator has the form f(x,y)=y⊤Vϕ(x)+ψ(ϕ(x))f(x,y)=y^\top V\phi(x)+\psi(\phi(x)): an unconditional head plus an inner product between a class embedding and the features. Concatenating yy at the input lets DD build the comparison in any way, including not at all. Projection builds it into the architecture. With spectral normalisation and the hinge loss it became the standard class-conditional discriminator.

Mirza & Osindero (2014), MNIST setup

Generator
z∈R100z\in\mathbb R^{100} uniform in the unit hypercube. zz and the one-hot yy are mapped to ReLU layers of 200 and 1000 units, then to a joint ReLU layer of 1200 units and a 784-unit sigmoid output
Discriminator
xx to a maxout layer of 240 units × 5 pieces, yy to a maxout layer of 50 units × 5 pieces, then a joint maxout layer of 240 units × 4 pieces and a sigmoid
Evaluation
Parzen-window log-likelihood 132 ± 1.8, below the unconditional GAN's 225 ± 2. The authors presented the results as a proof of concept
Second experiment
MIR Flickr 25,000: generate word-vector tags conditioned on ImageNet-pretrained CNN image features (multimodal, one-to-many labelling)
log p(x|y)/q(x|y): optimal conditional D logit for this y · + real-looking, − fake-lookingy=0y=1y=2○ real, y = 0 · ● G(z, y = 0) · clipped at ±3q(x|y) = (1 − ε)·own + ε/2·others, shift δε = 0.667 · δ = 0.00 · G ignores yJS(p(x) ‖ q(x)), marginal0.0000JS(p(x,y) ‖ q(x,y)), joint0.2681fakes classified as their y33.3%AC-GAN class loss, fakes4.660same loss on reals0.158← all an unconditional D can see← what a conditional D sees(real data: 94.0%)optimal logits for y = 0 at x = μ_kx = μ_kuncond+ class= condμ_00.001.101.10μ_10.00−5.93−5.93μ_20.00−5.54−5.54D*(x, y) = p(x, y) / (p(x, y) + q(x, y)); with q(y) = p(y) its logit splits into two terms:log p(x|y)/q(x|y) = log p(x)/q(x) + log p(y|x)/q(y|x)projection D: f(x, y) = ψ(φ(x)) + embed(y)ᵀφ(x), the same two-term form3 classes, p(y) = ⅓, p(x|y) = N(μ_y, 0.4²I); JS on a 160² grid over [−2, 2]²AC-GAN: the real-data classifier p(y|x) scores fakes (cross-entropy, nats)
heat map
label y

ε = 2/3 makes G ignore its label entirely. With δ = 0 its samples still match the data marginal, so only a discriminator that sees y can tell.

fig.Three classes with p(y) = ⅓ and p(x|y) = N(μ_y, 0.4²I), against a conditional generator that puts 1 − ε of its mass on the requested class's mode and ε/2 on each other mode, all shifted by δ. Every density, optimal logit and divergence is exact (integrated on a 160×160 grid over [−2, 2]²). At the default ε = 2/3, δ = 0 the generator ignores its label completely, yet its images are distributed exactly like the data. The unconditional optimal logit log p(x)/q(x) is 0 everywhere and the marginal JS is 0.0000, so an unconditional D cannot detect anything. The joint JS is 0.2681 nats. The conditional logit for y = 0 is +1.10 at μ₀, −5.93 at μ₁ and −5.54 at μ₂, and all of it is class term. The real-data classifier labels only 33.3% of the fakes with their requested class (real data: 94.0%), and the AC-GAN class loss on fakes is 4.660 nats against 0.158 on reals. Lower ε to 0.2: the joint JS falls to 0.0579 and accuracy rises to 75.8%, while the marginal JS stays 0. At ε = 0, δ = 0.3 the error is in the images, not the labels. Both divergences see it (marginal 0.0453, joint 0.0650), and at μ₀ the logit is almost entirely unconditional (+0.283, class term −0.002).

Worked example: a generator that ignores y

Two classes with p(y)=12p(y)=\tfrac12 and two possible images a,ba,b. The data has p(a∣0)=0.9p(a\mid0)=0.9, p(b∣0)=0.1p(b\mid0)=0.1 and p(a∣1)=0.1p(a\mid1)=0.1, p(b∣1)=0.9p(b\mid1)=0.9. The generator ignores its label: q(a∣y)=q(b∣y)=0.5q(a\mid y)=q(b\mid y)=0.5. Its marginal q(x)=(0.5,0.5)q(x)=(0.5,0.5) equals p(x)=(0.5,0.5)p(x)=(0.5,0.5).

(x, y)p(x, y)q(x, y)D*(x, y)log p(x)/q(x)log p(y|x)/q(y|x)logit
(a, 0)0.450.250.642900.58780.5878
(b, 0)0.050.250.16670−1.6094−1.6094
(a, 1)0.050.250.16670−1.6094−1.6094
(b, 1)0.450.250.642900.58780.5878
Optimal conditional discriminator and its two terms (computed in Python)

What each discriminator concludes

  1. 1Unconditional: D∗(x)=p(x)/(p(x)+q(x))=0.5D^*(x)=p(x)/(p(x)+q(x))=0.5 for both images. Its loss is already at the equilibrium value log⁡4\log4, and it gives GG no gradient.
  2. 2Conditional: the mismatched pairs (b,0)(b,0) and (a,1)(a,1) get D∗=0.1667D^*=0.1667. A label-ignoring generator produces them half the time, while real data produces them 10% of the time.
  3. 3JS(p(x,y)∥q(x,y))=0.1017\mathrm{JS}(p(x,y)\|q(x,y))=0.1017 nats, so V(D∗,G)=−log⁡4+2(0.1017)=−1.1828V(D^*,G)=-\log4+2(0.1017)=-1.1828, below the equilibrium −1.3863-1.3863. The conditional game still has work to do.
  4. 4The whole logit is the class term log⁡p(y∣x)/q(y∣x)\log p(y\mid x)/q(y\mid x). For example log⁡(0.9/0.5)=0.5878\log(0.9/0.5)=0.5878 for (a,0)(a,0). A projection discriminator represents it as embed(y)⊤ϕ(x)\mathrm{embed}(y)^\top\phi(x).
  5. 5AC-GAN view: a classifier trained on reals assigns the fakes their own label only 50% of the time. Its cross-entropy on fakes is 1.204 nats against 0.3251 on reals. The generator's class loss pushes it toward q(x∣y)=p(x∣y)q(x\mid y)=p(x\mid y), or further, toward the most classifiable image.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torch.nn.utils import spectral_norm as SN
3
4n_cls, nz, d = 10, 64, 784 # e.g. MNIST, flattened
5
6class CondBatchNorm(nn.Module):
7 """BatchNorm without its own affine; gamma and beta are looked up per class."""
8 def __init__(self, ch, n_cls):
9 super().__init__()
10 self.bn = nn.BatchNorm1d(ch, affine=False)
11 self.gamma, self.beta = nn.Embedding(n_cls, ch), nn.Embedding(n_cls, ch)
12 nn.init.ones_(self.gamma.weight); nn.init.zeros_(self.beta.weight)
13 def forward(self, h, y):
14 return self.gamma(y) * self.bn(h) + self.beta(y)
15
16class CondG(nn.Module):
17 def __init__(self, h=256):
18 super().__init__()
19 self.emb = nn.Embedding(n_cls, 32)
20 self.fc1 = nn.Linear(nz + 32, h) # Mirza & Osindero: condition by concatenation
21 self.cbn = CondBatchNorm(h, n_cls) # and/or modulate normalisation by y
22 self.fc2 = nn.Linear(h, d)
23 def forward(self, z, y):
24 h = self.fc1(torch.cat([z, self.emb(y)], 1))
25 return torch.tanh(self.fc2(F.relu(self.cbn(h, y))))
26
27class ProjectionD(nn.Module):
28 """f(x, y) = psi(phi(x)) + embed(y) . phi(x) (Miyato & Koyama 2018)"""
29 def __init__(self, h=256):
30 super().__init__()
31 self.phi = nn.Sequential(SN(nn.Linear(d, h)), nn.LeakyReLU(0.2), SN(nn.Linear(h, h)), nn.LeakyReLU(0.2))
32 self.psi = SN(nn.Linear(h, 1))
33 self.embed = SN(nn.Embedding(n_cls, h))
34 def forward(self, x, y):
35 f = self.phi(x) # (B, h)
36 return self.psi(f).squeeze(1) + (self.embed(y) * f).sum(1)
37
38class ACGanD(nn.Module):
39 """AC-GAN: unconditional real/fake logit plus an auxiliary class head."""
40 def __init__(self, h=256):
41 super().__init__()
42 self.phi = nn.Sequential(nn.Linear(d, h), nn.LeakyReLU(0.2))
43 self.src, self.cls = nn.Linear(h, 1), nn.Linear(h, n_cls)
44 def forward(self, x):
45 f = self.phi(x)
46 return self.src(f).squeeze(1), self.cls(f)
47
48G, D = CondG(), ProjectionD()
49B = 16
50z, y = torch.randn(B, nz), torch.randint(0, n_cls, (B,))
51x_fake = G(z, y)
52print(x_fake.shape, D(x_fake, y).shape) # torch.Size([16, 784]) torch.Size([16])
53
54# hinge loss, as used with the projection discriminator
55x_real, y_real = torch.rand(B, d) * 2 - 1, torch.randint(0, n_cls, (B,))
56loss_d = F.relu(1 - D(x_real, y_real)).mean() + F.relu(1 + D(x_fake.detach(), y)).mean()
57loss_g = -D(x_fake, y).mean()
58print(loss_d.shape, loss_g.shape) # torch.Size([]) torch.Size([])
59
60# AC-GAN losses: D maximises L_S + L_C, G maximises L_C - L_S (written here as minimisation)
61A = ACGanD()
62s_r, c_r = A(x_real); s_f, c_f = A(x_fake.detach())
63bce = nn.BCEWithLogitsLoss()
64loss_d_ac = bce(s_r, torch.ones(B)) + bce(s_f, torch.zeros(B)) + F.cross_entropy(c_r, y_real) + F.cross_entropy(c_f, y)
65s_g, c_g = A(x_fake)
66loss_g_ac = bce(s_g, torch.ones(B)) + F.cross_entropy(c_g, y) # non-saturating source term + class term
67print(c_r.shape, loss_d_ac.dim(), loss_g_ac.dim()) # torch.Size([16, 10]) 0 0
Conditional generator with conditional batch norm, a projection discriminator, and AC-GAN losses

Use when

  • +Control: sample from p(x∣y)p(x\mid y) for a chosen label, text or input image
  • +The label gives D extra information, which often stabilises training and improves quality
  • +Projection D and conditional BN add few parameters and scale to 1000 classes
  • +The same framework covers class-conditional, text-to-image and image-to-image models

Avoid when

  • −Needs paired (x, y) data; the label path in D can be ignored if it is weak
  • −AC-GAN-style class losses trade intra-class diversity for classifiability
  • −Strong conditions make G ignore z (little diversity per condition)
  • −Inherits GAN instability and mode collapse, now per condition

Check yourself

A conditional generator produces perfectly realistic images, but each image's class is unrelated to the requested label. What does the optimal unconditional discriminator do?

Check yourself

What form does the projection discriminator give to the conditional logit?

Remember

  • ✓cGAN game: min⁡Gmax⁡D Ep(x,y)[log⁡D(x,y)]+Ep(y),p(z)[log⁡(1−D(G(z,y),y))]\min_G\max_D\ \mathbb E_{p(x,y)}[\log D(x,y)]+\mathbb E_{p(y),p(z)}[\log(1-D(G(z,y),y))]; with optimal DD it minimises JS(p(x,y)∥q(x,y))\mathrm{JS}(p(x,y)\|q(x,y)).
  • ✓Optimal logit =log⁡p(x)q(x)+log⁡p(y∣x)q(y∣x)=\log\frac{p(x)}{q(x)}+\log\frac{p(y\mid x)}{q(y\mid x)}: an unconditional term plus a class term only a yy-aware DD can estimate.
  • ✓Conditioning methods: concatenation (Mirza & Osindero 2014), AC-GAN auxiliary classifier (Odena et al. 2017), projection ψ(ϕ(x))+embed(y)⊤ϕ(x)\psi(\phi(x))+\mathrm{embed}(y)^\top\phi(x) (Miyato & Koyama 2018), conditional BN in GG.
  • ✓AC-GAN's class loss makes samples easy to classify but less diverse within a class.
  • ✓Condition on images and you get Pix2Pix; on text, text-to-image GANs; on 1000 ImageNet classes with projection D and conditional BN, BigGAN.
← back to Deep Learning roadmap