Definition
Conditional GAN (cGAN) A cGAN is a Generative adversarial network (GAN) in which both players receive extra information (Mirza & Osindero, "Conditional Generative Adversarial Nets", arXiv 2014). The generator maps to a sample , which defines . The discriminator scores a pair, : is a real sample for this ? The value function is the GAN objective with every term conditioned on :
- class label, attribute vector, text embedding or image; drawn from the data's for the fake term too
- a sample for condition ; supplies the variation that does not determine
- probability that is a real pair; a real image with the wrong label should be rejected
Goodfellow et al. (2014) listed conditioning as the first extension of their framework. Mehdi Mirza and Simon Osindero tried it on MNIST digits and on image tagging. On MNIST, the one-hot label went into both networks, and the generator produced the requested digit on demand, which an unconditional GAN cannot do. Conditioning on images rather than labels later gave image-to-image translation (Pix2Pix). Conditioning on text gave text-to-image GANs (Reed et al., 2016). Class conditioning at scale gave BigGAN. The question that drove the design work after Mirza and Osindero was how to give to the discriminator.
What the optimal discriminator sees
The analysis of the unconditional game carries over, applied to the joint distribution. Let the fakes be pairs with , so their joint density is . For fixed :
- real joint distribution; the generator's, with the same marginal
- zero only if for every ; a generator that ignores is penalised even if its images look right
- Bayes' rule with ; the first term is all an unconditional discriminator can estimate, the second needs
The split is the key to the later designs. An unconditional only learns . A generator that draws realistic images but ignores its label has , and such a is exactly indifferent to it. The figure shows this case: the joint JS is 0.2681 nats while the marginal JS is 0. To catch label errors, must estimate the class term , which is a comparison of two classifiers.
Four ways to condition
| method | generator | discriminator | notes |
|---|---|---|---|
| Concatenation (Mirza & Osindero 2014) | concat [z; embed(y)] at the input | concat [x; y] at the input or a hidden layer | simplest; for images y is tiled as extra channels; D can learn to ignore y |
| AC-GAN (Odena, Olah & Shlens, ICML 2017) | concat or one-hot in z | unconditional real/fake head + auxiliary classifier head | both players also minimise class cross-entropy; sharp, but less diverse within a class |
| Projection (Miyato & Koyama, ICLR 2018) | conditional batch norm | the form of the optimal logit under a log-linear class model; used by SNGAN, SAGAN and BigGAN | |
| Conditional normalisation (de Vries et al. 2017; Dumoulin et al. 2017) | in every norm layer | (any of the above) | y modulates every layer instead of only the input; BigGAN feeds a shared class embedding plus a chunk of z |
AC-GAN. The discriminator returns a source probability and a class distribution . With and , maximises and maximises . The auxiliary classifier makes produce samples that are easy to classify. That raised image quality on ImageNet: the authors trained 100 AC-GANs of 10 classes each at 128×128. It also biases toward the confident interior of each class and away from class boundaries, which reduces intra-class diversity (Gong et al., 2019, TAC-GAN).
Projection discriminator. Miyato and Koyama start from the optimal logit above. They model both and as softmax classifiers on a shared feature . The class term then becomes plus a function of alone, which merges into . So the right discriminator has the form : an unconditional head plus an inner product between a class embedding and the features. Concatenating at the input lets build the comparison in any way, including not at all. Projection builds it into the architecture. With spectral normalisation and the hinge loss it became the standard class-conditional discriminator.
Mirza & Osindero (2014), MNIST setup
- Generator
- uniform in the unit hypercube. and the one-hot are mapped to ReLU layers of 200 and 1000 units, then to a joint ReLU layer of 1200 units and a 784-unit sigmoid output
- Discriminator
- to a maxout layer of 240 units × 5 pieces, to a maxout layer of 50 units × 5 pieces, then a joint maxout layer of 240 units × 4 pieces and a sigmoid
- Evaluation
- Parzen-window log-likelihood 132 ± 1.8, below the unconditional GAN's 225 ± 2. The authors presented the results as a proof of concept
- Second experiment
- MIR Flickr 25,000: generate word-vector tags conditioned on ImageNet-pretrained CNN image features (multimodal, one-to-many labelling)
ε = 2/3 makes G ignore its label entirely. With δ = 0 its samples still match the data marginal, so only a discriminator that sees y can tell.
Worked example: a generator that ignores y
Two classes with and two possible images . The data has , and , . The generator ignores its label: . Its marginal equals .
| (x, y) | p(x, y) | q(x, y) | D*(x, y) | log p(x)/q(x) | log p(y|x)/q(y|x) | logit |
|---|---|---|---|---|---|---|
| (a, 0) | 0.45 | 0.25 | 0.6429 | 0 | 0.5878 | 0.5878 |
| (b, 0) | 0.05 | 0.25 | 0.1667 | 0 | −1.6094 | −1.6094 |
| (a, 1) | 0.05 | 0.25 | 0.1667 | 0 | −1.6094 | −1.6094 |
| (b, 1) | 0.45 | 0.25 | 0.6429 | 0 | 0.5878 | 0.5878 |
What each discriminator concludes
- 1Unconditional: for both images. Its loss is already at the equilibrium value , and it gives no gradient.
- 2Conditional: the mismatched pairs and get . A label-ignoring generator produces them half the time, while real data produces them 10% of the time.
- 3 nats, so , below the equilibrium . The conditional game still has work to do.
- 4The whole logit is the class term . For example for . A projection discriminator represents it as .
- 5AC-GAN view: a classifier trained on reals assigns the fakes their own label only 50% of the time. Its cross-entropy on fakes is 1.204 nats against 0.3251 on reals. The generator's class loss pushes it toward , or further, toward the most classifiable image.
1import torch, torch.nn as nn, torch.nn.functional as F2from torch.nn.utils import spectral_norm as SN34n_cls, nz, d = 10, 64, 784 # e.g. MNIST, flattened56class CondBatchNorm(nn.Module):7 """BatchNorm without its own affine; gamma and beta are looked up per class."""8 def __init__(self, ch, n_cls):9 super().__init__()10 self.bn = nn.BatchNorm1d(ch, affine=False)11 self.gamma, self.beta = nn.Embedding(n_cls, ch), nn.Embedding(n_cls, ch)12 nn.init.ones_(self.gamma.weight); nn.init.zeros_(self.beta.weight)13 def forward(self, h, y):14 return self.gamma(y) * self.bn(h) + self.beta(y)1516class CondG(nn.Module):17 def __init__(self, h=256):18 super().__init__()19 self.emb = nn.Embedding(n_cls, 32)20 self.fc1 = nn.Linear(nz + 32, h) # Mirza & Osindero: condition by concatenation21 self.cbn = CondBatchNorm(h, n_cls) # and/or modulate normalisation by y22 self.fc2 = nn.Linear(h, d)23 def forward(self, z, y):24 h = self.fc1(torch.cat([z, self.emb(y)], 1))25 return torch.tanh(self.fc2(F.relu(self.cbn(h, y))))2627class ProjectionD(nn.Module):28 """f(x, y) = psi(phi(x)) + embed(y) . phi(x) (Miyato & Koyama 2018)"""29 def __init__(self, h=256):30 super().__init__()31 self.phi = nn.Sequential(SN(nn.Linear(d, h)), nn.LeakyReLU(0.2), SN(nn.Linear(h, h)), nn.LeakyReLU(0.2))32 self.psi = SN(nn.Linear(h, 1))33 self.embed = SN(nn.Embedding(n_cls, h))34 def forward(self, x, y):35 f = self.phi(x) # (B, h)36 return self.psi(f).squeeze(1) + (self.embed(y) * f).sum(1)3738class ACGanD(nn.Module):39 """AC-GAN: unconditional real/fake logit plus an auxiliary class head."""40 def __init__(self, h=256):41 super().__init__()42 self.phi = nn.Sequential(nn.Linear(d, h), nn.LeakyReLU(0.2))43 self.src, self.cls = nn.Linear(h, 1), nn.Linear(h, n_cls)44 def forward(self, x):45 f = self.phi(x)46 return self.src(f).squeeze(1), self.cls(f)4748G, D = CondG(), ProjectionD()49B = 1650z, y = torch.randn(B, nz), torch.randint(0, n_cls, (B,))51x_fake = G(z, y)52print(x_fake.shape, D(x_fake, y).shape) # torch.Size([16, 784]) torch.Size([16])5354# hinge loss, as used with the projection discriminator55x_real, y_real = torch.rand(B, d) * 2 - 1, torch.randint(0, n_cls, (B,))56loss_d = F.relu(1 - D(x_real, y_real)).mean() + F.relu(1 + D(x_fake.detach(), y)).mean()57loss_g = -D(x_fake, y).mean()58print(loss_d.shape, loss_g.shape) # torch.Size([]) torch.Size([])5960# AC-GAN losses: D maximises L_S + L_C, G maximises L_C - L_S (written here as minimisation)61A = ACGanD()62s_r, c_r = A(x_real); s_f, c_f = A(x_fake.detach())63bce = nn.BCEWithLogitsLoss()64loss_d_ac = bce(s_r, torch.ones(B)) + bce(s_f, torch.zeros(B)) + F.cross_entropy(c_r, y_real) + F.cross_entropy(c_f, y)65s_g, c_g = A(x_fake)66loss_g_ac = bce(s_g, torch.ones(B)) + F.cross_entropy(c_g, y) # non-saturating source term + class term67print(c_r.shape, loss_d_ac.dim(), loss_g_ac.dim()) # torch.Size([16, 10]) 0 0Use when
- +Control: sample from for a chosen label, text or input image
- +The label gives D extra information, which often stabilises training and improves quality
- +Projection D and conditional BN add few parameters and scale to 1000 classes
- +The same framework covers class-conditional, text-to-image and image-to-image models
Avoid when
- −Needs paired (x, y) data; the label path in D can be ignored if it is weak
- −AC-GAN-style class losses trade intra-class diversity for classifiability
- −Strong conditions make G ignore z (little diversity per condition)
- −Inherits GAN instability and mode collapse, now per condition
Check yourself
A conditional generator produces perfectly realistic images, but each image's class is unrelated to the requested label. What does the optimal unconditional discriminator do?
Check yourself
What form does the projection discriminator give to the conditional logit?
Remember
- ✓cGAN game: ; with optimal it minimises .
- ✓Optimal logit : an unconditional term plus a class term only a -aware can estimate.
- ✓Conditioning methods: concatenation (Mirza & Osindero 2014), AC-GAN auxiliary classifier (Odena et al. 2017), projection (Miyato & Koyama 2018), conditional BN in .
- ✓AC-GAN's class loss makes samples easy to classify but less diverse within a class.
- ✓Condition on images and you get Pix2Pix; on text, text-to-image GANs; on 1000 ImageNet classes with projection D and conditional BN, BigGAN.