# Conditional GAN (cGAN)

> A conditional GAN gives side information y (a label, text or an image) to both networks and plays the GAN game on pairs (x, y), so G models p(x|y). y can enter by concatenation, an auxiliary classifier (AC-GAN), a projection term yᵀVφ(x), or conditional batch norm.

Source: https://www.mlfoundry.in/learn/conditional-gan-cgan  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 14 min read · updated 2026-10-12

**Conditional GAN (cGAN):** A cGAN is a **Generative adversarial network (GAN)** in which both players receive extra information $y$ (Mirza & Osindero, "Conditional Generative Adversarial Nets", arXiv 2014). The generator maps $(z,y)$ to a sample $G(z,y)$, which defines $p_g(x\mid y)$. The discriminator scores a pair, $D(x,y)$: is $x$ a real sample **for this $y$**? The value function is the GAN objective with every term conditioned on $y$:

$$
\min_G\max_D\ \mathbb E_{(x,y)\sim p_{\text{data}}}\big[\log D(x,y)\big]+\mathbb E_{y\sim p(y),\,z\sim p(z)}\big[\log\big(1-D(G(z,y),y)\big)\big]
$$

- $y$: class label, attribute vector, text embedding or image; drawn from the data's $p(y)$ for the fake term too
- $G(z,y)$: a sample for condition $y$; $z$ supplies the variation that $y$ does not determine
- $D(x,y)$: probability that $(x,y)$ is a real pair; a real image with the wrong label should be rejected

Goodfellow et al. (2014) listed conditioning as the first extension of their framework. Mehdi Mirza and Simon Osindero tried it on MNIST digits and on image tagging. On MNIST, the one-hot label went into both networks, and the generator produced the requested digit on demand, which an unconditional GAN cannot do. Conditioning on images rather than labels later gave image-to-image translation (**Pix2Pix**). Conditioning on text gave text-to-image GANs (Reed et al., 2016). Class conditioning at scale gave **BigGAN**. The question that drove the design work after Mirza and Osindero was **how** to give $y$ to the discriminator.

## What the optimal discriminator sees

The analysis of the unconditional game carries over, applied to the joint distribution. Let the fakes be pairs $(G(z,y),y)$ with $y\sim p(y)$, so their joint density is $q(x,y)=p(y)\,q(x\mid y)$. For fixed $G$:

$$
D^*(x,y)=\frac{p(x,y)}{p(x,y)+q(x,y)},\qquad V(D^*,G)=-\log4+2\,\mathrm{JS}\big(p(x,y)\,\|\,q(x,y)\big),\qquad \log\frac{D^*}{1-D^*}=\underbrace{\log\frac{p(x)}{q(x)}}_{\text{unconditional}}+\underbrace{\log\frac{p(y\mid x)}{q(y\mid x)}}_{\text{class term}}
$$

- $p(x,y)$: real joint distribution; $q(x,y)$ the generator's, with the same marginal $p(y)$
- $\mathrm{JS}(p(x,y)\|q(x,y))$: zero only if $q(x\mid y)=p(x\mid y)$ for every $y$; a generator that ignores $y$ is penalised even if its images look right
- $\text{split}$: Bayes' rule with $q(y)=p(y)$; the first term is all an unconditional discriminator can estimate, the second needs $y$

The split is the key to the later designs. An unconditional $D$ only learns $\log p(x)/q(x)$. A generator that draws realistic images but ignores its label has $q(x)=p(x)$, and such a $D$ is exactly indifferent to it. The figure shows this case: the joint JS is 0.2681 nats while the marginal JS is 0. To catch label errors, $D$ must estimate the class term $\log p(y\mid x)/q(y\mid x)$, which is a comparison of two classifiers.

## Four ways to condition

How y enters the networks

| method | generator | discriminator | notes |
| --- | --- | --- | --- |
| Concatenation (Mirza & Osindero 2014) | concat [z; embed(y)] at the input | concat [x; y] at the input or a hidden layer | simplest; for images y is tiled as extra channels; D can learn to ignore y |
| AC-GAN (Odena, Olah & Shlens, ICML 2017) | concat or one-hot in z | unconditional real/fake head + auxiliary classifier head $\hat p(y\mid x)$ | both players also minimise class cross-entropy; sharp, but less diverse within a class |
| Projection (Miyato & Koyama, ICLR 2018) | conditional batch norm | $f(x,y)=\psi(\phi(x))+\mathrm{embed}(y)^\top\phi(x)$ | the form of the optimal logit under a log-linear class model; used by SNGAN, SAGAN and BigGAN |
| Conditional normalisation (de Vries et al. 2017; Dumoulin et al. 2017) | $\gamma(y)\odot\hat h+\beta(y)$ in every norm layer | (any of the above) | y modulates every layer instead of only the input; BigGAN feeds a shared class embedding plus a chunk of z |

**AC-GAN.** The discriminator returns a source probability $P(S\mid x)$ and a class distribution $P(C\mid x)$. With $L_S=\mathbb E[\log P(S=\text{real}\mid x_{\text{real}})]+\mathbb E[\log P(S=\text{fake}\mid x_{\text{fake}})]$ and $L_C=\mathbb E[\log P(C=c\mid x_{\text{real}})]+\mathbb E[\log P(C=c\mid x_{\text{fake}})]$, $D$ maximises $L_S+L_C$ and $G$ maximises $L_C-L_S$. The auxiliary classifier makes $G$ produce samples that are easy to classify. That raised image quality on ImageNet: the authors trained 100 AC-GANs of 10 classes each at 128×128. It also biases $q(x\mid y)$ toward the confident interior of each class and away from class boundaries, which reduces intra-class diversity (Gong et al., 2019, TAC-GAN).

**Projection discriminator.** Miyato and Koyama start from the optimal logit above. They model both $p(y\mid x)$ and $q(y\mid x)$ as softmax classifiers on a shared feature $\phi(x)$. The class term then becomes $y^\top(V^p-V^q)\phi(x)$ plus a function of $x$ alone, which merges into $\log p(x)/q(x)$. So the right discriminator has the form $f(x,y)=y^\top V\phi(x)+\psi(\phi(x))$: an unconditional head plus an inner product between a class embedding and the features. Concatenating $y$ at the input lets $D$ build the comparison in any way, including not at all. Projection builds it into the architecture. With spectral normalisation and the hinge loss it became the standard class-conditional discriminator.

**Mirza & Osindero (2014), MNIST setup**

- **Generator:** $z\in\mathbb R^{100}$ uniform in the unit hypercube. $z$ and the one-hot $y$ are mapped to ReLU layers of 200 and 1000 units, then to a joint ReLU layer of 1200 units and a 784-unit sigmoid output
- **Discriminator:** $x$ to a maxout layer of 240 units × 5 pieces, $y$ to a maxout layer of 50 units × 5 pieces, then a joint maxout layer of 240 units × 4 pieces and a sigmoid
- **Evaluation:** Parzen-window log-likelihood 132 ± 1.8, below the unconditional GAN's 225 ± 2. The authors presented the results as a proof of concept
- **Second experiment:** MIR Flickr 25,000: generate word-vector tags conditioned on ImageNet-pretrained CNN image features (multimodal, one-to-many labelling)

> Figure: Three classes with p(y) = ⅓ and p(x|y) = N(μ_y, 0.4²I), against a conditional generator that puts 1 − ε of its mass on the requested class's mode and ε/2 on each other mode, all shifted by δ. Every density, optimal logit and divergence is exact (integrated on a 160×160 grid over [−2, 2]²). At the default ε = 2/3, δ = 0 the generator ignores its label completely, yet its images are distributed exactly like the data. The unconditional optimal logit log p(x)/q(x) is 0 everywhere and the marginal JS is 0.0000, so an unconditional D cannot detect anything. The joint JS is 0.2681 nats. The conditional logit for y = 0 is +1.10 at μ₀, −5.93 at μ₁ and −5.54 at μ₂, and all of it is class term. The real-data classifier labels only 33.3% of the fakes with their requested class (real data: 94.0%), and the AC-GAN class loss on fakes is 4.660 nats against 0.158 on reals. Lower ε to 0.2: the joint JS falls to 0.0579 and accuracy rises to 75.8%, while the marginal JS stays 0. At ε = 0, δ = 0.3 the error is in the images, not the labels. Both divergences see it (marginal 0.0453, joint 0.0650), and at μ₀ the logit is almost entirely unconditional (+0.283, class term −0.002).

## Worked example: a generator that ignores y

Two classes with $p(y)=\tfrac12$ and two possible images $a,b$. The data has $p(a\mid0)=0.9$, $p(b\mid0)=0.1$ and $p(a\mid1)=0.1$, $p(b\mid1)=0.9$. The generator ignores its label: $q(a\mid y)=q(b\mid y)=0.5$. Its marginal $q(x)=(0.5,0.5)$ equals $p(x)=(0.5,0.5)$.

Optimal conditional discriminator and its two terms (computed in Python)

| (x, y) | p(x, y) | q(x, y) | D*(x, y) | log p(x)/q(x) | log p(y\|x)/q(y\|x) | logit |
| --- | --- | --- | --- | --- | --- | --- |
| (a, 0) | 0.45 | 0.25 | 0.6429 | 0 | 0.5878 | 0.5878 |
| (b, 0) | 0.05 | 0.25 | 0.1667 | 0 | −1.6094 | −1.6094 |
| (a, 1) | 0.05 | 0.25 | 0.1667 | 0 | −1.6094 | −1.6094 |
| (b, 1) | 0.45 | 0.25 | 0.6429 | 0 | 0.5878 | 0.5878 |

**What each discriminator concludes**

1. Unconditional: $D^*(x)=p(x)/(p(x)+q(x))=0.5$ for both images. Its loss is already at the equilibrium value $\log4$, and it gives $G$ no gradient.
2. Conditional: the mismatched pairs $(b,0)$ and $(a,1)$ get $D^*=0.1667$. A label-ignoring generator produces them half the time, while real data produces them 10% of the time.
3. $\mathrm{JS}(p(x,y)\|q(x,y))=0.1017$ nats, so $V(D^*,G)=-\log4+2(0.1017)=-1.1828$, below the equilibrium $-1.3863$. The conditional game still has work to do.
4. The whole logit is the class term $\log p(y\mid x)/q(y\mid x)$. For example $\log(0.9/0.5)=0.5878$ for $(a,0)$. A projection discriminator represents it as $\mathrm{embed}(y)^\top\phi(x)$.
5. AC-GAN view: a classifier trained on reals assigns the fakes their own label only 50% of the time. Its cross-entropy on fakes is 1.204 nats against 0.3251 on reals. The generator's class loss pushes it toward $q(x\mid y)=p(x\mid y)$, or further, toward the most classifiable image.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torch.nn.utils import spectral_norm as SN

n_cls, nz, d = 10, 64, 784                              # e.g. MNIST, flattened

class CondBatchNorm(nn.Module):
    """BatchNorm without its own affine; gamma and beta are looked up per class."""
    def __init__(self, ch, n_cls):
        super().__init__()
        self.bn = nn.BatchNorm1d(ch, affine=False)
        self.gamma, self.beta = nn.Embedding(n_cls, ch), nn.Embedding(n_cls, ch)
        nn.init.ones_(self.gamma.weight); nn.init.zeros_(self.beta.weight)
    def forward(self, h, y):
        return self.gamma(y) * self.bn(h) + self.beta(y)

class CondG(nn.Module):
    def __init__(self, h=256):
        super().__init__()
        self.emb = nn.Embedding(n_cls, 32)
        self.fc1 = nn.Linear(nz + 32, h)                # Mirza & Osindero: condition by concatenation
        self.cbn = CondBatchNorm(h, n_cls)              # and/or modulate normalisation by y
        self.fc2 = nn.Linear(h, d)
    def forward(self, z, y):
        h = self.fc1(torch.cat([z, self.emb(y)], 1))
        return torch.tanh(self.fc2(F.relu(self.cbn(h, y))))

class ProjectionD(nn.Module):
    """f(x, y) = psi(phi(x)) + embed(y) . phi(x)   (Miyato & Koyama 2018)"""
    def __init__(self, h=256):
        super().__init__()
        self.phi = nn.Sequential(SN(nn.Linear(d, h)), nn.LeakyReLU(0.2), SN(nn.Linear(h, h)), nn.LeakyReLU(0.2))
        self.psi = SN(nn.Linear(h, 1))
        self.embed = SN(nn.Embedding(n_cls, h))
    def forward(self, x, y):
        f = self.phi(x)                                  # (B, h)
        return self.psi(f).squeeze(1) + (self.embed(y) * f).sum(1)

class ACGanD(nn.Module):
    """AC-GAN: unconditional real/fake logit plus an auxiliary class head."""
    def __init__(self, h=256):
        super().__init__()
        self.phi = nn.Sequential(nn.Linear(d, h), nn.LeakyReLU(0.2))
        self.src, self.cls = nn.Linear(h, 1), nn.Linear(h, n_cls)
    def forward(self, x):
        f = self.phi(x)
        return self.src(f).squeeze(1), self.cls(f)

G, D = CondG(), ProjectionD()
B = 16
z, y = torch.randn(B, nz), torch.randint(0, n_cls, (B,))
x_fake = G(z, y)
print(x_fake.shape, D(x_fake, y).shape)                  # torch.Size([16, 784]) torch.Size([16])

# hinge loss, as used with the projection discriminator
x_real, y_real = torch.rand(B, d) * 2 - 1, torch.randint(0, n_cls, (B,))
loss_d = F.relu(1 - D(x_real, y_real)).mean() + F.relu(1 + D(x_fake.detach(), y)).mean()
loss_g = -D(x_fake, y).mean()
print(loss_d.shape, loss_g.shape)                        # torch.Size([]) torch.Size([])

# AC-GAN losses: D maximises L_S + L_C, G maximises L_C - L_S (written here as minimisation)
A = ACGanD()
s_r, c_r = A(x_real); s_f, c_f = A(x_fake.detach())
bce = nn.BCEWithLogitsLoss()
loss_d_ac = bce(s_r, torch.ones(B)) + bce(s_f, torch.zeros(B)) + F.cross_entropy(c_r, y_real) + F.cross_entropy(c_f, y)
s_g, c_g = A(x_fake)
loss_g_ac = bce(s_g, torch.ones(B)) + F.cross_entropy(c_g, y)   # non-saturating source term + class term
print(c_r.shape, loss_d_ac.dim(), loss_g_ac.dim())        # torch.Size([16, 10]) 0 0
```

Conditional generator with conditional batch norm, a projection discriminator, and AC-GAN losses

> **The generator may ignore z, or the discriminator may ignore y:** With a strong condition (a full input image in **Pix2Pix**), $G$ often learns to ignore $z$, and the "conditional distribution" becomes nearly deterministic. Isola et al. (2017) found that input noise was simply ignored and used dropout as the only source of randomness. With a weak path for $y$ in $D$ (concatenation deep in the network), the opposite happens: $D$ judges realism only, and nothing forces samples to match their labels. Check both. Vary $z$ at a fixed $y$, and classify the samples with an independent classifier.

> **Why projection beats concatenation:** The optimal logit is a sum of a label-free term and a log ratio of two classifiers. For softmax classifiers, that ratio is linear in the one-hot $y$: an inner product $y^\top W\phi(x)$. The projection discriminator hard-codes this structure and leaves $\phi$ and $\psi$ free. A concatenation discriminator has to discover it, and an AC-GAN replaces the ratio $p(y\mid x)/q(y\mid x)$ by $p(y\mid x)$ alone. The missing $q(y\mid x)$ term is what lets AC-GAN reward over-confident samples.

> **Conditional normalisation is the generator-side twin:** In $G$, conditioning only the input lets the signal fade with depth. Conditional batch norm replaces BN's learned $\gamma,\beta$ with class-dependent ones, $\gamma(y)\odot\frac{h-\mu}{\sigma}+\beta(y)$, in every layer. The idea comes from de Vries et al. (2017) for language-conditioned vision and Dumoulin et al. (2017) for style transfer. BigGAN computes $\gamma$ and $\beta$ by linear maps of a shared class embedding concatenated with a chunk of $z$. **StyleGAN**'s AdaIN is the same mechanism with a learned style vector in place of $y$.

**Strengths**

- Control: sample from $p(x\mid y)$ for a chosen label, text or input image
- The label gives D extra information, which often stabilises training and improves quality
- Projection D and conditional BN add few parameters and scale to 1000 classes
- The same framework covers class-conditional, text-to-image and image-to-image models

**Limitations**

- Needs paired (x, y) data; the label path in D can be ignored if it is weak
- AC-GAN-style class losses trade intra-class diversity for classifiability
- Strong conditions make G ignore z (little diversity per condition)
- Inherits GAN instability and mode collapse, now per condition

**Question:** A conditional generator produces perfectly realistic images, but each image's class is unrelated to the requested label. What does the optimal unconditional discriminator do?

- A. It rejects the fakes because the labels are wrong
- B. Nothing: $q(x)=p(x)$, so $D^*(x)=\tfrac12$ everywhere and $G$ gets no gradient
- C. It learns the class term on its own
- D. It outputs 0 for every fake

**Answer:** B. Nothing: $q(x)=p(x)$, so $D^*(x)=\tfrac12$ everywhere and $G$ gets no gradient — An unconditional $D$ only sees $x$, and the marginal of the fakes matches the data. Only $D(x,y)$ can estimate the class term $\log p(y\mid x)/q(y\mid x)$. In the figure at $\varepsilon=2/3$ the marginal JS is 0 while the joint JS is 0.2681 nats.

**Question:** What form does the projection discriminator give to the conditional logit?

- A. $f(x,y)=\psi(\phi(x))+\mathrm{embed}(y)^\top\phi(x)$
- B. $f(x,y)=\psi([\phi(x);y])$
- C. $f(x,y)=\log\hat p(y\mid x)$ only
- D. $f(x,y)=\psi(\phi(x))\cdot\mathrm{embed}(y)$

**Answer:** A. $f(x,y)=\psi(\phi(x))+\mathrm{embed}(y)^\top\phi(x)$ — It mirrors the decomposition $\log p(x)/q(x)+\log p(y\mid x)/q(y\mid x)$. If both class posteriors are softmax classifiers on a shared feature $\phi(x)$, the second term is linear in the one-hot $y$, an inner product with a class embedding. $\psi([\phi(x);y])$ is concatenation, which must discover this structure, and $\log\hat p(y\mid x)$ alone is an AC-GAN-like classifier without the unconditional term.

**Key points**

- cGAN game: $\min_G\max_D\ \mathbb E_{p(x,y)}[\log D(x,y)]+\mathbb E_{p(y),p(z)}[\log(1-D(G(z,y),y))]$; with optimal $D$ it minimises $\mathrm{JS}(p(x,y)\|q(x,y))$.
- Optimal logit $=\log\frac{p(x)}{q(x)}+\log\frac{p(y\mid x)}{q(y\mid x)}$: an unconditional term plus a class term only a $y$-aware $D$ can estimate.
- Conditioning methods: concatenation (Mirza & Osindero 2014), AC-GAN auxiliary classifier (Odena et al. 2017), projection $\psi(\phi(x))+\mathrm{embed}(y)^\top\phi(x)$ (Miyato & Koyama 2018), conditional BN in $G$.
- AC-GAN's class loss makes samples easy to classify but less diverse within a class.
- Condition on images and you get Pix2Pix; on text, text-to-image GANs; on 1000 ImageNet classes with projection D and conditional BN, BigGAN.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
