# Pix2Pix

> Pix2Pix (Isola et al., CVPR 2017) learns paired image-to-image translation with a conditional GAN plus an L1 term weighted λ = 100. A U-Net generator with skip connections maps x to y, and a 70×70 PatchGAN discriminator judges every patch of the pair (x, y) as real or fake.

Source: https://www.mlfoundry.in/learn/pix2pix  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 13 min read · updated 2026-10-12

**Pix2Pix:** is the general-purpose image-to-image translation method of Isola, Zhu, Zhou and Efros, "Image-to-Image Translation with Conditional Adversarial Networks" (CVPR 2017). Given **paired** training images $(x,y)$, such as a label map and its photo, it trains a generator $G$ to output $y$ from $x$. The objective is a **conditional GAN** loss, in which the discriminator sees the pair $(x,\,\cdot)$, plus $\lambda\,\|y-G(x)\|_1$ with $\lambda=100$. $G$ is a **U-Net**, an encoder–decoder with skip connections. $D$ is a **PatchGAN**, a small fully convolutional network that outputs one real/fake score per $70\times70$ patch.

Many vision problems turn one image into another: labels to photo, edges to photo, day to night, black-and-white to colour, aerial photo to map. Before 2017 each had its own architecture and hand-designed loss. A plain regression loss is the obvious common choice, and it fails in a known way. When several outputs are plausible, minimising Euclidean distance predicts their **average**, which is a blurry image. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei Efros proposed to *learn* the loss instead. The discriminator of a **Conditional GAN (cGAN)** (see also **Generative adversarial network (GAN)**) decides whether an output is a realistic answer for its input, so blur and other unrealistic outputs get penalised. One recipe, with no task-specific changes, handled all of the tasks above. Its public code (pix2pix, later pytorch-CycleGAN-and-pix2pix) set the template for **CycleGAN** and much later conditional image synthesis.

## Objective

$$
\mathcal L_{cGAN}(G,D)=\mathbb E_{x,y}\big[\log D(x,y)\big]+\mathbb E_{x,z}\big[\log\big(1-D(x,G(x,z))\big)\big],\qquad \mathcal L_{L1}(G)=\mathbb E_{x,y,z}\big[\|y-G(x,z)\|_1\big]
$$

- $x$: the input (condition) image, e.g. a semantic label map
- $y$: the paired target image; the training data are pairs $(x,y)$ from the same scene
- $z$: noise; in the final model it enters only as dropout inside $G$
- $D(x,y)$: probability that $y$ is a real output for input $x$; $x$ and $y$ are concatenated along channels
- $\|\cdot\|_1$: mean absolute pixel error; L1 rather than L2 because it blurs less

$$
G^\ast=\arg\min_G\max_D\ \mathcal L_{cGAN}(G,D)+\lambda\,\mathcal L_{L1}(G),\qquad \lambda=100
$$

- $\lambda$: weight of the reconstruction term; 100 in all experiments of the paper
- $\max_D$: in practice the loss of $D$ is halved to slow its learning relative to $G$
- $\min_G$: $G$ maximises $\log D(x,G(x,z))$ instead of minimising $\log(1-D)$: the non-saturating loss of the original GAN paper (see **Adversarial loss**)

The two terms split the work. L1 pins down the **low frequencies**: overall layout, colours, where the edges are. Its minimiser for each pixel is the median of the plausible values, so it never invents structure. The adversarial term enforces **high frequencies**, the texture that makes a patch look real. The ablation shows each term's role. L1 alone is blurry. cGAN alone is sharp but has more artifacts. An *unconditional* GAN (where $D$ sees only $y$) collapsed to producing nearly the same image for every input, because nothing forced the output to match $x$.

## Noise only as dropout

A conditional GAN normally takes noise $z$ so it can model a distribution $p(y\mid x)$. Isola et al. found that a Gaussian $z$ fed to the generator was simply ignored. The final model provides noise only as **dropout**, applied to several decoder layers at training *and* test time. It produces only minor stochasticity. Pix2Pix is close to deterministic, a known limitation: one label map maps to essentially one photo. Producing diverse outputs for the same input was left to later work.

## U-Net generator

In translation tasks the input and output share structure: the edges of a label map are the edges of the photo. A plain encoder–decoder forces all of that through a $1\times1\times512$ bottleneck. The **U-Net** adds a skip connection from encoder layer $i$ to decoder layer $n-i$, concatenating their channels, so low-level detail goes around the bottleneck. With $Ck$ = Conv–BatchNorm–ReLU with $k$ filters and $CDk$ the same with 50% dropout, all $4\times4$ with stride 2, the encoder is C64-C128-C256-C512-C512-C512-C512-C512. It takes $256^2$ down to $1^2$, using LeakyReLU 0.2 and no BatchNorm on the first layer. The decoder mirrors it with plain ReLUs and dropout in its first three blocks, then a final convolution and $\tanh$.

## PatchGAN discriminator

Because L1 already handles the low frequencies, $D$ only has to model local structure. The **PatchGAN** classifies each $N\times N$ patch as real or fake. It is a fully convolutional net, and every output unit is the logit for its receptive field. The loss averages over all units. This treats the image as a **Markov random field** in which pixels more than a patch diameter apart are independent, so the loss acts like a learned texture/style loss. The default $70\times70$ discriminator is C64-C128-C256-C512 ($4\times4$ convolutions, the first three with stride 2), then a $4\times4$ convolution to one channel, with LeakyReLU 0.2 and no BatchNorm on the first layer. For a $256^2$ input it outputs a $30\times30$ map of logits, has fewer parameters than a full-image discriminator, and runs on images of any size.

**Setup in the paper**

- **Optimiser:** Adam, learning rate $2\cdot10^{-4}$, $\beta_1=0.5$, $\beta_2=0.999$; alternate one $D$ step and one $G$ step; the objective of $D$ divided by 2
- **Batch:** 1 to 10 depending on the dataset; at batch size 1 BatchNorm becomes instance normalisation. BatchNorm uses test-batch statistics at test time, and dropout stays on
- **Images:** $256\times256$, random jitter (resize to $286^2$, crop $256^2$) and mirroring; weights initialised from $\mathcal N(0,0.02)$
- **Loss weight:** $\lambda=100$ on L1
- **Tasks:** labels→photo (Cityscapes, CMP Facades), map↔aerial photo, black-and-white→colour, edges→photo (shoes, handbags), sketch→photo, day→night, thermal→colour, inpainting
- **Speed:** the 400-image facades set trained in under two hours on one Pascal Titan X
- **Evaluation:** Amazon Mechanical Turk real-vs-fake studies, and the FCN-score: a semantic segmentation network trained on real Cityscapes photos is run on the generated photos

> Figure: The left panel is a toy 256×256 generated image G(x). Its dashed box (80×80 px, 9.77% of the image) is blurred, as an L1-only generator leaves a region where several answers were plausible. The right panel is the discriminator's output map, shaded by how much of each unit's receptive field lies in the blurred box. The geometry is computed layer by layer from the reference code. The default 70×70 PatchGAN (n = 3 strided layers) gives a 30×30 = 900 map with stride 8 px and 2,768,705 parameters. 324 of the 900 patches (36.0%) see some blur, and the mean blurred share is 0.1128. The selected unit (11, 21) covers x ∈ [145, 214], y ∈ [65, 134] and lies wholly inside the blur (share 1.000). The 1×1 PixelGAN has 256×256 outputs and only 9,024 parameters, but no patch sees any spatial structure, so it can judge colour but not sharpness. 16×16 gives 126×126 outputs (139,585 parameters). The 286×286 ImageGAN has 6×6 outputs and 11,159,361 parameters, and all 36 of its units see the blur, each mixing it with the whole image. Click a cell or move the centre sliders to trace a unit back to its patch.

## Worked example: a 70×70 receptive field and one generator loss

The receptive field grows as $r\leftarrow r+(k-1)\,J$ and the jump as $J\leftarrow J\cdot s$, starting from $r=J=1$. Every layer uses $k=4$ and padding 1. The input has 6 channels: the 3-channel condition $x$ concatenated with the 3-channel output.

70×70 PatchGAN on a 256×256 pair (computed in Node; the parameter total matches PyTorch)

| layer | k / stride | output side | jump J | receptive field | parameters |
| --- | --- | --- | --- | --- | --- |
| C64 (no BN) | 4 / 2 | 128 | 2 | 4 | 6,208 |
| C128 | 4 / 2 | 64 | 4 | 10 | 131,328 |
| C256 | 4 / 2 | 32 | 8 | 22 | 524,800 |
| C512 | 4 / 1 | 31 | 8 | 46 | 2,098,176 |
| conv → 1 logit | 4 / 1 | 30 | 8 | 70 | 8,193 |
| total |  | 30 × 30 |  | 70 × 70 | 2,768,705 |

**One generator loss on a 4-patch output map**

1. $D$ returns the patch logits $(-1.0,\ 0.0,\ 1.0,\ -2.0)$ on the fake pair $(x,G(x))$. These are probabilities $D=(0.2689,\ 0.5,\ 0.7311,\ 0.1192)$ of being real.
2. Non-saturating adversarial loss per patch: $-\log D=\mathrm{softplus}(-\ell)=(1.3133,\ 0.6931,\ 0.3133,\ 2.1269)$, mean $1.1116$. Patches that look fake (here the fourth) give the largest gradient.
3. Four output pixels $G(x)=(0.7,\ 0.1,\ 0.4,\ -0.2)$ against target $y=(0.8,\ -0.2,\ 0.4,\ -0.6)$: L1 $=(0.1+0.3+0+0.4)/4=0.2$, so $\lambda\,\mathcal L_{L1}=20$.
4. Total $\mathcal L_G=1.1116+20=21.1116$. L1 dominates the *value*, but the adversarial gradient is what pushes texture into regions where L1 is satisfied by a blurred average.
5. For $D$ with real-pair logits $(1.5,\ 0.5,\ 2.0,\ -0.5)$: $\tfrac12\big(\mathrm{BCE}_{\text{real}}+\mathrm{BCE}_{\text{fake}}\big)=\tfrac12(0.4441+0.6116)=0.5279$.

## Results

Cityscapes labels→photo, FCN-scores by loss, with the U-Net and 70×70 PatchGAN (Isola et al. 2017)

| loss | per-pixel acc. | per-class acc. | class IoU |
| --- | --- | --- | --- |
| L1 | 0.42 | 0.15 | 0.11 |
| GAN (unconditional) | 0.22 | 0.05 | 0.01 |
| cGAN | 0.57 | 0.22 | 0.16 |
| L1 + GAN | 0.64 | 0.20 | 0.15 |
| L1 + cGAN | 0.66 | 0.23 | 0.17 |
| ground truth | 0.80 | 0.26 | 0.21 |

Same task, L1 + cGAN, varying the discriminator's receptive field

| discriminator | per-pixel acc. | per-class acc. | class IoU |
| --- | --- | --- | --- |
| 1×1 PixelGAN | 0.39 | 0.15 | 0.10 |
| 16×16 PatchGAN | 0.65 | 0.21 | 0.17 |
| 70×70 PatchGAN | 0.66 | 0.23 | 0.17 |
| 286×286 ImageGAN | 0.42 | 0.16 | 0.11 |

Conditioning matters: the unconditional GAN scores far below everything else because its outputs ignore the labels. The full $286\times286$ discriminator is *worse* than $70\times70$, even though it sees more. It has many more parameters and is harder to train. In the Turk study on aerial photos, L1 + cGAN outputs were taken for real 18.9% of the time in map→photo, against 2.8% for L1 alone. In photo→map they fooled only 6.1% of raters, because maps have crisp geometry where small errors are easy to see.

```python
import torch, torch.nn as nn, torch.nn.functional as F

def down(cin, cout, norm=True):          # Ck encoder block: 4×4 conv stride 2, BatchNorm, LeakyReLU(0.2)
    layers = [nn.Conv2d(cin, cout, 4, 2, 1, bias=not norm)]
    if norm:
        layers.append(nn.BatchNorm2d(cout))
    return nn.Sequential(*layers, nn.LeakyReLU(0.2))

def up(cin, cout, dropout=False):        # Ck / CDk decoder block: 4×4 transposed conv stride 2, BatchNorm, (Dropout 0.5), ReLU
    layers = [nn.ConvTranspose2d(cin, cout, 4, 2, 1, bias=False), nn.BatchNorm2d(cout)]
    if dropout:
        layers.append(nn.Dropout(0.5))
    return nn.Sequential(*layers, nn.ReLU())

class UNet(nn.Module):
    """Encoder C64-C128-C256-C512-C512-C512-C512-C512, decoder CD512-CD512-CD512-C512-C256-C128-C64 with skips."""
    def __init__(self, cin=3, cout=3):
        super().__init__()
        enc = [64, 128, 256, 512, 512, 512, 512, 512]
        self.downs = nn.ModuleList([down(cin, 64, norm=False)] +
                                   [down(enc[i - 1], enc[i], norm=(i < 7)) for i in range(1, 8)])  # no BN on the 1×1 bottleneck
        dec = [512, 512, 512, 512, 256, 128, 64]
        self.ups = nn.ModuleList([up(512, 512, dropout=True)] +
                                 [up(2 * dec[i - 1], dec[i], dropout=(i < 3)) for i in range(1, 7)])  # input = [up; skip]
        self.out = nn.ConvTranspose2d(2 * 64, cout, 4, 2, 1)

    def forward(self, x):
        skips = []
        for d in self.downs:
            x = d(x)
            skips.append(x)                   # 128, 64, 32, 16, 8, 4, 2, 1
        for u, s in zip(self.ups, reversed(skips[:-1])):
            x = torch.cat([u(x), s], 1)       # mirror layer i ↔ n − i
        return torch.tanh(self.out(x))

class PatchGAN(nn.Module):
    """70×70 PatchGAN: C64-C128-C256-C512, then a 1-channel conv; input = [condition; image]."""
    def __init__(self, cin=6, n_layers=3):
        super().__init__()
        layers, c = [nn.Conv2d(cin, 64, 4, 2, 1), nn.LeakyReLU(0.2)], 64
        for i in range(1, n_layers + 1):
            co, s = 64 * min(2 ** i, 8), (2 if i < n_layers else 1)
            layers += [nn.Conv2d(c, co, 4, s, 1, bias=False), nn.BatchNorm2d(co), nn.LeakyReLU(0.2)]
            c = co
        layers.append(nn.Conv2d(c, 1, 4, 1, 1))        # one logit per patch
        self.net = nn.Sequential(*layers)

    def forward(self, x, y):
        return self.net(torch.cat([x, y], 1))

G, D = UNet(), PatchGAN()
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce, lam = nn.BCEWithLogitsLoss(), 100.0

x = torch.randn(1, 3, 256, 256)    # condition (e.g. label map), scaled to [-1, 1] in practice
y = torch.randn(1, 3, 256, 256)    # paired target photo
fake = G(x)                         # dropout is the only noise source
print(fake.shape, D(x, fake).shape)                    # torch.Size([1, 3, 256, 256]) torch.Size([1, 1, 30, 30])
print(sum(p.numel() for p in D.parameters()))          # 2768705

# discriminator step: real pairs → 1, fake pairs → 0, objective halved to slow D down
d_real, d_fake = D(x, y), D(x, fake.detach())
loss_d = 0.5 * (bce(d_real, torch.ones_like(d_real)) + bce(d_fake, torch.zeros_like(d_fake)))
opt_d.zero_grad(); loss_d.backward(); opt_d.step()

# generator step: non-saturating cGAN term + λ·L1
d_fake = D(x, fake)
loss_g = bce(d_fake, torch.ones_like(d_fake)) + lam * F.l1_loss(fake, y)
opt_g.zero_grad(); loss_g.backward(); opt_g.step()
print(f"{sum(p.numel() for p in G.parameters()) / 1e6:.1f}M generator parameters")   # 54.4M
```

U-Net generator, 70×70 PatchGAN, and one training step of L_cGAN + 100·L1 (run with PyTorch 2 on CPU)

> **Feed D the pair, not just the output:** The discriminator must see $[x;\,y]$ (6 channels). If it sees only $y$, it judges realism but not correspondence. $G$ can then output any realistic photo, and in the paper's ablation the unconditional GAN collapsed to nearly one output (0.22 per-pixel accuracy against 0.57 for cGAN). The same holds for the fake pair: use $[x;\,G(x)]$ with the *same* $x$.

> **Why L1, and why so much of it:** For one pixel whose true value is $-1$ or $+1$ with probabilities $0.3$ and $0.7$, L2 is minimised at the mean $0.4$, a value that never occurs. L1 is minimised at the median, $+1$. For a 50/50 split L1 is flat on $[-1,1]$, and the adversarial term picks a sharp answer. $\lambda=100$ makes L1 dominate the loss value (20 against 1.1 in the worked example), so outputs stay close to the target, while the PatchGAN gradient only refines texture.

> **Paired data is the hard requirement:** Pix2Pix needs aligned $(x,y)$ pairs. They are cheap when one side can be computed from the other (edges via a detector, greyscale via conversion, maps via rendering), and expensive or impossible otherwise (photo↔painting, horse↔zebra). **CycleGAN** keeps the PatchGAN and drops the pairing by adding a second generator and a cycle-consistency loss.

**Strengths**

- One architecture and loss for many paired translation tasks, with no task-specific engineering
- L1 + cGAN gives outputs that are both faithful to the target and sharp
- PatchGAN: few parameters, fully convolutional, runs on images larger than the training crops
- Trains on small datasets (400 facades) in hours

**Limitations**

- Needs pixel-aligned training pairs
- Almost deterministic: dropout gives little diversity, so one input yields essentially one output
- A 70×70 discriminator cannot enforce global consistency beyond the patch size
- Struggles on outputs with strict geometry (photo→map) and at high resolution, which pix2pixHD later addressed

**Question:** Why does Pix2Pix's discriminator take the condition x as an extra input?

- A. To make D larger and more stable
- B. So D can penalise outputs that are realistic but do not match the input, which also prevents collapse to a single output
- C. Because the U-Net needs x at the bottleneck
- D. It is only needed at test time

**Answer:** B. So D can penalise outputs that are realistic but do not match the input, which also prevents collapse to a single output — $D(x,y)$ judges whether $y$ is a plausible output *for* $x$. Without $x$ the GAN term rewards any realistic image, and in the paper's ablation the unconditional variant collapsed to nearly the same output for every input.

**Question:** A 70×70 PatchGAN processes a 256×256 pair. What does it output, and how is that used in the loss?

- A. One logit for the whole image
- B. A 70×70 map, one logit per pixel
- C. A 30×30 map of logits, each judging one 70×70 patch; the loss averages over them
- D. A 256×256 map, then a max over pixels

**Answer:** C. A 30×30 map of logits, each judging one 70×70 patch; the loss averages over them — Three stride-2 and two stride-1 $4\times4$ convolutions take $256\to128\to64\to32\to31\to30$, and each unit's receptive field is $70\times70$ input pixels. The BCE (or other GAN loss) is averaged over the 900 patch decisions.

**Key points**

- Pix2Pix objective: $G^\ast=\arg\min_G\max_D\ \mathcal L_{cGAN}(G,D)+\lambda\,\mathbb E\|y-G(x)\|_1$ with $\lambda=100$; $D$ sees the pair $(x,y)$.
- Architecture: U-Net generator (skips from layer $i$ to $n-i$, $256^2\to1^2\to256^2$) and a $70\times70$ PatchGAN that outputs a $30\times30$ map of patch logits.
- L1 handles low frequencies (layout, colour); the PatchGAN acts as a learned texture loss for high frequencies.
- Noise enters only as dropout at train and test time, so outputs are nearly deterministic.
- Cityscapes labels→photo FCN-score: L1 + cGAN 0.66 per-pixel accuracy vs 0.42 for L1 alone; 70×70 patches beat both 1×1 and 286×286.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
