Definition
Pix2Pix is the general-purpose image-to-image translation method of Isola, Zhu, Zhou and Efros, "Image-to-Image Translation with Conditional Adversarial Networks" (CVPR 2017). Given paired training images , such as a label map and its photo, it trains a generator to output from . The objective is a conditional GAN loss, in which the discriminator sees the pair , plus with . is a U-Net, an encoder–decoder with skip connections. is a PatchGAN, a small fully convolutional network that outputs one real/fake score per patch.
Many vision problems turn one image into another: labels to photo, edges to photo, day to night, black-and-white to colour, aerial photo to map. Before 2017 each had its own architecture and hand-designed loss. A plain regression loss is the obvious common choice, and it fails in a known way. When several outputs are plausible, minimising Euclidean distance predicts their average, which is a blurry image. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei Efros proposed to learn the loss instead. The discriminator of a Conditional GAN (cGAN) (see also Generative adversarial network (GAN)) decides whether an output is a realistic answer for its input, so blur and other unrealistic outputs get penalised. One recipe, with no task-specific changes, handled all of the tasks above. Its public code (pix2pix, later pytorch-CycleGAN-and-pix2pix) set the template for CycleGAN and much later conditional image synthesis.
Objective
- the input (condition) image, e.g. a semantic label map
- the paired target image; the training data are pairs from the same scene
- noise; in the final model it enters only as dropout inside
- probability that is a real output for input ; and are concatenated along channels
- mean absolute pixel error; L1 rather than L2 because it blurs less
- weight of the reconstruction term; 100 in all experiments of the paper
- in practice the loss of is halved to slow its learning relative to
- maximises instead of minimising : the non-saturating loss of the original GAN paper (see Adversarial loss)
The two terms split the work. L1 pins down the low frequencies: overall layout, colours, where the edges are. Its minimiser for each pixel is the median of the plausible values, so it never invents structure. The adversarial term enforces high frequencies, the texture that makes a patch look real. The ablation shows each term's role. L1 alone is blurry. cGAN alone is sharp but has more artifacts. An unconditional GAN (where sees only ) collapsed to producing nearly the same image for every input, because nothing forced the output to match .
Noise only as dropout
A conditional GAN normally takes noise so it can model a distribution . Isola et al. found that a Gaussian fed to the generator was simply ignored. The final model provides noise only as dropout, applied to several decoder layers at training and test time. It produces only minor stochasticity. Pix2Pix is close to deterministic, a known limitation: one label map maps to essentially one photo. Producing diverse outputs for the same input was left to later work.
U-Net generator
In translation tasks the input and output share structure: the edges of a label map are the edges of the photo. A plain encoder–decoder forces all of that through a bottleneck. The U-Net adds a skip connection from encoder layer to decoder layer , concatenating their channels, so low-level detail goes around the bottleneck. With = Conv–BatchNorm–ReLU with filters and the same with 50% dropout, all with stride 2, the encoder is C64-C128-C256-C512-C512-C512-C512-C512. It takes down to , using LeakyReLU 0.2 and no BatchNorm on the first layer. The decoder mirrors it with plain ReLUs and dropout in its first three blocks, then a final convolution and .
PatchGAN discriminator
Because L1 already handles the low frequencies, only has to model local structure. The PatchGAN classifies each patch as real or fake. It is a fully convolutional net, and every output unit is the logit for its receptive field. The loss averages over all units. This treats the image as a Markov random field in which pixels more than a patch diameter apart are independent, so the loss acts like a learned texture/style loss. The default discriminator is C64-C128-C256-C512 ( convolutions, the first three with stride 2), then a convolution to one channel, with LeakyReLU 0.2 and no BatchNorm on the first layer. For a input it outputs a map of logits, has fewer parameters than a full-image discriminator, and runs on images of any size.
Setup in the paper
- Optimiser
- Adam, learning rate , , ; alternate one step and one step; the objective of divided by 2
- Batch
- 1 to 10 depending on the dataset; at batch size 1 BatchNorm becomes instance normalisation. BatchNorm uses test-batch statistics at test time, and dropout stays on
- Images
- , random jitter (resize to , crop ) and mirroring; weights initialised from
- Loss weight
- on L1
- Tasks
- labels→photo (Cityscapes, CMP Facades), map↔aerial photo, black-and-white→colour, edges→photo (shoes, handbags), sketch→photo, day→night, thermal→colour, inpainting
- Speed
- the 400-image facades set trained in under two hours on one Pascal Titan X
- Evaluation
- Amazon Mechanical Turk real-vs-fake studies, and the FCN-score: a semantic segmentation network trained on real Cityscapes photos is run on the generated photos
The blurred box stands in for a region where an L1-only generator averaged several plausible answers. Number of layers n = 0…5 in the reference code; n = 3 is the paper's 70×70 PatchGAN.
Worked example: a 70×70 receptive field and one generator loss
The receptive field grows as and the jump as , starting from . Every layer uses and padding 1. The input has 6 channels: the 3-channel condition concatenated with the 3-channel output.
| layer | k / stride | output side | jump J | receptive field | parameters |
|---|---|---|---|---|---|
| C64 (no BN) | 4 / 2 | 128 | 2 | 4 | 6,208 |
| C128 | 4 / 2 | 64 | 4 | 10 | 131,328 |
| C256 | 4 / 2 | 32 | 8 | 22 | 524,800 |
| C512 | 4 / 1 | 31 | 8 | 46 | 2,098,176 |
| conv → 1 logit | 4 / 1 | 30 | 8 | 70 | 8,193 |
| total | 30 × 30 | 70 × 70 | 2,768,705 |
One generator loss on a 4-patch output map
- 1 returns the patch logits on the fake pair . These are probabilities of being real.
- 2Non-saturating adversarial loss per patch: , mean . Patches that look fake (here the fourth) give the largest gradient.
- 3Four output pixels against target : L1 , so .
- 4Total . L1 dominates the value, but the adversarial gradient is what pushes texture into regions where L1 is satisfied by a blurred average.
- 5For with real-pair logits : .
Results
| loss | per-pixel acc. | per-class acc. | class IoU |
|---|---|---|---|
| L1 | 0.42 | 0.15 | 0.11 |
| GAN (unconditional) | 0.22 | 0.05 | 0.01 |
| cGAN | 0.57 | 0.22 | 0.16 |
| L1 + GAN | 0.64 | 0.20 | 0.15 |
| L1 + cGAN | 0.66 | 0.23 | 0.17 |
| ground truth | 0.80 | 0.26 | 0.21 |
| discriminator | per-pixel acc. | per-class acc. | class IoU |
|---|---|---|---|
| 1×1 PixelGAN | 0.39 | 0.15 | 0.10 |
| 16×16 PatchGAN | 0.65 | 0.21 | 0.17 |
| 70×70 PatchGAN | 0.66 | 0.23 | 0.17 |
| 286×286 ImageGAN | 0.42 | 0.16 | 0.11 |
Conditioning matters: the unconditional GAN scores far below everything else because its outputs ignore the labels. The full discriminator is worse than , even though it sees more. It has many more parameters and is harder to train. In the Turk study on aerial photos, L1 + cGAN outputs were taken for real 18.9% of the time in map→photo, against 2.8% for L1 alone. In photo→map they fooled only 6.1% of raters, because maps have crisp geometry where small errors are easy to see.
1import torch, torch.nn as nn, torch.nn.functional as F23def down(cin, cout, norm=True): # Ck encoder block: 4×4 conv stride 2, BatchNorm, LeakyReLU(0.2)4 layers = [nn.Conv2d(cin, cout, 4, 2, 1, bias=not norm)]5 if norm:6 layers.append(nn.BatchNorm2d(cout))7 return nn.Sequential(*layers, nn.LeakyReLU(0.2))89def up(cin, cout, dropout=False): # Ck / CDk decoder block: 4×4 transposed conv stride 2, BatchNorm, (Dropout 0.5), ReLU10 layers = [nn.ConvTranspose2d(cin, cout, 4, 2, 1, bias=False), nn.BatchNorm2d(cout)]11 if dropout:12 layers.append(nn.Dropout(0.5))13 return nn.Sequential(*layers, nn.ReLU())1415class UNet(nn.Module):16 """Encoder C64-C128-C256-C512-C512-C512-C512-C512, decoder CD512-CD512-CD512-C512-C256-C128-C64 with skips."""17 def __init__(self, cin=3, cout=3):18 super().__init__()19 enc = [64, 128, 256, 512, 512, 512, 512, 512]20 self.downs = nn.ModuleList([down(cin, 64, norm=False)] +21 [down(enc[i - 1], enc[i], norm=(i < 7)) for i in range(1, 8)]) # no BN on the 1×1 bottleneck22 dec = [512, 512, 512, 512, 256, 128, 64]23 self.ups = nn.ModuleList([up(512, 512, dropout=True)] +24 [up(2 * dec[i - 1], dec[i], dropout=(i < 3)) for i in range(1, 7)]) # input = [up; skip]25 self.out = nn.ConvTranspose2d(2 * 64, cout, 4, 2, 1)2627 def forward(self, x):28 skips = []29 for d in self.downs:30 x = d(x)31 skips.append(x) # 128, 64, 32, 16, 8, 4, 2, 132 for u, s in zip(self.ups, reversed(skips[:-1])):33 x = torch.cat([u(x), s], 1) # mirror layer i ↔ n − i34 return torch.tanh(self.out(x))3536class PatchGAN(nn.Module):37 """70×70 PatchGAN: C64-C128-C256-C512, then a 1-channel conv; input = [condition; image]."""38 def __init__(self, cin=6, n_layers=3):39 super().__init__()40 layers, c = [nn.Conv2d(cin, 64, 4, 2, 1), nn.LeakyReLU(0.2)], 6441 for i in range(1, n_layers + 1):42 co, s = 64 * min(2 ** i, 8), (2 if i < n_layers else 1)43 layers += [nn.Conv2d(c, co, 4, s, 1, bias=False), nn.BatchNorm2d(co), nn.LeakyReLU(0.2)]44 c = co45 layers.append(nn.Conv2d(c, 1, 4, 1, 1)) # one logit per patch46 self.net = nn.Sequential(*layers)4748 def forward(self, x, y):49 return self.net(torch.cat([x, y], 1))5051G, D = UNet(), PatchGAN()52opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))53opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))54bce, lam = nn.BCEWithLogitsLoss(), 100.05556x = torch.randn(1, 3, 256, 256) # condition (e.g. label map), scaled to [-1, 1] in practice57y = torch.randn(1, 3, 256, 256) # paired target photo58fake = G(x) # dropout is the only noise source59print(fake.shape, D(x, fake).shape) # torch.Size([1, 3, 256, 256]) torch.Size([1, 1, 30, 30])60print(sum(p.numel() for p in D.parameters())) # 27687056162# discriminator step: real pairs → 1, fake pairs → 0, objective halved to slow D down63d_real, d_fake = D(x, y), D(x, fake.detach())64loss_d = 0.5 * (bce(d_real, torch.ones_like(d_real)) + bce(d_fake, torch.zeros_like(d_fake)))65opt_d.zero_grad(); loss_d.backward(); opt_d.step()6667# generator step: non-saturating cGAN term + λ·L168d_fake = D(x, fake)69loss_g = bce(d_fake, torch.ones_like(d_fake)) + lam * F.l1_loss(fake, y)70opt_g.zero_grad(); loss_g.backward(); opt_g.step()71print(f"{sum(p.numel() for p in G.parameters()) / 1e6:.1f}M generator parameters") # 54.4MUse when
- +One architecture and loss for many paired translation tasks, with no task-specific engineering
- +L1 + cGAN gives outputs that are both faithful to the target and sharp
- +PatchGAN: few parameters, fully convolutional, runs on images larger than the training crops
- +Trains on small datasets (400 facades) in hours
Avoid when
- −Needs pixel-aligned training pairs
- −Almost deterministic: dropout gives little diversity, so one input yields essentially one output
- −A 70×70 discriminator cannot enforce global consistency beyond the patch size
- −Struggles on outputs with strict geometry (photo→map) and at high resolution, which pix2pixHD later addressed
Check yourself
Why does Pix2Pix's discriminator take the condition x as an extra input?
Check yourself
A 70×70 PatchGAN processes a 256×256 pair. What does it output, and how is that used in the loss?
Remember
- ✓Pix2Pix objective: with ; sees the pair .
- ✓Architecture: U-Net generator (skips from layer to , ) and a PatchGAN that outputs a map of patch logits.
- ✓L1 handles low frequencies (layout, colour); the PatchGAN acts as a learned texture loss for high frequencies.
- ✓Noise enters only as dropout at train and test time, so outputs are nearly deterministic.
- ✓Cityscapes labels→photo FCN-score: L1 + cGAN 0.66 per-pixel accuracy vs 0.42 for L1 alone; 70×70 patches beat both 1×1 and 286×286.