Pix2Pix

Pix2Pix (Isola et al., CVPR 2017) learns paired image-to-image translation with a conditional GAN plus an L1 term weighted λ = 100. A U-Net generator with skip connections maps x to y, and a 70×70 PatchGAN discriminator judges every patch of the pair (x, y) as real or fake.

13 min read1152 wordscore0 viewsupdated 2026-10-12

Definition

Pix2Pix is the general-purpose image-to-image translation method of Isola, Zhu, Zhou and Efros, "Image-to-Image Translation with Conditional Adversarial Networks" (CVPR 2017). Given paired training images (x,y)(x,y), such as a label map and its photo, it trains a generator GG to output yy from xx. The objective is a conditional GAN loss, in which the discriminator sees the pair (x, ⋅)(x,\,\cdot), plus λ ∥y−G(x)∥1\lambda\,\|y-G(x)\|_1 with λ=100\lambda=100. GG is a U-Net, an encoder–decoder with skip connections. DD is a PatchGAN, a small fully convolutional network that outputs one real/fake score per 70×7070\times70 patch.

Many vision problems turn one image into another: labels to photo, edges to photo, day to night, black-and-white to colour, aerial photo to map. Before 2017 each had its own architecture and hand-designed loss. A plain regression loss is the obvious common choice, and it fails in a known way. When several outputs are plausible, minimising Euclidean distance predicts their average, which is a blurry image. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei Efros proposed to learn the loss instead. The discriminator of a Conditional GAN (cGAN) (see also Generative adversarial network (GAN)) decides whether an output is a realistic answer for its input, so blur and other unrealistic outputs get penalised. One recipe, with no task-specific changes, handled all of the tasks above. Its public code (pix2pix, later pytorch-CycleGAN-and-pix2pix) set the template for CycleGAN and much later conditional image synthesis.

Objective

LcGAN(G,D)=Ex,y[log⁡D(x,y)]+Ex,z[log⁡(1−D(x,G(x,z)))],LL1(G)=Ex,y,z[∥y−G(x,z)∥1]\mathcal L_{cGAN}(G,D)=\mathbb E_{x,y}\big[\log D(x,y)\big]+\mathbb E_{x,z}\big[\log\big(1-D(x,G(x,z))\big)\big],\qquad \mathcal L_{L1}(G)=\mathbb E_{x,y,z}\big[\|y-G(x,z)\|_1\big]
xx
the input (condition) image, e.g. a semantic label map
yy
the paired target image; the training data are pairs (x,y)(x,y) from the same scene
zz
noise; in the final model it enters only as dropout inside GG
D(x,y)D(x,y)
probability that yy is a real output for input xx; xx and yy are concatenated along channels
∥⋅∥1\|\cdot\|_1
mean absolute pixel error; L1 rather than L2 because it blurs less
G∗=arg⁡min⁡Gmax⁡D LcGAN(G,D)+λ LL1(G),λ=100G^\ast=\arg\min_G\max_D\ \mathcal L_{cGAN}(G,D)+\lambda\,\mathcal L_{L1}(G),\qquad \lambda=100
λ\lambda
weight of the reconstruction term; 100 in all experiments of the paper
max⁡D\max_D
in practice the loss of DD is halved to slow its learning relative to GG
min⁡G\min_G
GG maximises log⁡D(x,G(x,z))\log D(x,G(x,z)) instead of minimising log⁡(1−D)\log(1-D): the non-saturating loss of the original GAN paper (see Adversarial loss)

The two terms split the work. L1 pins down the low frequencies: overall layout, colours, where the edges are. Its minimiser for each pixel is the median of the plausible values, so it never invents structure. The adversarial term enforces high frequencies, the texture that makes a patch look real. The ablation shows each term's role. L1 alone is blurry. cGAN alone is sharp but has more artifacts. An unconditional GAN (where DD sees only yy) collapsed to producing nearly the same image for every input, because nothing forced the output to match xx.

Noise only as dropout

A conditional GAN normally takes noise zz so it can model a distribution p(y∣x)p(y\mid x). Isola et al. found that a Gaussian zz fed to the generator was simply ignored. The final model provides noise only as dropout, applied to several decoder layers at training and test time. It produces only minor stochasticity. Pix2Pix is close to deterministic, a known limitation: one label map maps to essentially one photo. Producing diverse outputs for the same input was left to later work.

U-Net generator

In translation tasks the input and output share structure: the edges of a label map are the edges of the photo. A plain encoder–decoder forces all of that through a 1×1×5121\times1\times512 bottleneck. The U-Net adds a skip connection from encoder layer ii to decoder layer n−in-i, concatenating their channels, so low-level detail goes around the bottleneck. With CkCk = Conv–BatchNorm–ReLU with kk filters and CDkCDk the same with 50% dropout, all 4×44\times4 with stride 2, the encoder is C64-C128-C256-C512-C512-C512-C512-C512. It takes 2562256^2 down to 121^2, using LeakyReLU 0.2 and no BatchNorm on the first layer. The decoder mirrors it with plain ReLUs and dropout in its first three blocks, then a final convolution and tanh⁡\tanh.

PatchGAN discriminator

Because L1 already handles the low frequencies, DD only has to model local structure. The PatchGAN classifies each N×NN\times N patch as real or fake. It is a fully convolutional net, and every output unit is the logit for its receptive field. The loss averages over all units. This treats the image as a Markov random field in which pixels more than a patch diameter apart are independent, so the loss acts like a learned texture/style loss. The default 70×7070\times70 discriminator is C64-C128-C256-C512 (4×44\times4 convolutions, the first three with stride 2), then a 4×44\times4 convolution to one channel, with LeakyReLU 0.2 and no BatchNorm on the first layer. For a 2562256^2 input it outputs a 30×3030\times30 map of logits, has fewer parameters than a full-image discriminator, and runs on images of any size.

Setup in the paper

Optimiser
Adam, learning rate 2⋅10−42\cdot10^{-4}, β1=0.5\beta_1=0.5, β2=0.999\beta_2=0.999; alternate one DD step and one GG step; the objective of DD divided by 2
Batch
1 to 10 depending on the dataset; at batch size 1 BatchNorm becomes instance normalisation. BatchNorm uses test-batch statistics at test time, and dropout stays on
Images
256×256256\times256, random jitter (resize to 2862286^2, crop 2562256^2) and mirroring; weights initialised from N(0,0.02)\mathcal N(0,0.02)
Loss weight
λ=100\lambda=100 on L1
Tasks
labels→photo (Cityscapes, CMP Facades), map↔aerial photo, black-and-white→colour, edges→photo (shoes, handbags), sketch→photo, day→night, thermal→colour, inpainting
Speed
the 400-image facades set trained in under two hours on one Pascal Titan X
Evaluation
Amazon Mechanical Turk real-vs-fake studies, and the FCN-score: a semantic segmentation network trained on real Cityscapes photos is run on the generated photos
D scores [x, G(x)] (6 channels) patch by patch; the loss averages the logitsG(x) 256×256 · dashed box blurredD output 30×30 · shade = blurred sharereceptive field70×70 pxoutput map30×30 = 900stride (jump)8 pxparameters2,768,705patches seeing blur324 (36.0%)this patch(11, 21)its blurred share1.0004×4 convs, channels/stride: C64/2 → C128/2 → C256/2 → C512/1 → 1/1spatial size: 256 → 128 → 64 → 32 → 31 → 30 (padding 1)RF from the top: r ← r + (k − 1)·J, J ← J·s ⇒ r = 70, unit 0 starts at pixel -23selected patch covers x ∈ [145, 214], y ∈ [65, 134] (clipped to the image)mean blurred share over all patches = 0.1128 (blurred box is 0.0977 of the image)70×70 (paper default): local enough to judge texture, small enough to run on any image sizeL1 handles low frequencies (the layout); the PatchGAN term penalises blur inside each windowclick a cell of the output map, or move the patch centre below
patch size

The blurred box stands in for a region where an L1-only generator averaged several plausible answers. Number of layers n = 0…5 in the reference code; n = 3 is the paper's 70×70 PatchGAN.

fig.The left panel is a toy 256×256 generated image G(x). Its dashed box (80×80 px, 9.77% of the image) is blurred, as an L1-only generator leaves a region where several answers were plausible. The right panel is the discriminator's output map, shaded by how much of each unit's receptive field lies in the blurred box. The geometry is computed layer by layer from the reference code. The default 70×70 PatchGAN (n = 3 strided layers) gives a 30×30 = 900 map with stride 8 px and 2,768,705 parameters. 324 of the 900 patches (36.0%) see some blur, and the mean blurred share is 0.1128. The selected unit (11, 21) covers x ∈ [145, 214], y ∈ [65, 134] and lies wholly inside the blur (share 1.000). The 1×1 PixelGAN has 256×256 outputs and only 9,024 parameters, but no patch sees any spatial structure, so it can judge colour but not sharpness. 16×16 gives 126×126 outputs (139,585 parameters). The 286×286 ImageGAN has 6×6 outputs and 11,159,361 parameters, and all 36 of its units see the blur, each mixing it with the whole image. Click a cell or move the centre sliders to trace a unit back to its patch.

Worked example: a 70×70 receptive field and one generator loss

The receptive field grows as r←r+(k−1) Jr\leftarrow r+(k-1)\,J and the jump as J←J⋅sJ\leftarrow J\cdot s, starting from r=J=1r=J=1. Every layer uses k=4k=4 and padding 1. The input has 6 channels: the 3-channel condition xx concatenated with the 3-channel output.

layerk / strideoutput sidejump Jreceptive fieldparameters
C64 (no BN)4 / 2128246,208
C1284 / 264410131,328
C2564 / 232822524,800
C5124 / 1318462,098,176
conv → 1 logit4 / 1308708,193
total30 × 3070 × 702,768,705
70×70 PatchGAN on a 256×256 pair (computed in Node; the parameter total matches PyTorch)

One generator loss on a 4-patch output map

  1. 1DD returns the patch logits (−1.0, 0.0, 1.0, −2.0)(-1.0,\ 0.0,\ 1.0,\ -2.0) on the fake pair (x,G(x))(x,G(x)). These are probabilities D=(0.2689, 0.5, 0.7311, 0.1192)D=(0.2689,\ 0.5,\ 0.7311,\ 0.1192) of being real.
  2. 2Non-saturating adversarial loss per patch: −log⁡D=softplus(−ℓ)=(1.3133, 0.6931, 0.3133, 2.1269)-\log D=\mathrm{softplus}(-\ell)=(1.3133,\ 0.6931,\ 0.3133,\ 2.1269), mean 1.11161.1116. Patches that look fake (here the fourth) give the largest gradient.
  3. 3Four output pixels G(x)=(0.7, 0.1, 0.4, −0.2)G(x)=(0.7,\ 0.1,\ 0.4,\ -0.2) against target y=(0.8, −0.2, 0.4, −0.6)y=(0.8,\ -0.2,\ 0.4,\ -0.6): L1 =(0.1+0.3+0+0.4)/4=0.2=(0.1+0.3+0+0.4)/4=0.2, so λ LL1=20\lambda\,\mathcal L_{L1}=20.
  4. 4Total LG=1.1116+20=21.1116\mathcal L_G=1.1116+20=21.1116. L1 dominates the value, but the adversarial gradient is what pushes texture into regions where L1 is satisfied by a blurred average.
  5. 5For DD with real-pair logits (1.5, 0.5, 2.0, −0.5)(1.5,\ 0.5,\ 2.0,\ -0.5): 12(BCEreal+BCEfake)=12(0.4441+0.6116)=0.5279\tfrac12\big(\mathrm{BCE}_{\text{real}}+\mathrm{BCE}_{\text{fake}}\big)=\tfrac12(0.4441+0.6116)=0.5279.

Results

lossper-pixel acc.per-class acc.class IoU
L10.420.150.11
GAN (unconditional)0.220.050.01
cGAN0.570.220.16
L1 + GAN0.640.200.15
L1 + cGAN0.660.230.17
ground truth0.800.260.21
Cityscapes labels→photo, FCN-scores by loss, with the U-Net and 70×70 PatchGAN (Isola et al. 2017)
discriminatorper-pixel acc.per-class acc.class IoU
1×1 PixelGAN0.390.150.10
16×16 PatchGAN0.650.210.17
70×70 PatchGAN0.660.230.17
286×286 ImageGAN0.420.160.11
Same task, L1 + cGAN, varying the discriminator's receptive field

Conditioning matters: the unconditional GAN scores far below everything else because its outputs ignore the labels. The full 286×286286\times286 discriminator is worse than 70×7070\times70, even though it sees more. It has many more parameters and is harder to train. In the Turk study on aerial photos, L1 + cGAN outputs were taken for real 18.9% of the time in map→photo, against 2.8% for L1 alone. In photo→map they fooled only 6.1% of raters, because maps have crisp geometry where small errors are easy to see.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def down(cin, cout, norm=True): # Ck encoder block: 4×4 conv stride 2, BatchNorm, LeakyReLU(0.2)
4 layers = [nn.Conv2d(cin, cout, 4, 2, 1, bias=not norm)]
5 if norm:
6 layers.append(nn.BatchNorm2d(cout))
7 return nn.Sequential(*layers, nn.LeakyReLU(0.2))
8
9def up(cin, cout, dropout=False): # Ck / CDk decoder block: 4×4 transposed conv stride 2, BatchNorm, (Dropout 0.5), ReLU
10 layers = [nn.ConvTranspose2d(cin, cout, 4, 2, 1, bias=False), nn.BatchNorm2d(cout)]
11 if dropout:
12 layers.append(nn.Dropout(0.5))
13 return nn.Sequential(*layers, nn.ReLU())
14
15class UNet(nn.Module):
16 """Encoder C64-C128-C256-C512-C512-C512-C512-C512, decoder CD512-CD512-CD512-C512-C256-C128-C64 with skips."""
17 def __init__(self, cin=3, cout=3):
18 super().__init__()
19 enc = [64, 128, 256, 512, 512, 512, 512, 512]
20 self.downs = nn.ModuleList([down(cin, 64, norm=False)] +
21 [down(enc[i - 1], enc[i], norm=(i < 7)) for i in range(1, 8)]) # no BN on the 1×1 bottleneck
22 dec = [512, 512, 512, 512, 256, 128, 64]
23 self.ups = nn.ModuleList([up(512, 512, dropout=True)] +
24 [up(2 * dec[i - 1], dec[i], dropout=(i < 3)) for i in range(1, 7)]) # input = [up; skip]
25 self.out = nn.ConvTranspose2d(2 * 64, cout, 4, 2, 1)
26
27 def forward(self, x):
28 skips = []
29 for d in self.downs:
30 x = d(x)
31 skips.append(x) # 128, 64, 32, 16, 8, 4, 2, 1
32 for u, s in zip(self.ups, reversed(skips[:-1])):
33 x = torch.cat([u(x), s], 1) # mirror layer i ↔ n − i
34 return torch.tanh(self.out(x))
35
36class PatchGAN(nn.Module):
37 """70×70 PatchGAN: C64-C128-C256-C512, then a 1-channel conv; input = [condition; image]."""
38 def __init__(self, cin=6, n_layers=3):
39 super().__init__()
40 layers, c = [nn.Conv2d(cin, 64, 4, 2, 1), nn.LeakyReLU(0.2)], 64
41 for i in range(1, n_layers + 1):
42 co, s = 64 * min(2 ** i, 8), (2 if i < n_layers else 1)
43 layers += [nn.Conv2d(c, co, 4, s, 1, bias=False), nn.BatchNorm2d(co), nn.LeakyReLU(0.2)]
44 c = co
45 layers.append(nn.Conv2d(c, 1, 4, 1, 1)) # one logit per patch
46 self.net = nn.Sequential(*layers)
47
48 def forward(self, x, y):
49 return self.net(torch.cat([x, y], 1))
50
51G, D = UNet(), PatchGAN()
52opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
53opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
54bce, lam = nn.BCEWithLogitsLoss(), 100.0
55
56x = torch.randn(1, 3, 256, 256) # condition (e.g. label map), scaled to [-1, 1] in practice
57y = torch.randn(1, 3, 256, 256) # paired target photo
58fake = G(x) # dropout is the only noise source
59print(fake.shape, D(x, fake).shape) # torch.Size([1, 3, 256, 256]) torch.Size([1, 1, 30, 30])
60print(sum(p.numel() for p in D.parameters())) # 2768705
61
62# discriminator step: real pairs → 1, fake pairs → 0, objective halved to slow D down
63d_real, d_fake = D(x, y), D(x, fake.detach())
64loss_d = 0.5 * (bce(d_real, torch.ones_like(d_real)) + bce(d_fake, torch.zeros_like(d_fake)))
65opt_d.zero_grad(); loss_d.backward(); opt_d.step()
66
67# generator step: non-saturating cGAN term + λ·L1
68d_fake = D(x, fake)
69loss_g = bce(d_fake, torch.ones_like(d_fake)) + lam * F.l1_loss(fake, y)
70opt_g.zero_grad(); loss_g.backward(); opt_g.step()
71print(f"{sum(p.numel() for p in G.parameters()) / 1e6:.1f}M generator parameters") # 54.4M
U-Net generator, 70×70 PatchGAN, and one training step of L_cGAN + 100·L1 (run with PyTorch 2 on CPU)

Use when

  • +One architecture and loss for many paired translation tasks, with no task-specific engineering
  • +L1 + cGAN gives outputs that are both faithful to the target and sharp
  • +PatchGAN: few parameters, fully convolutional, runs on images larger than the training crops
  • +Trains on small datasets (400 facades) in hours

Avoid when

  • −Needs pixel-aligned training pairs
  • −Almost deterministic: dropout gives little diversity, so one input yields essentially one output
  • −A 70×70 discriminator cannot enforce global consistency beyond the patch size
  • −Struggles on outputs with strict geometry (photo→map) and at high resolution, which pix2pixHD later addressed

Check yourself

Why does Pix2Pix's discriminator take the condition x as an extra input?

Check yourself

A 70×70 PatchGAN processes a 256×256 pair. What does it output, and how is that used in the loss?

Remember

  • ✓Pix2Pix objective: G∗=arg⁡min⁡Gmax⁡D LcGAN(G,D)+λ E∥y−G(x)∥1G^\ast=\arg\min_G\max_D\ \mathcal L_{cGAN}(G,D)+\lambda\,\mathbb E\|y-G(x)\|_1 with λ=100\lambda=100; DD sees the pair (x,y)(x,y).
  • ✓Architecture: U-Net generator (skips from layer ii to n−in-i, 2562→12→2562256^2\to1^2\to256^2) and a 70×7070\times70 PatchGAN that outputs a 30×3030\times30 map of patch logits.
  • ✓L1 handles low frequencies (layout, colour); the PatchGAN acts as a learned texture loss for high frequencies.
  • ✓Noise enters only as dropout at train and test time, so outputs are nearly deterministic.
  • ✓Cityscapes labels→photo FCN-score: L1 + cGAN 0.66 per-pixel accuracy vs 0.42 for L1 alone; 70×70 patches beat both 1×1 and 286×286.
← back to Deep Learning roadmap