Image-to-image translation

Image-to-image translation maps an image from one domain to another while keeping its structure: Pix2Pix with aligned pairs, CycleGAN with unpaired sets, diffusion editors (SDEdit, ControlNet) with pretrained models. FID measures realism.

8 min read825 wordscore0 viewsupdated 2026-10-15

Definition

Image-to-image translation learns a mapping G:X→YG: X\to Y that turns an image xx from a source domain (label maps, edges, summer photos, horses) into an image y^\hat y of a target domain (photos, shoes, winter photos, zebras). The layout of xx is kept and its appearance changes. The task was framed as one general problem by Pix2Pix (Isola, Zhu, Zhou and Efros, CVPR 2017) for paired data and CycleGAN (Zhu, Park, Isola and Efros, ICCV 2017) for unpaired data. Diffusion-based editors such as SDEdit (Meng et al., ICLR 2022) and ControlNet (Zhang, Rao and Agrawala, ICCV 2023) reuse a pretrained generator instead.

Many vision tasks have this shape: semantic labels to a street photo, a sketch to a product image, greyscale to colour, day to night, map to aerial view. Each used to need its own pipeline and hand-made loss. Translation methods replace the hand-made loss with a learned one: a discriminator that judges whether the output looks like a member of YY.

Three things make it hard. Per-pixel losses such as L2 average all plausible outputs into a blur. Paired data is often impossible to collect, because no one has a zebra photo of the same horse in the same pose. And many outputs can be correct for one input, since an edge map does not fix the colour of a shoe.

Paired: conditional GAN plus L1

With aligned pairs (xi,yi)(x_i,y_i), Pix2Pix trains a conditional GAN. The discriminator sees the input with either the real or the generated output. The L1 term keeps low frequencies right, and the GAN term adds sharp detail:

G∗=arg⁡min⁡Gmax⁡D Ex,y[log⁡D(x,y)]+Ex[log⁡(1−D(x,G(x)))]+λ Ex,y∥y−G(x)∥1G^*=\arg\min_G\max_D\ \mathbb E_{x,y}\big[\log D(x,y)\big]+\mathbb E_{x}\big[\log\big(1-D(x,G(x))\big)\big]+\lambda\,\mathbb E_{x,y}\big\|y-G(x)\big\|_1
D(x,⋅)D(x,\cdot)
a 70×70 PatchGAN: it classifies each overlapping patch as real or fake, judging texture, while L1 handles global layout
GG
a U-Net whose skip connections carry the input's edges straight to the output
λ=100\lambda=100
weight of the L1 term in Pix2Pix
image-to-image translation: learn G: X → Y between image domainspaired · Pix2Pix 2017unpaired · CycleGAN 2017pretrained diffusionx labelsU-Netŷ photoL = L_cGAN(G, D) + λ‖y − G(x)‖₁, λ = 100D sees (x, y): 70×70 PatchGANneeds aligned pairs (xᵢ, yᵢ)labels→photo, edges→shoe,map↔aerial, day→nightPix2PixHD: 2048×1024X horseY zebraGFL_GAN(G,D_Y) + L_GAN(F,D_X) + λ(‖F(G(x))−x‖₁ + ‖G(F(y))−y‖₁), λ = 10only two collections, no pairshorse↔zebra, summer↔winter,photo↔paintingone-sided: CUT (PatchNCE)guide+εdenoiseoutputSDEdit: noise the guide x, x_t0 = √ᾱ x + √(1−ᾱ) ε, then denoise from t0 to 0ControlNet: edges, depth, pose → trainable encoder copy → zero convs → frozen U-Netno GAN; reuses text-to-imageone input, many outputsBicycleGAN: G(x, z), z recoverableMUNIT: content code + style codethe L1 term and a deterministic Gaverage the modes: one sketch, onecolour; a latent z or a style code(or diffusion noise) gives varietyevaluation: three separate questionsrealistic? FID, KID vs real Yfaithful? task metric on output (FCN-score), LPIPS to xdiverse? LPIPS between samplesuser studies (AMT) for realism;FID is biased at small sample sizes
fig.Three settings. Paired: a label map goes through a U-Net to a photo, and the PatchGAN sees the input together with the output; L1 to the true photo has weight 100. Unpaired: horse and zebra collections are linked by G and F, with a cycle loss of weight 10. Diffusion: a pretrained model edits a noised guide (SDEdit) or follows an edge, depth or pose map through ControlNet. Below: why one input needs many outputs, and the three separate evaluation questions.

Unpaired: cycle consistency

Without pairs, an adversarial loss alone only makes G(x)G(x) look like some member of YY. It does not have to correspond to xx. CycleGAN adds a second generator F:Y→XF: Y\to X and asks that translating there and back returns the input:

L=LGAN(G,DY)+LGAN(F,DX)+λ(Ex∥F(G(x))−x∥1+Ey∥G(F(y))−y∥1)\mathcal L=\mathcal L_{\text{GAN}}(G,D_Y)+\mathcal L_{\text{GAN}}(F,D_X)+\lambda\Big(\mathbb E_x\big\|F(G(x))-x\big\|_1+\mathbb E_y\big\|G(F(y))-y\big\|_1\Big)
DX, DYD_X,\ D_Y
one PatchGAN per domain, trained with the least-squares (LSGAN) objective
λ=10\lambda=10
the cycle weight in CycleGAN
F(G(x))≈xF(G(x))\approx x
forces G to keep enough of x to be undone, which rules out ignoring the input

Cycle consistency assumes the mapping is close to a bijection. It struggles when the domains differ in geometry, as in cat to dog, and generators can hide information in faint high-frequency patterns to pass the cycle check. CUT (Park, Efros, Zhang and Zhu, ECCV 2020) drops the second generator. It uses a patchwise contrastive loss, InfoNCE loss between corresponding input and output patches, so each output patch must be recognisable as coming from its input patch.

Reference configurations

Pix2Pix
256×256 images, U-Net generator, 70×70 PatchGAN, λ = 100, Adam with learning rate 0.0002 and β₁ = 0.5
Pix2PixHD (Wang et al., CVPR 2018)
2048×1024 label-to-photo; coarse-to-fine generator, three multi-scale discriminators, feature-matching loss
CycleGAN
ResNet generator with 9 residual blocks at 256×256, LSGAN losses, λ = 10, a buffer of 50 past generated images for the discriminators
Multimodal
BicycleGAN (Zhu et al., NeurIPS 2017) ties a latent code z to the output so different z give different results; MUNIT (Huang, Liu, Belongie and Kautz, ECCV 2018) splits an image into a content code and a style code
ControlNet
a trainable copy of the Stable Diffusion encoder takes the condition (Canny edges, depth, pose, segmentation) and joins the frozen U-Net through zero-initialised 1×1 convolutions

Diffusion-based translation

A pretrained diffusion model already knows the target domain (see Diffusion models and Stable Diffusion). SDEdit uses it without any training. It adds noise to the guide up to level t0t_0, xt0=αˉt0 x+1−αˉt0 εx_{t_0}=\sqrt{\bar\alpha_{t_0}}\,x+\sqrt{1-\bar\alpha_{t_0}}\,\varepsilon, then runs the reverse process from t0t_0 to 0. Noise hides the guide's unrealistic details while its coarse layout survives. t0t_0 trades faithfulness to the guide against realism. In libraries this is the img2img strength parameter.

SDEdit: noise the guide to t0 = 0.50, then denoise (40 samples, 40 DDIM steps)real imagesguide× guide · • noised x_t0 · • output · ○ 2σ of a modemean distance vs t0012300.51to guide (unfaithful ↑)to data (unrealistic ↑)distance to guide 1.644distance to data 0.226within 2σ of data 80%real samples score ≈ 0.20middle t0: on the data and near the guide (useful range)
guide
x_t0 = √ᾱ·guide + √(1−ᾱ)·ε with ᾱ = cos²(π·t0/2); exact score of the noised data mixture
fig.SDEdit in a 2-D toy image space. Real images are 15 Gaussian modes (σ = 0.16) on an arc, and the denoiser uses their exact noised score. The default guide sits 1.388 from the nearest mode. At t0 = 0.5 the 40 outputs are on average 0.226 from the data (80% within 2σ) and 1.644 from the guide. At t0 = 0.2 they stay closer (0.926) but are not yet real (0.584, 3% within 2σ). At t0 = 1 they are realistic (0.190, 93%), but spread along the arc at 2.213 from the guide. Real samples score about 0.20.

ControlNet trains a copy of the encoder on (condition, image) pairs and leaves the original model frozen. Its zero convolutions output exactly zero at the start, so training begins from the unchanged pretrained model. InstructPix2Pix (Brooks, Holynski and Efros, CVPR 2023) instead learns edits from a text instruction, trained on image pairs that were themselves generated.

Evaluation

A translation should be realistic, faithful to the input and, where the task allows, diverse. No one number covers all three. Realism is usually measured with the Fréchet Inception Distance (Heusel et al., 2017) between Inception-v3 pool features (2048-d) of generated and real target images:

FID=∥μr−μg∥22+Tr⁡ ⁣(Σr+Σg−2(ΣrΣg)1/2)\mathrm{FID}=\big\|\mu_r-\mu_g\big\|_2^2+\operatorname{Tr}\!\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r\Sigma_g\big)^{1/2}\Big)
μ, Σ\mu,\ \Sigma
mean and covariance of the features of real (r) and generated (g) images
1-D case\text{1-D case}
reduces to (μr−μg)2+(σr−σg)2(\mu_r-\mu_g)^2+(\sigma_r-\sigma_g)^2: the first term catches a wrong average look, the second a loss of variety

Faithfulness needs a task-specific check. Pix2Pix ran an FCN segmenter on generated Cityscapes photos and compared the predicted labels with the input labels, the FCN-score. Diversity is often the mean LPIPS distance between outputs for the same input. Human studies remain common for realism.

Generated features(μ_r − μ_g)²(σ_r − σ_g)²FID
N(0, 1), matches the real data000
N(0.5, 1), shifted look0.2500.25
N(0, 0.3²), mode collapse00.490.49
N(0.5, 0.3²), both0.250.490.74
Worked example: 1-D FID against real features N(0, 1) (computed in Python)

The collapsed generator has the right average and still scores worse than the shifted one. FID penalises a lack of variety, which a per-image metric cannot see.

python
1import torch
2from diffusers import AutoPipelineForImage2Image
3from torchmetrics.image.fid import FrechetInceptionDistance
4
5pipe = AutoPipelineForImage2Image.from_pretrained(
6 "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
7# strength = t0: noise the input to 60% of the schedule, so 0.6 * 50 = 30 denoising steps run
8out = pipe(prompt="a zebra standing in a field", image=horse_img,
9 strength=0.6, num_inference_steps=50, guidance_scale=7.5).images[0]
10
11fid = FrechetInceptionDistance(feature=2048) # Inception-v3 pool features
12fid.update(real_zebras_uint8, real=True) # uint8 tensors, shape (N, 3, H, W)
13fid.update(fake_zebras_uint8, real=False)
14print(float(fid.compute()))
SDEdit-style translation with diffusers img2img, then FID with torchmetrics (not run here)

Check yourself

Why does CycleGAN need the cycle-consistency loss when it already has adversarial losses?

Check yourself

In SDEdit, what happens as the noise level t0 approaches 1?

Remember

  • ✓Translation learns G:X→YG: X\to Y that keeps the structure of xx and changes its appearance.
  • ✓Paired data: Pix2Pix, a conditional GAN with a U-Net, a 70×70 PatchGAN and L1 weighted 100.
  • ✓Unpaired data: CycleGAN with two generators and λ∥F(G(x))−x∥1\lambda\|F(G(x))-x\|_1 cycle loss (λ=10\lambda=10); CUT uses a patchwise contrastive loss instead.
  • ✓Diffusion: SDEdit noises the guide to t0 and denoises (t0 trades faithfulness for realism); ControlNet adds spatial conditions to a frozen model via zero convolutions.
  • ✓Evaluate realism (FID), faithfulness (task metric such as FCN-score) and diversity (LPIPS between samples) separately.
← back to Deep Learning roadmap