# Image-to-image translation

> Image-to-image translation maps an image from one domain to another while keeping its structure: Pix2Pix with aligned pairs, CycleGAN with unpaired sets, diffusion editors (SDEdit, ControlNet) with pretrained models. FID measures realism.

Source: https://www.mlfoundry.in/learn/image-to-image-translation  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Image-to-image translation:** learns a mapping $G: X\to Y$ that turns an image $x$ from a source domain (label maps, edges, summer photos, horses) into an image $\hat y$ of a target domain (photos, shoes, winter photos, zebras). The layout of $x$ is kept and its appearance changes. The task was framed as one general problem by **Pix2Pix** (Isola, Zhu, Zhou and Efros, CVPR 2017) for paired data and **CycleGAN** (Zhu, Park, Isola and Efros, ICCV 2017) for unpaired data. Diffusion-based editors such as SDEdit (Meng et al., ICLR 2022) and ControlNet (Zhang, Rao and Agrawala, ICCV 2023) reuse a pretrained generator instead.

Many vision tasks have this shape: semantic labels to a street photo, a sketch to a product image, greyscale to colour, day to night, map to aerial view. Each used to need its own pipeline and hand-made loss. Translation methods replace the hand-made loss with a learned one: a discriminator that judges whether the output looks like a member of $Y$.

Three things make it hard. Per-pixel losses such as L2 average all plausible outputs into a blur. Paired data is often impossible to collect, because no one has a zebra photo of the same horse in the same pose. And many outputs can be correct for one input, since an edge map does not fix the colour of a shoe.

## Paired: conditional GAN plus L1

With aligned pairs $(x_i,y_i)$, Pix2Pix trains a conditional GAN. The discriminator sees the input with either the real or the generated output. The L1 term keeps low frequencies right, and the GAN term adds sharp detail:

$$
G^*=\arg\min_G\max_D\ \mathbb E_{x,y}\big[\log D(x,y)\big]+\mathbb E_{x}\big[\log\big(1-D(x,G(x))\big)\big]+\lambda\,\mathbb E_{x,y}\big\|y-G(x)\big\|_1
$$

- $D(x,\cdot)$: a 70×70 PatchGAN: it classifies each overlapping patch as real or fake, judging texture, while L1 handles global layout
- $G$: a U-Net whose skip connections carry the input's edges straight to the output
- $\lambda=100$: weight of the L1 term in Pix2Pix

> Figure: Three settings. Paired: a label map goes through a U-Net to a photo, and the PatchGAN sees the input together with the output; L1 to the true photo has weight 100. Unpaired: horse and zebra collections are linked by G and F, with a cycle loss of weight 10. Diffusion: a pretrained model edits a noised guide (SDEdit) or follows an edge, depth or pose map through ControlNet. Below: why one input needs many outputs, and the three separate evaluation questions.

## Unpaired: cycle consistency

Without pairs, an adversarial loss alone only makes $G(x)$ look like some member of $Y$. It does not have to correspond to $x$. CycleGAN adds a second generator $F: Y\to X$ and asks that translating there and back returns the input:

$$
\mathcal L=\mathcal L_{\text{GAN}}(G,D_Y)+\mathcal L_{\text{GAN}}(F,D_X)+\lambda\Big(\mathbb E_x\big\|F(G(x))-x\big\|_1+\mathbb E_y\big\|G(F(y))-y\big\|_1\Big)
$$

- $D_X,\ D_Y$: one PatchGAN per domain, trained with the least-squares (LSGAN) objective
- $\lambda=10$: the cycle weight in CycleGAN
- $F(G(x))\approx x$: forces G to keep enough of x to be undone, which rules out ignoring the input

Cycle consistency assumes the mapping is close to a bijection. It struggles when the domains differ in geometry, as in cat to dog, and generators can hide information in faint high-frequency patterns to pass the cycle check. CUT (Park, Efros, Zhang and Zhu, ECCV 2020) drops the second generator. It uses a patchwise contrastive loss, **InfoNCE loss** between corresponding input and output patches, so each output patch must be recognisable as coming from its input patch.

**Reference configurations**

- **Pix2Pix:** 256×256 images, U-Net generator, 70×70 PatchGAN, λ = 100, Adam with learning rate 0.0002 and β₁ = 0.5
- **Pix2PixHD (Wang et al., CVPR 2018):** 2048×1024 label-to-photo; coarse-to-fine generator, three multi-scale discriminators, feature-matching loss
- **CycleGAN:** ResNet generator with 9 residual blocks at 256×256, LSGAN losses, λ = 10, a buffer of 50 past generated images for the discriminators
- **Multimodal:** BicycleGAN (Zhu et al., NeurIPS 2017) ties a latent code z to the output so different z give different results; MUNIT (Huang, Liu, Belongie and Kautz, ECCV 2018) splits an image into a content code and a style code
- **ControlNet:** a trainable copy of the Stable Diffusion encoder takes the condition (Canny edges, depth, pose, segmentation) and joins the frozen U-Net through zero-initialised 1×1 convolutions

## Diffusion-based translation

A pretrained diffusion model already knows the target domain (see **Diffusion models** and **Stable Diffusion**). SDEdit uses it without any training. It adds noise to the guide up to level $t_0$, $x_{t_0}=\sqrt{\bar\alpha_{t_0}}\,x+\sqrt{1-\bar\alpha_{t_0}}\,\varepsilon$, then runs the reverse process from $t_0$ to 0. Noise hides the guide's unrealistic details while its coarse layout survives. $t_0$ trades **faithfulness** to the guide against **realism**. In libraries this is the img2img `strength` parameter.

> Figure: SDEdit in a 2-D toy image space. Real images are 15 Gaussian modes (σ = 0.16) on an arc, and the denoiser uses their exact noised score. The default guide sits 1.388 from the nearest mode. At t0 = 0.5 the 40 outputs are on average 0.226 from the data (80% within 2σ) and 1.644 from the guide. At t0 = 0.2 they stay closer (0.926) but are not yet real (0.584, 3% within 2σ). At t0 = 1 they are realistic (0.190, 93%), but spread along the arc at 2.213 from the guide. Real samples score about 0.20.

ControlNet trains a copy of the encoder on (condition, image) pairs and leaves the original model frozen. Its zero convolutions output exactly zero at the start, so training begins from the unchanged pretrained model. InstructPix2Pix (Brooks, Holynski and Efros, CVPR 2023) instead learns edits from a text instruction, trained on image pairs that were themselves generated.

## Evaluation

A translation should be realistic, faithful to the input and, where the task allows, diverse. No one number covers all three. Realism is usually measured with the Fréchet Inception Distance (Heusel et al., 2017) between Inception-v3 pool features (2048-d) of generated and real target images:

$$
\mathrm{FID}=\big\|\mu_r-\mu_g\big\|_2^2+\operatorname{Tr}\!\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r\Sigma_g\big)^{1/2}\Big)
$$

- $\mu,\ \Sigma$: mean and covariance of the features of real (r) and generated (g) images
- $\text{1-D case}$: reduces to $(\mu_r-\mu_g)^2+(\sigma_r-\sigma_g)^2$: the first term catches a wrong average look, the second a loss of variety

Faithfulness needs a task-specific check. Pix2Pix ran an FCN segmenter on generated Cityscapes photos and compared the predicted labels with the input labels, the **FCN-score**. Diversity is often the mean LPIPS distance between outputs for the same input. Human studies remain common for realism.

Worked example: 1-D FID against real features N(0, 1) (computed in Python)

| Generated features | (μ_r − μ_g)² | (σ_r − σ_g)² | FID |
| --- | --- | --- | --- |
| N(0, 1), matches the real data | 0 | 0 | 0 |
| N(0.5, 1), shifted look | 0.25 | 0 | 0.25 |
| N(0, 0.3²), mode collapse | 0 | 0.49 | 0.49 |
| N(0.5, 0.3²), both | 0.25 | 0.49 | 0.74 |

The collapsed generator has the right average and still scores worse than the shifted one. FID penalises a lack of variety, which a per-image metric cannot see.

```python
import torch
from diffusers import AutoPipelineForImage2Image
from torchmetrics.image.fid import FrechetInceptionDistance

pipe = AutoPipelineForImage2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
# strength = t0: noise the input to 60% of the schedule, so 0.6 * 50 = 30 denoising steps run
out = pipe(prompt="a zebra standing in a field", image=horse_img,
           strength=0.6, num_inference_steps=50, guidance_scale=7.5).images[0]

fid = FrechetInceptionDistance(feature=2048)   # Inception-v3 pool features
fid.update(real_zebras_uint8, real=True)        # uint8 tensors, shape (N, 3, H, W)
fid.update(fake_zebras_uint8, real=False)
print(float(fid.compute()))
```

SDEdit-style translation with diffusers img2img, then FID with torchmetrics (not run here)

> **FID numbers are rarely comparable:** FID is biased upwards at small sample sizes and depends on image resizing, the Inception weights and the number of images. Translation test sets are often only a few hundred images. Compare methods only with the same code, resolution and sample count, and report a faithfulness metric alongside it: an unconditional generator that ignores its input can still get a good FID.

> **Pick the setting from the data you have:** Aligned pairs (rendered labels, edge maps from photos, greyscale from colour) make the problem supervised, and Pix2Pix-style training is simple and strong. With two unaligned collections, use cycle or contrastive constraints. When a large text-to-image model covers the target domain, SDEdit or ControlNet often beats training a GAN from scratch, at the cost of tens of denoising steps per image.

**Question:** Why does CycleGAN need the cycle-consistency loss when it already has adversarial losses?

- A. Adversarial losses cannot be trained without paired data
- B. The adversarial loss only asks G(x) to look like some image of Y, not one that corresponds to x
- C. It makes the generators faster at test time
- D. It replaces the discriminators

**Answer:** B. The adversarial loss only asks G(x) to look like some image of Y, not one that corresponds to x — Any mapping that sends $X$ onto the distribution of $Y$, including one that ignores $x$, satisfies the GAN loss. Requiring $F(G(x))\approx x$ forces $G$ to preserve the content of $x$.

**Question:** In SDEdit, what happens as the noise level t0 approaches 1?

- A. The output copies the guide exactly
- B. Outputs become realistic but lose the guide's layout, approaching unconditional samples
- C. The output becomes pure noise
- D. Nothing; t0 only changes speed

**Answer:** B. Outputs become realistic but lose the guide's layout, approaching unconditional samples — At t0 = 1 the noised guide is almost pure Gaussian noise, so the reverse process produces an ordinary sample of the model. Small t0 keeps the guide but leaves its unrealistic details.

**Key points**

- Translation learns $G: X\to Y$ that keeps the structure of $x$ and changes its appearance.
- Paired data: Pix2Pix, a conditional GAN with a U-Net, a 70×70 PatchGAN and L1 weighted 100.
- Unpaired data: CycleGAN with two generators and $\lambda\|F(G(x))-x\|_1$ cycle loss ($\lambda=10$); CUT uses a patchwise contrastive loss instead.
- Diffusion: SDEdit noises the guide to t0 and denoises (t0 trades faithfulness for realism); ControlNet adds spatial conditions to a frozen model via zero convolutions.
- Evaluate realism (FID), faithfulness (task metric such as FCN-score) and diversity (LPIPS between samples) separately.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
