Definition
Image-to-image translation learns a mapping that turns an image from a source domain (label maps, edges, summer photos, horses) into an image of a target domain (photos, shoes, winter photos, zebras). The layout of is kept and its appearance changes. The task was framed as one general problem by Pix2Pix (Isola, Zhu, Zhou and Efros, CVPR 2017) for paired data and CycleGAN (Zhu, Park, Isola and Efros, ICCV 2017) for unpaired data. Diffusion-based editors such as SDEdit (Meng et al., ICLR 2022) and ControlNet (Zhang, Rao and Agrawala, ICCV 2023) reuse a pretrained generator instead.
Many vision tasks have this shape: semantic labels to a street photo, a sketch to a product image, greyscale to colour, day to night, map to aerial view. Each used to need its own pipeline and hand-made loss. Translation methods replace the hand-made loss with a learned one: a discriminator that judges whether the output looks like a member of .
Three things make it hard. Per-pixel losses such as L2 average all plausible outputs into a blur. Paired data is often impossible to collect, because no one has a zebra photo of the same horse in the same pose. And many outputs can be correct for one input, since an edge map does not fix the colour of a shoe.
Paired: conditional GAN plus L1
With aligned pairs , Pix2Pix trains a conditional GAN. The discriminator sees the input with either the real or the generated output. The L1 term keeps low frequencies right, and the GAN term adds sharp detail:
- a 70×70 PatchGAN: it classifies each overlapping patch as real or fake, judging texture, while L1 handles global layout
- a U-Net whose skip connections carry the input's edges straight to the output
- weight of the L1 term in Pix2Pix
Unpaired: cycle consistency
Without pairs, an adversarial loss alone only makes look like some member of . It does not have to correspond to . CycleGAN adds a second generator and asks that translating there and back returns the input:
- one PatchGAN per domain, trained with the least-squares (LSGAN) objective
- the cycle weight in CycleGAN
- forces G to keep enough of x to be undone, which rules out ignoring the input
Cycle consistency assumes the mapping is close to a bijection. It struggles when the domains differ in geometry, as in cat to dog, and generators can hide information in faint high-frequency patterns to pass the cycle check. CUT (Park, Efros, Zhang and Zhu, ECCV 2020) drops the second generator. It uses a patchwise contrastive loss, InfoNCE loss between corresponding input and output patches, so each output patch must be recognisable as coming from its input patch.
Reference configurations
- Pix2Pix
- 256×256 images, U-Net generator, 70×70 PatchGAN, λ = 100, Adam with learning rate 0.0002 and β₁ = 0.5
- Pix2PixHD (Wang et al., CVPR 2018)
- 2048×1024 label-to-photo; coarse-to-fine generator, three multi-scale discriminators, feature-matching loss
- CycleGAN
- ResNet generator with 9 residual blocks at 256×256, LSGAN losses, λ = 10, a buffer of 50 past generated images for the discriminators
- Multimodal
- BicycleGAN (Zhu et al., NeurIPS 2017) ties a latent code z to the output so different z give different results; MUNIT (Huang, Liu, Belongie and Kautz, ECCV 2018) splits an image into a content code and a style code
- ControlNet
- a trainable copy of the Stable Diffusion encoder takes the condition (Canny edges, depth, pose, segmentation) and joins the frozen U-Net through zero-initialised 1×1 convolutions
Diffusion-based translation
A pretrained diffusion model already knows the target domain (see Diffusion models and Stable Diffusion). SDEdit uses it without any training. It adds noise to the guide up to level , , then runs the reverse process from to 0. Noise hides the guide's unrealistic details while its coarse layout survives. trades faithfulness to the guide against realism. In libraries this is the img2img strength parameter.
ControlNet trains a copy of the encoder on (condition, image) pairs and leaves the original model frozen. Its zero convolutions output exactly zero at the start, so training begins from the unchanged pretrained model. InstructPix2Pix (Brooks, Holynski and Efros, CVPR 2023) instead learns edits from a text instruction, trained on image pairs that were themselves generated.
Evaluation
A translation should be realistic, faithful to the input and, where the task allows, diverse. No one number covers all three. Realism is usually measured with the Fréchet Inception Distance (Heusel et al., 2017) between Inception-v3 pool features (2048-d) of generated and real target images:
- mean and covariance of the features of real (r) and generated (g) images
- reduces to : the first term catches a wrong average look, the second a loss of variety
Faithfulness needs a task-specific check. Pix2Pix ran an FCN segmenter on generated Cityscapes photos and compared the predicted labels with the input labels, the FCN-score. Diversity is often the mean LPIPS distance between outputs for the same input. Human studies remain common for realism.
| Generated features | (μ_r − μ_g)² | (σ_r − σ_g)² | FID |
|---|---|---|---|
| N(0, 1), matches the real data | 0 | 0 | 0 |
| N(0.5, 1), shifted look | 0.25 | 0 | 0.25 |
| N(0, 0.3²), mode collapse | 0 | 0.49 | 0.49 |
| N(0.5, 0.3²), both | 0.25 | 0.49 | 0.74 |
The collapsed generator has the right average and still scores worse than the shifted one. FID penalises a lack of variety, which a per-image metric cannot see.
1import torch2from diffusers import AutoPipelineForImage2Image3from torchmetrics.image.fid import FrechetInceptionDistance45pipe = AutoPipelineForImage2Image.from_pretrained(6 "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")7# strength = t0: noise the input to 60% of the schedule, so 0.6 * 50 = 30 denoising steps run8out = pipe(prompt="a zebra standing in a field", image=horse_img,9 strength=0.6, num_inference_steps=50, guidance_scale=7.5).images[0]1011fid = FrechetInceptionDistance(feature=2048) # Inception-v3 pool features12fid.update(real_zebras_uint8, real=True) # uint8 tensors, shape (N, 3, H, W)13fid.update(fake_zebras_uint8, real=False)14print(float(fid.compute()))Check yourself
Why does CycleGAN need the cycle-consistency loss when it already has adversarial losses?
Check yourself
In SDEdit, what happens as the noise level t0 approaches 1?
Remember
- ✓Translation learns that keeps the structure of and changes its appearance.
- ✓Paired data: Pix2Pix, a conditional GAN with a U-Net, a 70×70 PatchGAN and L1 weighted 100.
- ✓Unpaired data: CycleGAN with two generators and cycle loss (); CUT uses a patchwise contrastive loss instead.
- ✓Diffusion: SDEdit noises the guide to t0 and denoises (t0 trades faithfulness for realism); ControlNet adds spatial conditions to a frozen model via zero convolutions.
- ✓Evaluate realism (FID), faithfulness (task metric such as FCN-score) and diversity (LPIPS between samples) separately.