Definition
CycleGAN is the unpaired image-to-image translation method of Zhu, Park, Isola and Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks" (ICCV 2017). It needs only a set of images from domain (horses) and a set from domain (zebras), with no correspondence between them. It trains two generators, and , and two discriminators, and . Each generator fools its discriminator, and the cycle-consistency loss requires that translating there and back returns the input: and .
Pix2Pix showed that a conditional GAN with an L1 loss solves many translation problems, provided you have aligned pairs . For most interesting pairs of domains no such data exists. Nobody has the zebra version of a horse photo, the photo Monet painted from, or summer and winter shots taken from exactly the same viewpoint. Jun-Yan Zhu, Taesung Park, Phillip Isola and Alexei Efros asked what can be learned from two unrelated collections. An adversarial loss alone can make look like as a distribution, but it says nothing about which a given should become. Their fix borrows an old idea from language translation: a good translation should translate back. DualGAN and DiscoGAN proposed the same two-way constraint independently in 2017.
Objective
- e.g. horse → zebra
- e.g. zebra → horse; trained jointly with G
- forward cycle consistency
- backward cycle consistency
- pixel-wise L1; the authors tried an adversarial loss here and saw no improvement
- adversarial loss making indistinguishable from real
- relative weight of cycle consistency; 10 in all experiments
- PatchGANs, as in Pix2Pix, but unconditional: there is no paired input to show them
The model can be read as two autoencoders, and , whose latent code is an image of the other domain. The adversarial losses force the code to look like a real image of that domain. Cycle consistency forces it to keep enough information to rebuild the input.
Why the adversarial loss is not enough
With enough capacity, can map onto any arrangement of that has the right distribution. Every permutation of target images is an equally good optimum, so a single photo of a horse could become any zebra. In practice the outcome is often mode collapse: many inputs map to the same output and the optimisation stalls. Cycle consistency rules out mappings that lose information. If two horses go to the same zebra, cannot send that zebra back to both. It also ties each output to its input, since has to recover from . The figure below makes this exact with rings, where all rotations of a uniform ring have the same distribution.
LSGAN losses, history buffer, identity loss
- least-squares GAN (Mao et al. 2017) in place of the negative log-likelihood; more stable training and better results in their experiments
- raw PatchGAN output, no sigmoid; regressed toward 1 for real and 0 for fake
- the objective of is halved to slow it relative to
Two more stabilisers come from earlier work. Following Shrivastava et al. (2017), the discriminators are updated on a history of generated images rather than only the latest one: a buffer of the 50 most recently generated images per domain. In the reference code, once the buffer is full each new fake is either passed through or, with probability 0.5, swapped for a random stored one. This keeps from chasing the generator's latest mode. For painting↔photo the authors add an identity loss (after Taigman et al. 2017) that asks each generator to leave images of its target domain unchanged:
- a zebra (or a photo) fed to the horse→zebra generator should come out unchanged
- weight used in the paper's painting→photo experiments
Without it, and may change the colour of the input even when they don't need to. For example, Monet's daytime paintings were sometimes turned into sunset photos. Nothing in the cycle or adversarial losses forbids a reversible tint change. The identity term breaks the tie in favour of changing as little as possible.
Setup in the paper
- Generator
- Johnson et al. (2016): c7s1-64, d128, d256, residual blocks R256, u128, u64, c7s1-3, with instance normalisation and reflection padding. 6 blocks at , 9 blocks at and above
- Discriminator
- PatchGAN, C64-C128-C256-C512, instance norm, LeakyReLU 0.2
- Losses
- LSGAN; on ; identity loss for painting↔photo
- Optimiser
- Adam, batch size 1, learning rate for 100 epochs, then linear decay to 0 over the next 100; weights from
- Stabilisers
- history buffer of 50 generated images per discriminator; loss halved
- Applications
- horse↔zebra, apple↔orange, summer↔winter Yosemite, photo↔Monet/Van Gogh/Cézanne/Ukiyo-e, photo enhancement (shallow depth of field), labels↔photo and map↔aerial photo as quantitative benchmarks
Worked example: one generator and discriminator loss
Take 4-pixel images and 2-patch discriminator outputs. Inputs from and an unrelated from .
| quantity | value |
|---|---|
| D_Y(G(x)) | (0.6, 0.3) |
| D_X(F(y)) | (0.8, 0.4) |
| F(G(x)) | (0.45, −0.2, 0.8, 0.1) |
| G(F(y)) | (0.1, 0.6, −0.5, 0.5) |
| G(y) (identity) | (0.05, 0.6, −0.45, 0.35) |
| F(x) (identity) | (0.5, −0.25, 0.85, 0.0) |
| D_Y(y) (real) | (0.9, 0.7) |
Assembling the losses
- 1LSGAN generator terms: for , and for .
- 2Forward cycle: , mean . Backward: , mean . Weighted: .
- 3Without identity, . Most of the signal comes from the cycle term even though both reconstructions are close.
- 4Identity: and , sum , times : . Total .
- 5Discriminator on a real and a (buffered) fake: .
Results and failure cases
On Cityscapes labels→photo with FCN-scores, CycleGAN reached per-pixel accuracy 0.52, per-class accuracy 0.17 and class IoU 0.11 without any pairs. Pix2Pix, trained on the true pairs, scored 0.71, 0.25 and 0.18 in the same evaluation. Paired supervision still wins clearly. In Turk studies on maps↔aerial photos, CycleGAN outputs were taken for real in about a quarter of trials (26.8% for map→photo, 23.2% for photo→map). The baselines (CoGAN, BiGAN/ALI, SimGAN, feature loss + GAN) almost never fooled raters. In the ablation, removing the GAN losses or the cycle loss substantially degraded results. GAN loss with only one cycle direction often trained unstably and collapsed, particularly for the direction that was not constrained.
The authors document clear limits. Colour and texture changes work well. Geometric changes mostly fail: on dog→cat the learned translation degenerates into minimal changes to the input. The authors attribute this to generator architectures tailored to appearance changes. The training distribution matters too. ImageNet's wild-horse and zebra classes contain no riders, so a man riding a horse got zebra stripes on his skin. Translations can also be ambiguous, for example tree↔building labels on Cityscapes. The cycle loss enforces invertibility, not semantic correctness: Chu, Zhmoginov and Sandler (2017) showed that CycleGAN can hide the information needed to rebuild as a near-invisible, high-frequency signal in .
1import random, torch, torch.nn as nn, torch.nn.functional as F23class ResBlock(nn.Module):4 def __init__(self, c):5 super().__init__()6 self.body = nn.Sequential(nn.ReflectionPad2d(1), nn.Conv2d(c, c, 3), nn.InstanceNorm2d(c), nn.ReLU(True),7 nn.ReflectionPad2d(1), nn.Conv2d(c, c, 3), nn.InstanceNorm2d(c))8 def forward(self, x):9 return x + self.body(x)1011def resnet_generator(n_blocks=9): # c7s1-64, d128, d256, 9×R256, u128, u64, c7s1-312 L = [nn.ReflectionPad2d(3), nn.Conv2d(3, 64, 7), nn.InstanceNorm2d(64), nn.ReLU(True)]13 for c in (64, 128):14 L += [nn.Conv2d(c, 2 * c, 3, 2, 1), nn.InstanceNorm2d(2 * c), nn.ReLU(True)]15 L += [ResBlock(256) for _ in range(n_blocks)]16 for c in (256, 128):17 L += [nn.ConvTranspose2d(c, c // 2, 3, 2, 1, output_padding=1), nn.InstanceNorm2d(c // 2), nn.ReLU(True)]18 L += [nn.ReflectionPad2d(3), nn.Conv2d(64, 3, 7), nn.Tanh()]19 return nn.Sequential(*L)2021def patchgan(): # 70×70 PatchGAN C64-C128-C256-C512, one score per patch22 L, c = [nn.Conv2d(3, 64, 4, 2, 1), nn.LeakyReLU(0.2, True)], 6423 for co, s in ((128, 2), (256, 2), (512, 1)):24 L += [nn.Conv2d(c, co, 4, s, 1), nn.InstanceNorm2d(co), nn.LeakyReLU(0.2, True)]25 c = co26 return nn.Sequential(*L, nn.Conv2d(512, 1, 4, 1, 1)) # no sigmoid: LSGAN regresses the raw output2728class ImagePool: # history of 50 generated images for the discriminator29 def __init__(self, size=50):30 self.size, self.images = size, []31 def query(self, img):32 if len(self.images) < self.size:33 self.images.append(img.detach()); return img.detach()34 if random.random() < 0.5: # half the time: show an old fake, store the new one35 i = random.randrange(self.size)36 old, self.images[i] = self.images[i], img.detach()37 return old38 return img.detach()3940G, F_, D_X, D_Y = resnet_generator(), resnet_generator(), patchgan(), patchgan() # G: X→Y, F: Y→X41opt_g = torch.optim.Adam(list(G.parameters()) + list(F_.parameters()), lr=2e-4, betas=(0.5, 0.999))42opt_d = torch.optim.Adam(list(D_X.parameters()) + list(D_Y.parameters()), lr=2e-4, betas=(0.5, 0.999))43pool_x, pool_y, lam, lam_idt = ImagePool(), ImagePool(), 10.0, 0.544mse = nn.MSELoss()45gan = lambda d, real: mse(d, torch.full_like(d, 1.0 if real else 0.0))4647x = torch.rand(1, 3, 256, 256) * 2 - 1 # a horse (batch size 1, as in the paper)48y = torch.rand(1, 3, 256, 256) * 2 - 1 # an unrelated zebra: no pairing anywhere4950# generators: fool both discriminators, close both cycles, and (optionally) leave targets unchanged51fake_y, fake_x = G(x), F_(y)52loss_gan = gan(D_Y(fake_y), True) + gan(D_X(fake_x), True)53loss_cyc = F.l1_loss(F_(fake_y), x) + F.l1_loss(G(fake_x), y) # ‖F(G(x))−x‖₁ + ‖G(F(y))−y‖₁54loss_idt = F.l1_loss(G(y), y) + F.l1_loss(F_(x), x) # ‖G(y)−y‖₁ + ‖F(x)−x‖₁55loss_g = loss_gan + lam * loss_cyc + lam_idt * lam * loss_idt56opt_g.zero_grad(); loss_g.backward(); opt_g.step()5758# discriminators: LSGAN, real → 1, (pooled) fake → 0, halved59loss_d = 0.5 * (gan(D_Y(y), True) + gan(D_Y(pool_y.query(fake_y)), False)) \60 + 0.5 * (gan(D_X(x), True) + gan(D_X(pool_x.query(fake_x)), False))61opt_d.zero_grad(); loss_d.backward(); opt_d.step()6263print(fake_y.shape, D_Y(fake_y).shape) # torch.Size([1, 3, 256, 256]) torch.Size([1, 1, 30, 30])64print(f"{sum(p.numel() for p in G.parameters()) / 1e6:.2f}M per generator, "65 f"{sum(p.numel() for p in D_Y.parameters()) / 1e6:.2f}M per discriminator") # 11.38M, 2.76MUse when
- +Needs only two unpaired image collections
- +Strong for colour, texture and style changes (season transfer, painting↔photo, horse↔zebra)
- +Simple, reusable recipe: two GANs plus an L1 cycle term
- +Cycle consistency rules out many-to-one mappings that a pure adversarial loss allows
Avoid when
- −Twice the networks of Pix2Pix and six generator passes per step with identity loss
- −Fails on geometric or shape changes (dog↔cat), where it degenerates into minimal edits
- −Clearly below paired training when pairs exist (FCN per-pixel accuracy 0.52 vs 0.71)
- −Deterministic one-to-one mapping; can hide information steganographically instead of translating
Check yourself
Why can't CycleGAN rely on the two adversarial losses alone?
Check yourself
On painting→photo, G turns a daytime Monet into a convincing sunset photo, and F turns it back perfectly. Which term discourages this?
Remember
- ✓CycleGAN objective: with , on unpaired sets and .
- ✓Training details: LSGAN losses , a history buffer of 50 generated images, an identity loss weighted , ResNet generators and PatchGANs.
- ✓Adversarial losses match distributions only; cycle consistency makes the mapping invertible, which rules out many-to-one collapse.
- ✓Works for colour and texture, fails for geometric changes, and stays below paired Pix2Pix when pairs exist.
- ✓Zero cycle loss does not imply a correct translation: reversible tints and hidden steganographic codes also satisfy it.