Definition
Deep convolutional GAN (DCGAN) DCGAN is the convolutional architecture and training recipe from Radford, Metz and Chintala, "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (ICLR 2016). The generator turns a noise vector into an image through a stack of fractionally-strided (transposed) convolutions. Each one doubles the resolution and halves the channel count. The discriminator mirrors it with strided convolutions. Both use batch normalization and avoid pooling and hidden fully connected layers. The loss is the standard GAN objective of Generative adversarial network (GAN).
The original GANs of 2014 were multilayer perceptrons. Attempts to scale them up with the CNNs used for classification mostly failed: training diverged, or the generator produced noise. LAPGAN (Denton et al., 2015) worked around this by generating a Laplacian pyramid one scale at a time. Alec Radford, Luke Metz and Soumith Chintala instead searched for a single architecture that trained stably across datasets and at 64×64 resolution. Their paper is mostly an empirical recipe. It also showed that the trained networks learn useful representations. The discriminator's features are good for classification, and the generator's latent space supports smooth interpolation and vector arithmetic.
The architecture guidelines
Section 3 of the paper, verbatim in substance
- 1Replace pooling with strided convolutions in and fractionally-strided convolutions in , so each network learns its own down- and up-sampling (the all-convolutional net of Springenberg et al., 2015).
- 2Use batch normalization in both networks, except the generator's output layer and the discriminator's input layer. Applying it to every layer caused sample oscillation and instability.
- 3Remove fully connected hidden layers. 's first layer is a matrix multiplication of whose result is reshaped into a feature map. 's last convolution is flattened into a single sigmoid output.
- 4Use ReLU in every generator layer except the output, which uses tanh. Images are scaled to to match.
- 5Use LeakyReLU with slope 0.2 in every discriminator layer.
A transposed convolution is the gradient of a convolution with respect to its input. Each input pixel stamps a scaled copy of the kernel into the output, the copies step pixels apart, and overlapping copies add up. With stride , padding and output padding , the two layer types change size as follows:
- the standard DCGAN resizing layer: for the transposed conv, for the conv
- G's first layer (, the "project and reshape" of ) and D's last ()
- extra rows on one side of a transposed conv's output, to pick among the input sizes that a strided conv would map to the same output
Training details from the paper
- Optimiser
- Adam, learning rate (the suggested was too high), (0.9 caused oscillation and instability)
- Batches and init
- mini-batch SGD with batch size 128; all weights from ; no pre-processing beyond scaling to
- Noise
- from a uniform distribution
- Paper's generator
- Datasets
- LSUN bedrooms (about 3M images), ImageNet-1k (32×32 crops), and a scraped set of 3M face images of 10K people
- Features for classification
- trained on ImageNet-1k, all layers max-pooled to and fed to an L2-SVM: 82.8% on CIFAR-10. On SVHN with 1000 labels: 22.48% test error
The PyTorch reference DCGAN uses k = 4, stride 2, padding 1 for every resizing layer. The first G layer and the last D layer are 4×4, stride 1, no padding.
Worked example: sizes, parameters and overlap
| layer | in → out | k, s, p | weights | BN γ, β | params | MACs |
|---|---|---|---|---|---|---|
| project z | 100×1×1 → 512×4×4 | 4, 1, 0 | 819,200 | 1,024 | 820,224 | 0.82M |
| up 1 | 512×4×4 → 256×8×8 | 4, 2, 1 | 2,097,152 | 512 | 2,097,664 | 33.55M |
| up 2 | 256×8×8 → 128×16×16 | 4, 2, 1 | 524,288 | 256 | 524,544 | 33.55M |
| up 3 | 128×16×16 → 64×32×32 | 4, 2, 1 | 131,072 | 128 | 131,200 | 33.55M |
| to RGB + tanh | 64×32×32 → 3×64×64 | 4, 2, 1 | 3,072 | none | 3,072 | 3.15M |
Reading the table
- 1A transposed conv has weights: up 1 has . Every input pixel multiplies the whole kernel, so it costs MACs: M. Halving channels while doubling resolution keeps the middle layers at the same cost.
- 2Total: parameters, which is what
sum(p.numel() for p in G.parameters())prints for the PyTorch model below. More than half sit in up 1, where the channels are widest. - 3Overlap in 1-D. Upsampling a length-4 input of ones with a kernel of ones at , , , gives
[1, 2, 1, 2, 1, 2, 1, 1]. With , it gives[1, 2, 2, 2, 2, 2, 2, 1]: uniform away from the border. - 4In 2-D the counts multiply, so the interior repeats the block . The interior is all 4. The network can learn weights that cancel the pattern, but it must spend capacity doing so, and in practice the pattern often survives as checkerboard artefacts.
- 5The backward pass of a strided conv with , has the same
[1, 2, 1, 2, …]pattern in its input gradient. So an uneven discriminator also pushes an uneven gradient into .
What the latent space learned
The paper's most cited figures probe 's input space. Walking along a line between two gives smooth changes: a TV slowly turns into a window, and a room without a window gains a large one. Sudden jumps would suggest memorised images, and the walks showed none. Vector arithmetic on single was unstable, but averaging the of three samples per concept worked. produced smiling men. "Man with glasses" minus "man without glasses" plus "woman without glasses" produced women with glasses. The authors also used guided backpropagation to show that 's filters respond to beds and windows. When they removed the generator's "window" feature maps, drew other objects where the windows had been.
- latent vectors of three generated samples that show concept A (e.g. smiling woman), chosen by eye
- averaging the three made the arithmetic stable; single vectors often failed
- the paper also added small uniform noise to to show that the result is a region of latent space, not one point
1import torch, torch.nn as nn23nz, ngf, ndf, nc = 100, 64, 64, 345class Generator(nn.Module):6 def __init__(self):7 super().__init__()8 def up(i, o): # x2 upsampling: k=4, s=2, p=19 return [nn.ConvTranspose2d(i, o, 4, 2, 1, bias=False), nn.BatchNorm2d(o), nn.ReLU(True)]10 self.net = nn.Sequential(11 nn.ConvTranspose2d(nz, ngf * 8, 4, 1, 0, bias=False), # (100,1,1) -> (512,4,4): "project and reshape"12 nn.BatchNorm2d(ngf * 8), nn.ReLU(True),13 *up(ngf * 8, ngf * 4), # -> (256,8,8)14 *up(ngf * 4, ngf * 2), # -> (128,16,16)15 *up(ngf * 2, ngf), # -> (64,32,32)16 nn.ConvTranspose2d(ngf, nc, 4, 2, 1, bias=False), # -> (3,64,64), no BN on the output17 nn.Tanh(), # images scaled to [-1, 1]18 )19 def forward(self, z):20 return self.net(z)2122class Discriminator(nn.Module):23 def __init__(self):24 super().__init__()25 def down(i, o, bn=True): # x1/2 downsampling, no pooling26 return [nn.Conv2d(i, o, 4, 2, 1, bias=False), *([nn.BatchNorm2d(o)] if bn else []), nn.LeakyReLU(0.2, True)]27 self.net = nn.Sequential(28 *down(nc, ndf, bn=False), # (3,64,64) -> (64,32,32), no BN on the input layer29 *down(ndf, ndf * 2), # -> (128,16,16)30 *down(ndf * 2, ndf * 4), # -> (256,8,8)31 *down(ndf * 4, ndf * 8), # -> (512,4,4)32 nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False), # -> (1,1,1): one logit33 )34 def forward(self, x):35 return self.net(x).view(-1)3637def init_weights(m): # DCGAN: N(0, 0.02); BN gamma ~ N(1, 0.02), beta = 038 if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):39 nn.init.normal_(m.weight, 0.0, 0.02)40 elif isinstance(m, nn.BatchNorm2d):41 nn.init.normal_(m.weight, 1.0, 0.02); nn.init.zeros_(m.bias)4243G, D = Generator(), Discriminator()44G.apply(init_weights); D.apply(init_weights)45opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))46opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))47bce = nn.BCEWithLogitsLoss()4849z = torch.randn(128, nz, 1, 1)50x_fake = G(z)51print(x_fake.shape, D(x_fake).shape) # torch.Size([128, 3, 64, 64]) torch.Size([128])52print(sum(p.numel() for p in G.parameters()), sum(p.numel() for p in D.parameters())) # 3576704 27655685354def train_step(x_real): # x_real (B,3,64,64) in [-1, 1]55 B = x_real.size(0)56 fake = G(torch.randn(B, nz, 1, 1))57 loss_d = bce(D(x_real), torch.ones(B)) + bce(D(fake.detach()), torch.zeros(B))58 opt_d.zero_grad(); loss_d.backward(); opt_d.step()59 loss_g = bce(D(fake), torch.ones(B)) # non-saturating generator loss60 opt_g.zero_grad(); loss_g.backward(); opt_g.step()61 return loss_d.item(), loss_g.item()6263print(len(train_step(torch.rand(128, 3, 64, 64) * 2 - 1))) # 26465# latent arithmetic: average 3 z's per concept, then combine66z_smile_w, z_neutral_w, z_neutral_m = (torch.randn(3, nz, 1, 1).mean(0, keepdim=True) for _ in range(3))67with torch.no_grad():68 G.eval()69 print(G(z_smile_w - z_neutral_w + z_neutral_m).shape) # torch.Size([1, 3, 64, 64])Use when
- +Simple, symmetric recipe that trained stably across bedrooms, faces and ImageNet at 64×64
- +Learned up- and down-sampling; no pooling or hidden dense layers
- +Meaningful latent space: smooth walks and vector arithmetic on averaged codes
- +'s features transfer: 82.8% on CIFAR-10 with a linear SVM
Avoid when
- −Still mode-collapses and still uses the JS-type GAN loss; long runs can oscillate
- −Batch norm couples samples in a batch and the network to the batch size
- −Transposed convs with not divisible by create checkerboards
- −Does not scale well beyond 64×64 without progressive growing, better losses or normalisation
Check yourself
Which layers in DCGAN do NOT use batch normalization?
Check yourself
A transposed conv with stride 2 and kernel size 3 upsamples 8×8 to 16×16. Why does its output tend to show a checkerboard?
Remember
- ✓DCGAN rules: strided convs in , transposed convs in , no pooling and no hidden dense layers; BN everywhere except 's output and 's input.
- ✓Activations and optimiser: ReLU + tanh output in , LeakyReLU(0.2) in ; Adam with lr , , batch 128, init .
- ✓Resizing layer , , doubles or halves resolution exactly; transposed conv output .
- ✓ not divisible by gives uneven kernel overlap: checkerboard artefacts.
- ✓Latent space is smooth; averaged-z vector arithmetic (smiling woman − neutral woman + neutral man) works; D's features transfer to classification.