Definition
Vector-quantized VAE (VQ-VAE) An autoencoder with a discrete bottleneck, introduced by Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu in "Neural Discrete Representation Learning" (NeurIPS 2017). The encoder outputs a grid of vectors . Each vector is replaced by its nearest neighbour in a learned codebook , and the decoder reconstructs from the quantised grid . The image is thus represented by a grid of integers in . A powerful autoregressive prior (PixelCNN, WaveNet, a transformer) is trained afterwards on these integers to generate new samples.
A Variational autoencoder (VAE) with a strong autoregressive decoder tends to ignore its latent: the decoder models alone, the KL term pushes onto the prior, and the code carries nothing (posterior collapse). Discrete latents had been tried before, but they needed high-variance score-function gradients (NVIL, VIMCO) or continuous relaxations. Van den Oord et al. made the posterior deterministic and one-hot, so the KL term is a constant. A crude gradient approximation, the straight-through estimator (Bengio, Léonard and Courville, 2013), turned out to be enough. The result trained as easily as a plain autoencoder, did not collapse, and its codes were compact enough that a PixelCNN or WaveNet prior over them could generate images, video frames and speech. The same idea underlies VQ-VAE-2 (Razavi, van den Oord and Vinyals, NeurIPS 2019), VQGAN, the DALL·E dVAE and most image and audio tokenizers since.
Quantisation
- encoder output at one latent position, a -dimensional vector; quantisation is applied to every position of the grid independently
- codebook vectors (the paper's ImageNet model uses )
- the code index stored for this position: bits (9 bits for )
- the posterior is a one-hot categorical; with a uniform prior the KL term equals , a constant
The input space is divided into the Voronoi cells of the codebook, and every encoder output in a cell is replaced by that cell's code. Quantisation is lossy on purpose: the bottleneck is bits per position, so the ImageNet model's grid with stores bits. The raw 8-bit image has , about 42.7× more.
Training: three terms and a straight-through gradient
- stop-gradient: identity in the forward pass, zero derivative in the backward pass (
.detach()) - the selected code
- trains decoder and (through the straight-through copy) encoder; it does not train the codebook
- moves the selected code toward the encoder output; the encoder is frozen in this term
- commitment weight, 0.25 in all of the paper's experiments; results were robust for from 0.1 to 2.0
The argmin has no useful gradient, so the backward pass treats quantisation as the identity. The decoder's gradient at is copied unchanged to :
- the reconstruction gradient, passed through as if
- the commitment term's gradient, which pulls the encoder output toward its code
The copy is biased, but it points in a sensible direction because and are close. The decoder gradient tells the encoder how its output should move. If it moves far enough, a different code is picked in the next forward pass. The two quadratic terms keep and close from both sides. The codebook term moves codes toward the encoder outputs, which makes it online -means. The commitment term stops the encoder's outputs from drifting arbitrarily far from the codebook, since the embedding space is dimensionless and could otherwise grow without bound.
EMA codebook updates
The paper's appendix gives an alternative to the codebook loss: set each code to an exponential moving average of the encoder outputs assigned to it. This is -means with a running mean. It does not depend on the optimiser's learning rate and is used in VQ-VAE-2 and many later tokenizers. Only the commitment term then enters the loss.
- number of encoder outputs in the minibatch assigned to code
- running count and running sum; their ratio is the running mean
- decay, 0.99 in the paper; implementations add Laplace smoothing to so an unused code does not divide by zero
Codebook collapse
Both update rules only move codes that win at least one encoder output. A code that starts far from the data, or is overtaken by its neighbours, gets . It receives no gradient and no EMA update, and stays dead for the rest of training. Often only a fraction of the codes is used, and the effective bottleneck is smaller than bits. The usual diagnostic is the perplexity of the code-usage distribution . It equals when all codes are used equally and 1 when one code takes everything. Common remedies:
Keeping codes alive
- 1Initialise from data: set the codebook to encoder outputs, or run -means on the first batches, instead of a random or uniform init far from the data.
- 2Restart dead codes: periodically move codes whose usage falls below a threshold onto random recent encoder outputs (Jukebox, Dhariwal et al. 2020; SoundStream).
- 3Lower-dimensional, normalised codes: look up codes in a low-dimensional space and -normalise both and , so the lookup uses cosine similarity (ViT-VQGAN, Yu et al., ICLR 2022).
- 4Remove the learned codebook: finite scalar quantisation (FSQ, Mentzer et al. 2023) rounds each of a few bounded dimensions to a fixed grid, so there is nothing to collapse.
The prior over codes
During VQ-VAE training the prior is kept uniform. Afterwards the encoder is run over the training set, and an autoregressive model is fitted to the resulting index grids (see Autoregressive generative models). The paper used a PixelCNN over the ImageNet latents and a WaveNet over audio latents. To generate, sample a grid of indices from the prior, look up the codes and decode. The prior works on a sequence 48× shorter than the image's sub-pixels, and its capacity goes to global structure rather than pixel noise. VQ-VAE-2 makes this hierarchical. For ImageNet a top-level code captures global structure and a bottom-level code, conditioned on the top, adds local detail. Each level gets its own PixelCNN prior with self-attention. Replacing the PixelCNN prior by a transformer, and the reconstruction loss by perceptual and adversarial losses, gives VQGAN (see Autoregressive image generation).
Setup in the paper (van den Oord et al., 2017)
- Images
- ImageNet compressed to a grid with ; encoder of two stride-2 convolutions and two residual blocks, 256 hidden units; decoder mirrors it with transposed convolutions
- Optimiser
- Adam, learning rate , batch size 128, 250,000 steps
- Commitment
- in all experiments; EMA variant with in the appendix
- Likelihood
- CIFAR-10: 4.67 bits/dim, against 4.51 for a continuous VAE and 5.14 for VIMCO with discrete latents
- Audio
- VCTK speech; 6 stride-2 convolutions give latents at a 64× lower rate, ; a WaveNet decoder conditioned on the codes and a speaker embedding performs voice conversion
- VQ-VAE-2
- Razavi et al., NeurIPS 2019; two levels ( and ) for ImageNet, EMA codebooks, PixelCNN priors with self-attention
Worked example: one vector through the quantiser
Codebook , , , encoder output , . Suppose the decoder's gradient at is . Losses are summed squared errors, as in the equation above.
| code | eⱼ | ‖z_e − eⱼ‖² | selected |
|---|---|---|---|
| 1 | (1, 0) | 0.05 | yes |
| 2 | (0, 1) | 1.45 | |
| 3 | (−1, 0) | 3.65 |
Forward, backward and one codebook update
- 1Forward: and . Codebook loss . Commitment loss . The two terms have the same value up to but update different tensors.
- 2Encoder gradient: the straight-through copy gives , and the commitment term adds , so .
- 3Codebook gradient: only gets one, ; and get . One SGD step with learning rate 0.5 moves exactly onto .
- 4EMA instead, with , and a batch of two vectors and assigned to code 1: , , so . The code moves slowly, by a weighted share of the batch mean .
1import torch, torch.nn as nn, torch.nn.functional as F23class VectorQuantizer(nn.Module):4 def __init__(self, K=512, D=64, beta=0.25, ema=False, gamma=0.99, eps=1e-5):5 super().__init__()6 self.K, self.D, self.beta, self.ema, self.gamma, self.eps = K, D, beta, ema, gamma, eps7 self.codebook = nn.Embedding(K, D)8 self.codebook.weight.data.uniform_(-1 / K, 1 / K)9 if ema:10 self.codebook.weight.requires_grad_(False)11 self.register_buffer("N", torch.zeros(K))12 self.register_buffer("m", self.codebook.weight.data.clone())1314 def forward(self, z_e): # z_e (B, D, H, W) from the encoder15 B, D, H, W = z_e.shape16 z = z_e.permute(0, 2, 3, 1).reshape(-1, D) # (B·H·W, D)17 e = self.codebook.weight # (K, D)18 d = z.pow(2).sum(1, keepdim=True) - 2 * z @ e.t() + e.pow(2).sum(1) # squared distances (B·H·W, K)19 idx = d.argmin(1) # nearest code per vector20 z_q = e[idx]21 commit = self.beta * F.mse_loss(z, z_q.detach()) # β‖z_e − sg[e]‖²: trains the encoder22 if self.ema and self.training:23 with torch.no_grad():24 onehot = F.one_hot(idx, self.K).type(z.dtype) # (B·H·W, K)25 self.N.mul_(self.gamma).add_(onehot.sum(0), alpha=1 - self.gamma)26 self.m.mul_(self.gamma).add_(onehot.t() @ z, alpha=1 - self.gamma)27 n = self.N.sum()28 Nk = (self.N + self.eps) / (n + self.K * self.eps) * n # Laplace smoothing keeps Nk > 029 self.codebook.weight.copy_(self.m / Nk[:, None])30 loss = commit31 else:32 loss = F.mse_loss(z_q, z.detach()) + commit # codebook loss ‖sg[z_e] − e‖² + commitment33 z_q = z + (z_q - z).detach() # straight-through: forward z_q, backward identity34 counts = torch.bincount(idx, minlength=self.K).float()35 p = counts / counts.sum()36 perplexity = torch.exp(-(p[p > 0] * p[p > 0].log()).sum())37 return z_q.reshape(B, H, W, D).permute(0, 3, 1, 2), loss, idx.reshape(B, H, W), perplexity3839class VQVAE(nn.Module):40 def __init__(self, K=512, D=64, h=128, ema=False):41 super().__init__()42 self.enc = nn.Sequential(nn.Conv2d(3, h, 4, 2, 1), nn.ReLU(), nn.Conv2d(h, h, 4, 2, 1), nn.ReLU(), nn.Conv2d(h, D, 1))43 self.vq = VectorQuantizer(K, D, ema=ema)44 self.dec = nn.Sequential(nn.Conv2d(D, h, 3, 1, 1), nn.ReLU(), nn.ConvTranspose2d(h, h, 4, 2, 1), nn.ReLU(), nn.ConvTranspose2d(h, 3, 4, 2, 1))4546 def forward(self, x):47 z_q, vq_loss, idx, ppl = self.vq(self.enc(x))48 x_hat = self.dec(z_q)49 return x_hat, F.mse_loss(x_hat, x) + vq_loss, idx, ppl5051torch.manual_seed(0)52model = VQVAE(ema=True)53x = torch.rand(8, 3, 64, 64)54x_hat, loss, idx, ppl = model(x)55loss.backward()56print(x_hat.shape, idx.shape, idx.dtype) # torch.Size([8, 3, 64, 64]) torch.Size([8, 16, 16]) torch.int6457print(model.enc[0].weight.grad is not None) # True: the reconstruction gradient reaches the encoder58print(1 <= ppl.item() <= 512) # True: perplexity is between 1 and KUse when
- +Discrete codes give short sequences for transformer or PixelCNN priors: the basis of VQGAN, DALL·E and audio tokenizers
- +No posterior collapse: the KL term is constant and capacity is a hard bit budget
- +Simple to train: reconstruction + two squared errors, one hyperparameter that is not sensitive
- +EMA codebooks are optimiser-independent and stable
Avoid when
- −Straight-through gradients are biased; quantisation error is not optimised directly
- −Codebook collapse: dead codes waste capacity unless initialisation, restarts or normalised codes are used
- −Two-stage training: the prior is fitted after the autoencoder, so the ELBO is never optimised jointly
- −Blurry reconstructions with plain L2 loss; high fidelity needs perceptual or adversarial losses (VQGAN)
Check yourself
Which tensors receive gradient from each VQ-VAE loss term?
Check yourself
A codebook of K = 512 codes shows a usage perplexity of 32 after training. What does this mean?
Remember
- ✓Quantise: with ; the one-hot posterior makes the KL a constant , and each position stores bits.
- ✓Loss: with ; gradients reach the encoder by the straight-through copy .
- ✓EMA alternative: , , , .
- ✓Codebook collapse: unused codes get no update; monitor perplexity and use data init, dead-code restarts or normalised low-dimensional codes.
- ✓Generation is two-stage: fit an autoregressive prior (PixelCNN, WaveNet, transformer) to the code grid, sample indices, decode; VQ-VAE-2 adds a 32×32 + 64×64 hierarchy.