Vector-quantized VAE (VQ-VAE)

VQ-VAE (van den Oord et al., 2017) replaces the continuous VAE latent with the nearest of K learned code vectors. A straight-through gradient skips the argmin, codebook and β-weighted commitment losses pull codes and encoder outputs together, and an autoregressive prior is then fitted to the codes.

14 min read1258 wordscore0 viewsupdated 2026-10-12

Definition

Vector-quantized VAE (VQ-VAE) An autoencoder with a discrete bottleneck, introduced by Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu in "Neural Discrete Representation Learning" (NeurIPS 2017). The encoder outputs a grid of vectors ze(x)∈RDz_e(x)\in\mathbb R^{D}. Each vector is replaced by its nearest neighbour eke_k in a learned codebook {e1,…,eK}\{e_1,\dots,e_K\}, and the decoder reconstructs xx from the quantised grid zq(x)z_q(x). The image is thus represented by a grid of integers in {1,…,K}\{1,\dots,K\}. A powerful autoregressive prior (PixelCNN, WaveNet, a transformer) is trained afterwards on these integers to generate new samples.

A Variational autoencoder (VAE) with a strong autoregressive decoder tends to ignore its latent: the decoder models xx alone, the KL term pushes q(z∣x)q(z\mid x) onto the prior, and the code carries nothing (posterior collapse). Discrete latents had been tried before, but they needed high-variance score-function gradients (NVIL, VIMCO) or continuous relaxations. Van den Oord et al. made the posterior deterministic and one-hot, so the KL term is a constant. A crude gradient approximation, the straight-through estimator (Bengio, Léonard and Courville, 2013), turned out to be enough. The result trained as easily as a plain autoencoder, did not collapse, and its codes were compact enough that a PixelCNN or WaveNet prior over them could generate images, video frames and speech. The same idea underlies VQ-VAE-2 (Razavi, van den Oord and Vinyals, NeurIPS 2019), VQGAN, the DALL·E dVAE and most image and audio tokenizers since.

Quantisation

k(x)=arg⁡min⁡j∈{1..K}∥ze(x)−ej∥22,zq(x)=ek(x),q(z=k∣x)={1k=k(x)0otherwisek(x)=\arg\min_{j\in\{1..K\}}\big\|z_e(x)-e_j\big\|_2^2,\qquad z_q(x)=e_{k(x)},\qquad q(z=k\mid x)=\begin{cases}1 & k=k(x)\\0&\text{otherwise}\end{cases}
ze(x)z_e(x)
encoder output at one latent position, a DD-dimensional vector; quantisation is applied to every position of the grid independently
ej∈RDe_j\in\mathbb R^{D}
codebook vectors (the paper's ImageNet model uses K=512K=512)
k(x)k(x)
the code index stored for this position: log⁡2K\log_2K bits (9 bits for K=512K=512)
q(z∣x)q(z\mid x)
the posterior is a one-hot categorical; with a uniform prior p(z)=1/Kp(z)=1/K the KL term equals log⁡K\log K, a constant

The input space is divided into the Voronoi cells of the codebook, and every encoder output in a cell is replaced by that cell's code. Quantisation is lossy on purpose: the bottleneck is log⁡2K\log_2K bits per position, so the ImageNet model's 32×3232\times32 grid with K=512K=512 stores 32⋅32⋅9=921632\cdot32\cdot9=9216 bits. The raw 128×128×3128\times128\times3 8-bit image has 393,216393{,}216, about 42.7× more.

Training: three terms and a straight-through gradient

L=−log⁡p(x∣zq(x))⏟reconstruction+∥ sg[ze(x)]−e ∥22⏟codebook+β ∥ ze(x)−sg[e] ∥22⏟commitment\mathcal L=\underbrace{-\log p\big(x\mid z_q(x)\big)}_{\text{reconstruction}}+\underbrace{\big\|\,\mathrm{sg}[z_e(x)]-e\,\big\|_2^2}_{\text{codebook}}+\underbrace{\beta\,\big\|\,z_e(x)-\mathrm{sg}[e]\,\big\|_2^2}_{\text{commitment}}
sg[⋅]\mathrm{sg}[\cdot]
stop-gradient: identity in the forward pass, zero derivative in the backward pass (.detach())
ee
the selected code ek(x)e_{k(x)}
reconstruction\text{reconstruction}
trains decoder and (through the straight-through copy) encoder; it does not train the codebook
codebook\text{codebook}
moves the selected code toward the encoder output; the encoder is frozen in this term
β\beta
commitment weight, 0.25 in all of the paper's experiments; results were robust for β\beta from 0.1 to 2.0

The argmin has no useful gradient, so the backward pass treats quantisation as the identity. The decoder's gradient at zqz_q is copied unchanged to zez_e:

∂L∂ze:=∂L∂zq+2β (ze−e),implemented aszq=ze+sg[e−ze]\frac{\partial\mathcal L}{\partial z_e}:=\frac{\partial\mathcal L}{\partial z_q}+2\beta\,(z_e-e),\qquad\text{implemented as}\quad z_q=z_e+\mathrm{sg}[e-z_e]
∂L/∂zq\partial\mathcal L/\partial z_q
the reconstruction gradient, passed through as if zq=zez_q=z_e
2β(ze−e)2\beta(z_e-e)
the commitment term's gradient, which pulls the encoder output toward its code

The copy is biased, but it points in a sensible direction because zqz_q and zez_e are close. The decoder gradient tells the encoder how its output should move. If it moves far enough, a different code is picked in the next forward pass. The two quadratic terms keep zqz_q and zez_e close from both sides. The codebook term moves codes toward the encoder outputs, which makes it online kk-means. The commitment term stops the encoder's outputs from drifting arbitrarily far from the codebook, since the embedding space is dimensionless and zez_e could otherwise grow without bound.

EMA codebook updates

The paper's appendix gives an alternative to the codebook loss: set each code to an exponential moving average of the encoder outputs assigned to it. This is kk-means with a running mean. It does not depend on the optimiser's learning rate and is used in VQ-VAE-2 and many later tokenizers. Only the commitment term then enters the loss.

Nk←γNk+(1−γ) nk,mk←γmk+(1−γ)∑i: k(xi)=kze(xi),ek←mkNkN_k\leftarrow\gamma N_k+(1-\gamma)\,n_k,\qquad m_k\leftarrow\gamma m_k+(1-\gamma)\sum_{i:\,k(x_i)=k}z_e(x_i),\qquad e_k\leftarrow\frac{m_k}{N_k}
nkn_k
number of encoder outputs in the minibatch assigned to code kk
Nk, mkN_k,\ m_k
running count and running sum; their ratio is the running mean
γ\gamma
decay, 0.99 in the paper; implementations add Laplace smoothing to NkN_k so an unused code does not divide by zero

Codebook collapse

Both update rules only move codes that win at least one encoder output. A code that starts far from the data, or is overtaken by its neighbours, gets nk=0n_k=0. It receives no gradient and no EMA update, and stays dead for the rest of training. Often only a fraction of the KK codes is used, and the effective bottleneck is smaller than log⁡2K\log_2K bits. The usual diagnostic is the perplexity exp⁡(−∑kpklog⁡pk)\exp\big(-\sum_kp_k\log p_k\big) of the code-usage distribution pkp_k. It equals KK when all codes are used equally and 1 when one code takes everything. Common remedies:

Keeping codes alive

  1. 1Initialise from data: set the codebook to encoder outputs, or run kk-means on the first batches, instead of a random or uniform init far from the data.
  2. 2Restart dead codes: periodically move codes whose usage falls below a threshold onto random recent encoder outputs (Jukebox, Dhariwal et al. 2020; SoundStream).
  3. 3Lower-dimensional, normalised codes: look up codes in a low-dimensional space and ℓ2\ell_2-normalise both zez_e and ee, so the lookup uses cosine similarity (ViT-VQGAN, Yu et al., ICLR 2022).
  4. 4Remove the learned codebook: finite scalar quantisation (FSQ, Mentzer et al. 2023) rounds each of a few bounded dimensions to a fixed grid, so there is nothing to collapse.
z_q = e_k, k = argminⱼ ‖z_e − eⱼ‖² · cells = Voronoi regions of the 16 codesupdate step20 of 60codes in use5 of 16 (11 dead)perplexity exp H(usage)4.80 (log₂ = 2.26 of 4 bits)codebook ‖sg[z_e] − e‖²0.5792commitment β‖z_e − sg[e]‖²0.1448 (β = 0.25)dead-code resetsoffusage nₖ per code (of 360 vectors)max 91 · ● = dead code, never chosenEMA: Nₖ ← γNₖ + (1−γ)nₖ, mₖ ← γmₖ + (1−γ)Σz_e, eₖ = mₖ/Nₖ; unused codes stay where they aresame value up to β; the codebook term updates e, the commitment term only z_ethe encoder is frozen here, so only the codebook learns; ✕ = dead code
encoder out
codebook init
update
fig.The encoder is frozen: its 360 outputs z_e are fixed points (8 Gaussians or two moons), and only a K = 16 codebook learns, one full-batch step at a time. Shaded regions are the codes' Voronoi cells and lines are the code trajectories. With all codes initialised in the bottom-left corner, only 2 codes win any point at step 0 (perplexity 1.40, codebook loss 15.79). EMA (γ = 0.99) moves the winners to the mean of their points in one step, and then 5 codes are in use. At step 20, 5 of 16 codes are in use and 11 are dead (red ✕), with perplexity 4.80 (2.26 of 4 bits), codebook loss 0.5792 and commitment loss 0.25 × 0.5792 = 0.1448. The dead codes never move. Codebook-loss SGD (η = 0.5) is slower: at step 20 it has 4 codes in use, perplexity 2.10 and loss 2.3334, and only 3.83 and 1.1385 by step 60. Switching on dead-code restarts revives 11 codes after the first step and all 16 by step 5 (15 resets), reaching perplexity 13.40 and loss 0.0756 at step 60. Initialising from encoder outputs avoids the problem altogether: 15 codes are used at step 0 and 16 by step 1, with perplexity 13.70 and loss 0.0767 at step 60. On two moons the corner init leaves only 2 of 16 codes alive (loss 1.3534 at step 60); restarts bring it to 16 codes, perplexity 15.16 and loss 0.0690.

The prior over codes

During VQ-VAE training the prior is kept uniform. Afterwards the encoder is run over the training set, and an autoregressive model is fitted to the resulting index grids (see Autoregressive generative models). The paper used a PixelCNN over the 32×3232\times32 ImageNet latents and a WaveNet over audio latents. To generate, sample a grid of indices from the prior, look up the codes and decode. The prior works on a sequence 48× shorter than the image's 128⋅128⋅3128\cdot128\cdot3 sub-pixels, and its capacity goes to global structure rather than pixel noise. VQ-VAE-2 makes this hierarchical. For 256×256256\times256 ImageNet a 32×3232\times32 top-level code captures global structure and a 64×6464\times64 bottom-level code, conditioned on the top, adds local detail. Each level gets its own PixelCNN prior with self-attention. Replacing the PixelCNN prior by a transformer, and the reconstruction loss by perceptual and adversarial losses, gives VQGAN (see Autoregressive image generation).

Setup in the paper (van den Oord et al., 2017)

Images
ImageNet 128×128×3128\times128\times3 compressed to a 32×32×132\times32\times1 grid with K=512K=512; encoder of two stride-2 4×44\times4 convolutions and two 3×33\times3 residual blocks, 256 hidden units; decoder mirrors it with transposed convolutions
Optimiser
Adam, learning rate 2⋅10−42\cdot10^{-4}, batch size 128, 250,000 steps
Commitment
β=0.25\beta=0.25 in all experiments; EMA variant with γ=0.99\gamma=0.99 in the appendix
Likelihood
CIFAR-10: 4.67 bits/dim, against 4.51 for a continuous VAE and 5.14 for VIMCO with discrete latents
Audio
VCTK speech; 6 stride-2 convolutions give latents at a 64× lower rate, K=512K=512; a WaveNet decoder conditioned on the codes and a speaker embedding performs voice conversion
VQ-VAE-2
Razavi et al., NeurIPS 2019; two levels (32×3232\times32 and 64×6464\times64) for 256×256256\times256 ImageNet, EMA codebooks, PixelCNN priors with self-attention

Worked example: one vector through the quantiser

Codebook e1=(1,0)e_1=(1,0), e2=(0,1)e_2=(0,1), e3=(−1,0)e_3=(-1,0), encoder output ze=(0.9, 0.2)z_e=(0.9,\,0.2), β=0.25\beta=0.25. Suppose the decoder's gradient at zqz_q is g=∂Lrec/∂zq=(0.4, −0.6)g=\partial\mathcal L_{\text{rec}}/\partial z_q=(0.4,\,-0.6). Losses are summed squared errors, as in the equation above.

codeeⱼ‖z_e − eⱼ‖²selected
1(1, 0)0.05yes
2(0, 1)1.45
3(−1, 0)3.65
Squared distances from z_e to each code (computed with PyTorch)

Forward, backward and one codebook update

  1. 1Forward: k=1k=1 and zq=e1=(1,0)z_q=e_1=(1,0). Codebook loss ∥sg[ze]−e1∥2=0.01+0.04=0.05\|\mathrm{sg}[z_e]-e_1\|^2=0.01+0.04=0.05. Commitment loss 0.25⋅0.05=0.01250.25\cdot0.05=0.0125. The two terms have the same value up to β\beta but update different tensors.
  2. 2Encoder gradient: the straight-through copy gives g=(0.4,−0.6)g=(0.4,-0.6), and the commitment term adds 2β(ze−e1)=0.5⋅(−0.1, 0.2)=(−0.05, 0.1)2\beta(z_e-e_1)=0.5\cdot(-0.1,\,0.2)=(-0.05,\,0.1), so ∂L/∂ze=(0.35, −0.5)\partial\mathcal L/\partial z_e=(0.35,\,-0.5).
  3. 3Codebook gradient: only e1e_1 gets one, 2(e1−ze)=(0.2, −0.4)2(e_1-z_e)=(0.2,\,-0.4); e2e_2 and e3e_3 get (0,0)(0,0). One SGD step with learning rate 0.5 moves e1e_1 exactly onto ze=(0.9, 0.2)z_e=(0.9,\,0.2).
  4. 4EMA instead, with N1=10N_1=10, m1=(10,0)m_1=(10,0) and a batch of two vectors (0.9,0.2)(0.9,0.2) and (1.3,−0.1)(1.3,-0.1) assigned to code 1: N1=0.99⋅10+0.01⋅2=9.92N_1=0.99\cdot10+0.01\cdot2=9.92, m1=(9.922, 0.001)m_1=(9.922,\,0.001), so e1=(1.0002, 0.0001)e_1=(1.0002,\,0.0001). The code moves slowly, by a weighted share of the batch mean (1.1, 0.05)(1.1,\,0.05).
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class VectorQuantizer(nn.Module):
4 def __init__(self, K=512, D=64, beta=0.25, ema=False, gamma=0.99, eps=1e-5):
5 super().__init__()
6 self.K, self.D, self.beta, self.ema, self.gamma, self.eps = K, D, beta, ema, gamma, eps
7 self.codebook = nn.Embedding(K, D)
8 self.codebook.weight.data.uniform_(-1 / K, 1 / K)
9 if ema:
10 self.codebook.weight.requires_grad_(False)
11 self.register_buffer("N", torch.zeros(K))
12 self.register_buffer("m", self.codebook.weight.data.clone())
13
14 def forward(self, z_e): # z_e (B, D, H, W) from the encoder
15 B, D, H, W = z_e.shape
16 z = z_e.permute(0, 2, 3, 1).reshape(-1, D) # (B·H·W, D)
17 e = self.codebook.weight # (K, D)
18 d = z.pow(2).sum(1, keepdim=True) - 2 * z @ e.t() + e.pow(2).sum(1) # squared distances (B·H·W, K)
19 idx = d.argmin(1) # nearest code per vector
20 z_q = e[idx]
21 commit = self.beta * F.mse_loss(z, z_q.detach()) # β‖z_e − sg[e]‖²: trains the encoder
22 if self.ema and self.training:
23 with torch.no_grad():
24 onehot = F.one_hot(idx, self.K).type(z.dtype) # (B·H·W, K)
25 self.N.mul_(self.gamma).add_(onehot.sum(0), alpha=1 - self.gamma)
26 self.m.mul_(self.gamma).add_(onehot.t() @ z, alpha=1 - self.gamma)
27 n = self.N.sum()
28 Nk = (self.N + self.eps) / (n + self.K * self.eps) * n # Laplace smoothing keeps Nk > 0
29 self.codebook.weight.copy_(self.m / Nk[:, None])
30 loss = commit
31 else:
32 loss = F.mse_loss(z_q, z.detach()) + commit # codebook loss ‖sg[z_e] − e‖² + commitment
33 z_q = z + (z_q - z).detach() # straight-through: forward z_q, backward identity
34 counts = torch.bincount(idx, minlength=self.K).float()
35 p = counts / counts.sum()
36 perplexity = torch.exp(-(p[p > 0] * p[p > 0].log()).sum())
37 return z_q.reshape(B, H, W, D).permute(0, 3, 1, 2), loss, idx.reshape(B, H, W), perplexity
38
39class VQVAE(nn.Module):
40 def __init__(self, K=512, D=64, h=128, ema=False):
41 super().__init__()
42 self.enc = nn.Sequential(nn.Conv2d(3, h, 4, 2, 1), nn.ReLU(), nn.Conv2d(h, h, 4, 2, 1), nn.ReLU(), nn.Conv2d(h, D, 1))
43 self.vq = VectorQuantizer(K, D, ema=ema)
44 self.dec = nn.Sequential(nn.Conv2d(D, h, 3, 1, 1), nn.ReLU(), nn.ConvTranspose2d(h, h, 4, 2, 1), nn.ReLU(), nn.ConvTranspose2d(h, 3, 4, 2, 1))
45
46 def forward(self, x):
47 z_q, vq_loss, idx, ppl = self.vq(self.enc(x))
48 x_hat = self.dec(z_q)
49 return x_hat, F.mse_loss(x_hat, x) + vq_loss, idx, ppl
50
51torch.manual_seed(0)
52model = VQVAE(ema=True)
53x = torch.rand(8, 3, 64, 64)
54x_hat, loss, idx, ppl = model(x)
55loss.backward()
56print(x_hat.shape, idx.shape, idx.dtype) # torch.Size([8, 3, 64, 64]) torch.Size([8, 16, 16]) torch.int64
57print(model.enc[0].weight.grad is not None) # True: the reconstruction gradient reaches the encoder
58print(1 <= ppl.item() <= 512) # True: perplexity is between 1 and K
Vector quantiser with straight-through gradient, codebook + commitment loss or EMA updates, and perplexity

Use when

  • +Discrete codes give short sequences for transformer or PixelCNN priors: the basis of VQGAN, DALL·E and audio tokenizers
  • +No posterior collapse: the KL term is constant and capacity is a hard bit budget
  • +Simple to train: reconstruction + two squared errors, one hyperparameter β\beta that is not sensitive
  • +EMA codebooks are optimiser-independent and stable

Avoid when

  • −Straight-through gradients are biased; quantisation error is not optimised directly
  • −Codebook collapse: dead codes waste capacity unless initialisation, restarts or normalised codes are used
  • −Two-stage training: the prior is fitted after the autoencoder, so the ELBO is never optimised jointly
  • −Blurry reconstructions with plain L2 loss; high fidelity needs perceptual or adversarial losses (VQGAN)

Check yourself

Which tensors receive gradient from each VQ-VAE loss term?

Check yourself

A codebook of K = 512 codes shows a usage perplexity of 32 after training. What does this mean?

Remember

  • ✓Quantise: zq=ekz_q=e_k with k=arg⁡min⁡j∥ze−ej∥2k=\arg\min_j\|z_e-e_j\|^2; the one-hot posterior makes the KL a constant log⁡K\log K, and each position stores log⁡2K\log_2K bits.
  • ✓Loss: −log⁡p(x∣zq)+∥sg[ze]−e∥2+β∥ze−sg[e]∥2-\log p(x\mid z_q)+\|\mathrm{sg}[z_e]-e\|^2+\beta\|z_e-\mathrm{sg}[e]\|^2 with β=0.25\beta=0.25; gradients reach the encoder by the straight-through copy zq=ze+sg[e−ze]z_q=z_e+\mathrm{sg}[e-z_e].
  • ✓EMA alternative: Nk←γNk+(1−γ)nkN_k\leftarrow\gamma N_k+(1-\gamma)n_k, mk←γmk+(1−γ)∑zem_k\leftarrow\gamma m_k+(1-\gamma)\sum z_e, ek=mk/Nke_k=m_k/N_k, γ=0.99\gamma=0.99.
  • ✓Codebook collapse: unused codes get no update; monitor perplexity and use data init, dead-code restarts or normalised low-dimensional codes.
  • ✓Generation is two-stage: fit an autoregressive prior (PixelCNN, WaveNet, transformer) to the code grid, sample indices, decode; VQ-VAE-2 adds a 32×32 + 64×64 hierarchy.
← back to Deep Learning roadmap