# Vector-quantized VAE (VQ-VAE)

> VQ-VAE (van den Oord et al., 2017) replaces the continuous VAE latent with the nearest of K learned code vectors. A straight-through gradient skips the argmin, codebook and β-weighted commitment losses pull codes and encoder outputs together, and an autoregressive prior is then fitted to the codes.

Source: https://www.mlfoundry.in/learn/vector-quantized-vae-vq-vae  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 14 min read · updated 2026-10-12

**Vector-quantized VAE (VQ-VAE):** An autoencoder with a **discrete** bottleneck, introduced by Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu in "Neural Discrete Representation Learning" (NeurIPS 2017). The encoder outputs a grid of vectors $z_e(x)\in\mathbb R^{D}$. Each vector is replaced by its nearest neighbour $e_k$ in a learned **codebook** $\{e_1,\dots,e_K\}$, and the decoder reconstructs $x$ from the quantised grid $z_q(x)$. The image is thus represented by a grid of integers in $\{1,\dots,K\}$. A powerful autoregressive prior (PixelCNN, WaveNet, a transformer) is trained afterwards on these integers to generate new samples.

A **Variational autoencoder (VAE)** with a strong autoregressive decoder tends to ignore its latent: the decoder models $x$ alone, the KL term pushes $q(z\mid x)$ onto the prior, and the code carries nothing (**posterior collapse**). Discrete latents had been tried before, but they needed high-variance score-function gradients (NVIL, VIMCO) or continuous relaxations. Van den Oord et al. made the posterior **deterministic and one-hot**, so the KL term is a constant. A crude gradient approximation, the **straight-through estimator** (Bengio, Léonard and Courville, 2013), turned out to be enough. The result trained as easily as a plain autoencoder, did not collapse, and its codes were compact enough that a PixelCNN or WaveNet prior over them could generate images, video frames and speech. The same idea underlies VQ-VAE-2 (Razavi, van den Oord and Vinyals, NeurIPS 2019), VQGAN, the DALL·E dVAE and most image and audio tokenizers since.

## Quantisation

$$
k(x)=\arg\min_{j\in\{1..K\}}\big\|z_e(x)-e_j\big\|_2^2,\qquad z_q(x)=e_{k(x)},\qquad q(z=k\mid x)=\begin{cases}1 & k=k(x)\\0&\text{otherwise}\end{cases}
$$

- $z_e(x)$: encoder output at one latent position, a $D$-dimensional vector; quantisation is applied to every position of the grid independently
- $e_j\in\mathbb R^{D}$: codebook vectors (the paper's ImageNet model uses $K=512$)
- $k(x)$: the code index stored for this position: $\log_2K$ bits (9 bits for $K=512$)
- $q(z\mid x)$: the posterior is a one-hot categorical; with a uniform prior $p(z)=1/K$ the KL term equals $\log K$, a constant

The input space is divided into the **Voronoi cells** of the codebook, and every encoder output in a cell is replaced by that cell's code. Quantisation is lossy on purpose: the bottleneck is $\log_2K$ bits per position, so the ImageNet model's $32\times32$ grid with $K=512$ stores $32\cdot32\cdot9=9216$ bits. The raw $128\times128\times3$ 8-bit image has $393{,}216$, about 42.7× more.

## Training: three terms and a straight-through gradient

$$
\mathcal L=\underbrace{-\log p\big(x\mid z_q(x)\big)}_{\text{reconstruction}}+\underbrace{\big\|\,\mathrm{sg}[z_e(x)]-e\,\big\|_2^2}_{\text{codebook}}+\underbrace{\beta\,\big\|\,z_e(x)-\mathrm{sg}[e]\,\big\|_2^2}_{\text{commitment}}
$$

- $\mathrm{sg}[\cdot]$: stop-gradient: identity in the forward pass, zero derivative in the backward pass (`.detach()`)
- $e$: the selected code $e_{k(x)}$
- $\text{reconstruction}$: trains decoder and (through the straight-through copy) encoder; it does not train the codebook
- $\text{codebook}$: moves the selected code toward the encoder output; the encoder is frozen in this term
- $\beta$: commitment weight, 0.25 in all of the paper's experiments; results were robust for $\beta$ from 0.1 to 2.0

The argmin has no useful gradient, so the backward pass treats quantisation as the identity. The decoder's gradient at $z_q$ is copied unchanged to $z_e$:

$$
\frac{\partial\mathcal L}{\partial z_e}:=\frac{\partial\mathcal L}{\partial z_q}+2\beta\,(z_e-e),\qquad\text{implemented as}\quad z_q=z_e+\mathrm{sg}[e-z_e]
$$

- $\partial\mathcal L/\partial z_q$: the reconstruction gradient, passed through as if $z_q=z_e$
- $2\beta(z_e-e)$: the commitment term's gradient, which pulls the encoder output toward its code

The copy is biased, but it points in a sensible direction because $z_q$ and $z_e$ are close. The decoder gradient tells the encoder how its output should move. If it moves far enough, a different code is picked in the next forward pass. The two quadratic terms keep $z_q$ and $z_e$ close from both sides. The codebook term moves codes toward the encoder outputs, which makes it online $k$-means. The commitment term stops the encoder's outputs from drifting arbitrarily far from the codebook, since the embedding space is dimensionless and $z_e$ could otherwise grow without bound.

## EMA codebook updates

The paper's appendix gives an alternative to the codebook loss: set each code to an exponential moving average of the encoder outputs assigned to it. This is $k$-means with a running mean. It does not depend on the optimiser's learning rate and is used in VQ-VAE-2 and many later tokenizers. Only the commitment term then enters the loss.

$$
N_k\leftarrow\gamma N_k+(1-\gamma)\,n_k,\qquad m_k\leftarrow\gamma m_k+(1-\gamma)\sum_{i:\,k(x_i)=k}z_e(x_i),\qquad e_k\leftarrow\frac{m_k}{N_k}
$$

- $n_k$: number of encoder outputs in the minibatch assigned to code $k$
- $N_k,\ m_k$: running count and running sum; their ratio is the running mean
- $\gamma$: decay, 0.99 in the paper; implementations add Laplace smoothing to $N_k$ so an unused code does not divide by zero

## Codebook collapse

Both update rules only move codes that win at least one encoder output. A code that starts far from the data, or is overtaken by its neighbours, gets $n_k=0$. It receives no gradient and no EMA update, and stays **dead** for the rest of training. Often only a fraction of the $K$ codes is used, and the effective bottleneck is smaller than $\log_2K$ bits. The usual diagnostic is the **perplexity** $\exp\big(-\sum_kp_k\log p_k\big)$ of the code-usage distribution $p_k$. It equals $K$ when all codes are used equally and 1 when one code takes everything. Common remedies:

**Keeping codes alive**

1. **Initialise from data**: set the codebook to encoder outputs, or run $k$-means on the first batches, instead of a random or uniform init far from the data.
2. **Restart dead codes**: periodically move codes whose usage falls below a threshold onto random recent encoder outputs (Jukebox, Dhariwal et al. 2020; SoundStream).
3. **Lower-dimensional, normalised codes**: look up codes in a low-dimensional space and $\ell_2$-normalise both $z_e$ and $e$, so the lookup uses cosine similarity (ViT-VQGAN, Yu et al., ICLR 2022).
4. **Remove the learned codebook**: finite scalar quantisation (FSQ, Mentzer et al. 2023) rounds each of a few bounded dimensions to a fixed grid, so there is nothing to collapse.

> Figure: The encoder is frozen: its 360 outputs z_e are fixed points (8 Gaussians or two moons), and only a K = 16 codebook learns, one full-batch step at a time. Shaded regions are the codes' Voronoi cells and lines are the code trajectories. With all codes initialised in the bottom-left corner, only 2 codes win any point at step 0 (perplexity 1.40, codebook loss 15.79). EMA (γ = 0.99) moves the winners to the mean of their points in one step, and then 5 codes are in use. At step 20, 5 of 16 codes are in use and 11 are dead (red ✕), with perplexity 4.80 (2.26 of 4 bits), codebook loss 0.5792 and commitment loss 0.25 × 0.5792 = 0.1448. The dead codes never move. Codebook-loss SGD (η = 0.5) is slower: at step 20 it has 4 codes in use, perplexity 2.10 and loss 2.3334, and only 3.83 and 1.1385 by step 60. Switching on dead-code restarts revives 11 codes after the first step and all 16 by step 5 (15 resets), reaching perplexity 13.40 and loss 0.0756 at step 60. Initialising from encoder outputs avoids the problem altogether: 15 codes are used at step 0 and 16 by step 1, with perplexity 13.70 and loss 0.0767 at step 60. On two moons the corner init leaves only 2 of 16 codes alive (loss 1.3534 at step 60); restarts bring it to 16 codes, perplexity 15.16 and loss 0.0690.

## The prior over codes

During VQ-VAE training the prior is kept uniform. Afterwards the encoder is run over the training set, and an autoregressive model is fitted to the resulting index grids (see **Autoregressive generative models**). The paper used a PixelCNN over the $32\times32$ ImageNet latents and a WaveNet over audio latents. To generate, sample a grid of indices from the prior, look up the codes and decode. The prior works on a sequence 48× shorter than the image's $128\cdot128\cdot3$ sub-pixels, and its capacity goes to global structure rather than pixel noise. VQ-VAE-2 makes this hierarchical. For $256\times256$ ImageNet a $32\times32$ top-level code captures global structure and a $64\times64$ bottom-level code, conditioned on the top, adds local detail. Each level gets its own PixelCNN prior with self-attention. Replacing the PixelCNN prior by a transformer, and the reconstruction loss by perceptual and adversarial losses, gives VQGAN (see **Autoregressive image generation**).

**Setup in the paper (van den Oord et al., 2017)**

- **Images:** ImageNet $128\times128\times3$ compressed to a $32\times32\times1$ grid with $K=512$; encoder of two stride-2 $4\times4$ convolutions and two $3\times3$ residual blocks, 256 hidden units; decoder mirrors it with transposed convolutions
- **Optimiser:** Adam, learning rate $2\cdot10^{-4}$, batch size 128, 250,000 steps
- **Commitment:** $\beta=0.25$ in all experiments; EMA variant with $\gamma=0.99$ in the appendix
- **Likelihood:** CIFAR-10: 4.67 bits/dim, against 4.51 for a continuous VAE and 5.14 for VIMCO with discrete latents
- **Audio:** VCTK speech; 6 stride-2 convolutions give latents at a 64× lower rate, $K=512$; a WaveNet decoder conditioned on the codes and a speaker embedding performs voice conversion
- **VQ-VAE-2:** Razavi et al., NeurIPS 2019; two levels ($32\times32$ and $64\times64$) for $256\times256$ ImageNet, EMA codebooks, PixelCNN priors with self-attention

## Worked example: one vector through the quantiser

Codebook $e_1=(1,0)$, $e_2=(0,1)$, $e_3=(-1,0)$, encoder output $z_e=(0.9,\,0.2)$, $\beta=0.25$. Suppose the decoder's gradient at $z_q$ is $g=\partial\mathcal L_{\text{rec}}/\partial z_q=(0.4,\,-0.6)$. Losses are summed squared errors, as in the equation above.

Squared distances from z_e to each code (computed with PyTorch)

| code | eⱼ | ‖z_e − eⱼ‖² | selected |
| --- | --- | --- | --- |
| 1 | (1, 0) | 0.05 | yes |
| 2 | (0, 1) | 1.45 |  |
| 3 | (−1, 0) | 3.65 |  |

**Forward, backward and one codebook update**

1. Forward: $k=1$ and $z_q=e_1=(1,0)$. Codebook loss $\|\mathrm{sg}[z_e]-e_1\|^2=0.01+0.04=0.05$. Commitment loss $0.25\cdot0.05=0.0125$. The two terms have the same value up to $\beta$ but update different tensors.
2. Encoder gradient: the straight-through copy gives $g=(0.4,-0.6)$, and the commitment term adds $2\beta(z_e-e_1)=0.5\cdot(-0.1,\,0.2)=(-0.05,\,0.1)$, so $\partial\mathcal L/\partial z_e=(0.35,\,-0.5)$.
3. Codebook gradient: only $e_1$ gets one, $2(e_1-z_e)=(0.2,\,-0.4)$; $e_2$ and $e_3$ get $(0,0)$. One SGD step with learning rate 0.5 moves $e_1$ exactly onto $z_e=(0.9,\,0.2)$.
4. EMA instead, with $N_1=10$, $m_1=(10,0)$ and a batch of two vectors $(0.9,0.2)$ and $(1.3,-0.1)$ assigned to code 1: $N_1=0.99\cdot10+0.01\cdot2=9.92$, $m_1=(9.922,\,0.001)$, so $e_1=(1.0002,\,0.0001)$. The code moves slowly, by a weighted share of the batch mean $(1.1,\,0.05)$.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class VectorQuantizer(nn.Module):
    def __init__(self, K=512, D=64, beta=0.25, ema=False, gamma=0.99, eps=1e-5):
        super().__init__()
        self.K, self.D, self.beta, self.ema, self.gamma, self.eps = K, D, beta, ema, gamma, eps
        self.codebook = nn.Embedding(K, D)
        self.codebook.weight.data.uniform_(-1 / K, 1 / K)
        if ema:
            self.codebook.weight.requires_grad_(False)
            self.register_buffer("N", torch.zeros(K))
            self.register_buffer("m", self.codebook.weight.data.clone())

    def forward(self, z_e):                                   # z_e (B, D, H, W) from the encoder
        B, D, H, W = z_e.shape
        z = z_e.permute(0, 2, 3, 1).reshape(-1, D)            # (B·H·W, D)
        e = self.codebook.weight                              # (K, D)
        d = z.pow(2).sum(1, keepdim=True) - 2 * z @ e.t() + e.pow(2).sum(1)   # squared distances (B·H·W, K)
        idx = d.argmin(1)                                     # nearest code per vector
        z_q = e[idx]
        commit = self.beta * F.mse_loss(z, z_q.detach())      # β‖z_e − sg[e]‖²: trains the encoder
        if self.ema and self.training:
            with torch.no_grad():
                onehot = F.one_hot(idx, self.K).type(z.dtype)  # (B·H·W, K)
                self.N.mul_(self.gamma).add_(onehot.sum(0), alpha=1 - self.gamma)
                self.m.mul_(self.gamma).add_(onehot.t() @ z, alpha=1 - self.gamma)
                n = self.N.sum()
                Nk = (self.N + self.eps) / (n + self.K * self.eps) * n   # Laplace smoothing keeps Nk > 0
                self.codebook.weight.copy_(self.m / Nk[:, None])
            loss = commit
        else:
            loss = F.mse_loss(z_q, z.detach()) + commit       # codebook loss ‖sg[z_e] − e‖² + commitment
        z_q = z + (z_q - z).detach()                          # straight-through: forward z_q, backward identity
        counts = torch.bincount(idx, minlength=self.K).float()
        p = counts / counts.sum()
        perplexity = torch.exp(-(p[p > 0] * p[p > 0].log()).sum())
        return z_q.reshape(B, H, W, D).permute(0, 3, 1, 2), loss, idx.reshape(B, H, W), perplexity

class VQVAE(nn.Module):
    def __init__(self, K=512, D=64, h=128, ema=False):
        super().__init__()
        self.enc = nn.Sequential(nn.Conv2d(3, h, 4, 2, 1), nn.ReLU(), nn.Conv2d(h, h, 4, 2, 1), nn.ReLU(), nn.Conv2d(h, D, 1))
        self.vq = VectorQuantizer(K, D, ema=ema)
        self.dec = nn.Sequential(nn.Conv2d(D, h, 3, 1, 1), nn.ReLU(), nn.ConvTranspose2d(h, h, 4, 2, 1), nn.ReLU(), nn.ConvTranspose2d(h, 3, 4, 2, 1))

    def forward(self, x):
        z_q, vq_loss, idx, ppl = self.vq(self.enc(x))
        x_hat = self.dec(z_q)
        return x_hat, F.mse_loss(x_hat, x) + vq_loss, idx, ppl

torch.manual_seed(0)
model = VQVAE(ema=True)
x = torch.rand(8, 3, 64, 64)
x_hat, loss, idx, ppl = model(x)
loss.backward()
print(x_hat.shape, idx.shape, idx.dtype)          # torch.Size([8, 3, 64, 64]) torch.Size([8, 16, 16]) torch.int64
print(model.enc[0].weight.grad is not None)        # True: the reconstruction gradient reaches the encoder
print(1 <= ppl.item() <= 512)                      # True: perplexity is between 1 and K
```

Vector quantiser with straight-through gradient, codebook + commitment loss or EMA updates, and perplexity

> **Forgetting the straight-through line:** Without `z_q = z + (z_q - z).detach()` the decoder receives `e[idx]`, and the reconstruction gradient flows into the selected codebook rows instead of the encoder. The encoder then learns only from the commitment term, which merely pulls it toward whatever codes it already uses. Training still runs and the loss still falls, but the encoder never learns to encode. Check that `enc.weight.grad` is non-zero and changes when you change the decoder loss.

> **Watch perplexity, not just reconstruction:** A model using 40 of 512 codes can reconstruct reasonably and look healthy on loss curves, but its bottleneck is $\log_2 40\approx5.3$ bits per position, not 9. Log code usage or perplexity every few hundred steps. If it plateaus far below $K$, initialise from data, restart dead codes, or lower the code dimension. In the figure the corner init stalls at perplexity 4.80 out of 16, while restarts reach 13.40.

> **Why the latent is not ignored:** In a VAE the KL term rewards the encoder for carrying *less* information, and a strong decoder can make the latent useless. In a VQ-VAE the KL is the constant $\log K$, so there is no pressure toward an uninformative code. The bottleneck is a hard capacity limit, $\log_2K$ bits per position, not a soft penalty. With a deterministic decoder the code is the only route to a good reconstruction, and the paper's speech model uses its codes even with a powerful autoregressive WaveNet decoder.

**Strengths**

- Discrete codes give short sequences for transformer or PixelCNN priors: the basis of VQGAN, DALL·E and audio tokenizers
- No posterior collapse: the KL term is constant and capacity is a hard bit budget
- Simple to train: reconstruction + two squared errors, one hyperparameter $\beta$ that is not sensitive
- EMA codebooks are optimiser-independent and stable

**Limitations**

- Straight-through gradients are biased; quantisation error is not optimised directly
- Codebook collapse: dead codes waste capacity unless initialisation, restarts or normalised codes are used
- Two-stage training: the prior is fitted after the autoencoder, so the ELBO is never optimised jointly
- Blurry reconstructions with plain L2 loss; high fidelity needs perceptual or adversarial losses (VQGAN)

**Question:** Which tensors receive gradient from each VQ-VAE loss term?

- A. Reconstruction → encoder and decoder (via straight-through); codebook term → codes only; commitment term → encoder only
- B. All three terms → encoder, decoder and codes
- C. Reconstruction → codes; codebook term → encoder; commitment → decoder
- D. Reconstruction → decoder only; the encoder learns only from the commitment term

**Answer:** A. Reconstruction → encoder and decoder (via straight-through); codebook term → codes only; commitment term → encoder only — The stop-gradients decide it. $\|\mathrm{sg}[z_e]-e\|^2$ has no path to the encoder, and $\beta\|z_e-\mathrm{sg}[e]\|^2$ has none to the codebook. The straight-through copy $z_q=z_e+\mathrm{sg}[e-z_e]$ sends the reconstruction gradient to the encoder and not to the codebook.

**Question:** A codebook of K = 512 codes shows a usage perplexity of 32 after training. What does this mean?

- A. Exactly 32 codes are used, each equally often
- B. The code-usage distribution has the entropy of a uniform distribution over 32 codes, so the effective bottleneck is about 5 bits per position instead of 9
- C. Reconstruction error is 32 times too high
- D. The commitment weight β should be 32

**Answer:** B. The code-usage distribution has the entropy of a uniform distribution over 32 codes, so the effective bottleneck is about 5 bits per position instead of 9 — Perplexity is $\exp(H)$ of the usage distribution. $\log_2 32=5$ bits, against $\log_2512=9$ available. More than 32 codes may be in use, unevenly. Either way most of the codebook's capacity is wasted, which is codebook collapse.

**Key points**

- Quantise: $z_q=e_k$ with $k=\arg\min_j\|z_e-e_j\|^2$; the one-hot posterior makes the KL a constant $\log K$, and each position stores $\log_2K$ bits.
- Loss: $-\log p(x\mid z_q)+\|\mathrm{sg}[z_e]-e\|^2+\beta\|z_e-\mathrm{sg}[e]\|^2$ with $\beta=0.25$; gradients reach the encoder by the straight-through copy $z_q=z_e+\mathrm{sg}[e-z_e]$.
- EMA alternative: $N_k\leftarrow\gamma N_k+(1-\gamma)n_k$, $m_k\leftarrow\gamma m_k+(1-\gamma)\sum z_e$, $e_k=m_k/N_k$, $\gamma=0.99$.
- Codebook collapse: unused codes get no update; monitor perplexity and use data init, dead-code restarts or normalised low-dimensional codes.
- Generation is two-stage: fit an autoregressive prior (PixelCNN, WaveNet, transformer) to the code grid, sample indices, decode; VQ-VAE-2 adds a 32×32 + 64×64 hierarchy.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
