Definition
Variational autoencoder (VAE) is a generative model with a simple prior and a neural decoder . It is trained jointly with a neural encoder by maximizing the evidence lower bound (ELBO) on . Introduced by Kingma & Welling (2013) and, independently, Rezende, Mohamed & Wierstra (2014).
A plain autoencoder (see the Autoencoder article) compresses well, but its code space is not a distribution you can sample from. Codes of training points land wherever the encoder puts them, and a random point between them usually decodes to garbage. A VAE fixes this by making the codes probabilistic and pulling every code distribution toward the same prior. The space of then gets filled with no holes, and decoding produces new data.
The latent-variable model and its problem
Generate in two steps: draw a code , then draw , for example with a decoder network . Maximum likelihood wants . That integral is intractable for a neural decoder, and so is the posterior . Sampling from the prior to estimate it is hopeless in high dimensions, because almost no prior samples explain a given .
The fix is amortized variational inference. Train an encoder to propose the codes that could have produced , and optimize a bound that only needs samples from .
Deriving the ELBO
Multiply and divide by inside the integral, then apply Jensen's inequality (log is concave):
- encoder's approximate posterior, any distribution we can sample from and evaluate
- Jensen:
The slack in the inequality has an exact form. Write inside the ELBO and collect terms:
- how far the encoder is from the true posterior; this is the gap
So maximizing the ELBO over pulls toward the true posterior, and maximizing it over raises a lower bound on the likelihood. Regrouping the same expression gives the form actually implemented:
- for a Gaussian decoder: + const, a scaled squared error; for binary data: BCE
- keeps each code distribution close to the prior, so the prior is a valid sampler
For a diagonal Gaussian and the prior the regularizer is closed-form, so it needs no sampling:
- penalizes codes far from the origin
- , zero only at : penalizes posteriors much narrower (or wider) than the prior
The reparameterization trick
The reconstruction term is an expectation over , and we need its gradient with respect to . We cannot backpropagate through "draw a random sample": the sample is not a differentiable function of and . The score-function (REINFORCE) estimator works in principle but has high variance. Kingma & Welling instead move the randomness into an input:
- parameter-free noise, sampled outside the network
- ordinary backprop now reaches the encoder
Now , and one sample of per example gives a low-variance, unbiased gradient. Encoders output rather than , so that the variance is always positive and the parameterization is numerically stable.
Worked example
Encoder output μ = (0.5, −1), σ = (0.5, 0.2)
- 1Dimension 1: nats.
- 2Dimension 2: nats. The narrow costs far more than the offset alone would ().
- 3Total nats: the information this code carries about , measured against the prior.
- 4Reparameterized sample with : . Decode , score , and backprop flows to (factor 1) and to (factor ).
| β | −E log p(x|z) | KL(q‖p(z)) | ELBO | log p(x) (IS) | mean σ of q | Prior samples near data |
|---|---|---|---|---|---|---|
| 0.1 | −1.251 | 4.185 | −2.934 | −2.024 | 0.13 | 46% |
| 0.3 | −1.022 | 2.948 | −1.925 | −1.391 | 0.23 | 60% |
| 1 | −0.388 | 2.214 | −1.827 | −1.101 | 0.34 | 72% |
| 3 | 0.781 | 1.676 | −2.457 | −1.046 | 0.43 | 72% |
| 10 | 4.460 | 1.126 | −5.586 | −1.355 | 0.58 | 54% |
| 20 | 17.425 | 0.306 | −17.731 | −4.508 | 0.86 | 3% |
| 30 | 23.629 | 0.002 | −23.631 | −22.004 | 1.00 | 0% |
β = 1 gives the best ELBO, as expected, since that is the objective it optimizes. The gap to the importance-sampled is about nats, an estimate of the average (a slight underestimate, because the importance-sampled value is itself a lower bound). Note that β = 3 has the higher : a looser bound does not always mean a worse model. Small β buys sharp reconstructions with codes the prior does not cover, so fewer prior samples look like data. Large β makes match the prior, until at β = 30 the code carries no information (KL ≈ 0) and the decoder outputs the data mean for every . That is posterior collapse.
The collapse point can be predicted. Dividing the β-weighted loss by β shows that a β-VAE with decoder variance is an ordinary VAE with variance . For a linear-Gaussian model a latent direction is used only if the data variance along it exceeds that noise level (Lucas et al., 2019). Here the variances are about 1.01 and 0.98, and , which predicts collapse near . The trained VAE's KL is 0.306 at β = 20 and 0.005 at β = 25.
Training step
- 1Encode a batch: .
- 2Sample and set (one sample per example is standard).
- 3Decode and compute the reconstruction term: , e.g. squared error or BCE, summed over the dimensions of .
- 4Add , with for the true ELBO.
- 5Backpropagate through decoder, sample and encoder; update and together.
- 6To generate: , output (usually the mean rather than a noisy sample).
1import torch, torch.nn as nn, torch.nn.functional as F23class VAE(nn.Module):4 def __init__(self, d=784, h=400, k=20):5 super().__init__()6 self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, 2 * k)) # -> (mu, logvar)7 self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d)) # -> logits of p(x|z)89 def forward(self, x):10 mu, logvar = self.enc(x).chunk(2, dim=-1)11 z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu) # reparameterization12 return self.dec(z), mu, logvar1314def neg_elbo(model, x, beta=1.0):15 logits, mu, logvar = model(x)16 recon = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / len(x) # -E log p(x|z)17 kl = 0.5 * (mu.pow(2) + logvar.exp() - 1 - logvar).sum(-1).mean() # KL(q || N(0, I))18 return recon + beta * kl1920@torch.no_grad()21def sample(model, n, k=20):22 return torch.sigmoid(model.dec(torch.randn(n, k))) # decode prior drawsβ-VAE, posterior collapse and blur
β-VAE (Higgins et al., 2017) uses on the KL term. The stronger pull toward a factorized prior tends to give disentangled latents, with one factor of variation (position, size, rotation) per dimension, at the cost of reconstruction quality. Later work showed that disentanglement is not guaranteed without inductive biases or supervision (Locatello et al., 2019).
Posterior collapse happens when the decoder can model without . A powerful autoregressive decoder (an LSTM over text, as in Bowman et al., 2016) can do this, and so can large β, or a code dimension that buys too little reconstruction. Then , the KL term goes to 0, and the latent is ignored. Common remedies: KL annealing (ramp β from 0 to 1), free bits (no penalty below a minimum KL per dimension, Kingma et al. 2016), and weaker decoders.
Blurry samples. A Gaussian decoder with fixed variance is squared error, and squared error is minimized by the average of the plausible outputs. When the code leaves uncertainty (and the KL term ensures it does), the decoder averages, which shows up as blur. The figure shows the low-dimensional version: prior samples that fall between posteriors decode to points between the clusters. GANs, autoregressive decoders, VQ-VAE and diffusion trade this away in different ways. Latent diffusion (Stable Diffusion) keeps a KL-regularized autoencoder only for compression, and the diffusion model in its latent space does the generating.
Use when
- +A real generative model with a tractable training objective, a likelihood bound you can compare, and a fast encoder for inference.
- +Smooth, sampleable latent space: interpolation and arithmetic on codes give plausible outputs.
- +Stable to train (single objective, no adversary), and the building block of latent-diffusion pipelines.
Avoid when
- −Samples and reconstructions are blurrier than GANs or diffusion with simple Gaussian or Bernoulli decoders.
- −Posterior collapse with powerful decoders; β, annealing and free bits need tuning.
- −The diagonal-Gaussian posterior is restrictive: the ELBO gap KL(q‖p(z|x)) can be large (normalizing flows and IWAE tighten it).
Where it's used
Latent diffusion
The KL-regularized autoencoder of Stable Diffusion compresses images 8× per side, so the diffusion model runs on small latents.
Representation and disentanglement
β-VAE and variants learn factorized codes for controllable generation and analysis.
Anomaly detection
Low ELBO, or high reconstruction error under the decoder, flags inputs unlike the training data.
Science
Molecule design in a continuous latent space (Gómez-Bombarelli et al., 2018), single-cell genomics (scVI).
Check yourself
Why is the reparameterization z = μ + σ ⊙ ε needed?
Check yourself
An encoder outputs , for every input. What is the KL term, and what does it mean?
Remember
- ✓Model ; encoder .
- ✓, and .
- ✓Closed form: .
- ✓Reparameterization makes sampling differentiable (Kingma & Welling, 2013).
- ✓β trades reconstruction against matching the prior: too large gives posterior collapse; Gaussian decoders average, so samples blur.