Variational autoencoder (VAE)

A latent-variable generative model trained by maximizing the ELBO: the encoder outputs a Gaussian over codes, and the reparameterization trick makes sampling differentiable.

12 min read1143 wordscore0 viewsupdated 2026-10-11

Definition

Variational autoencoder (VAE) is a generative model pθ(x)=∫pθ(x∣z) p(z) dzp_\theta(x)=\int p_\theta(x\mid z)\,p(z)\,dz with a simple prior p(z)=N(0,I)p(z)=\mathcal N(0,I) and a neural decoder pθ(x∣z)p_\theta(x\mid z). It is trained jointly with a neural encoder qϕ(z∣x)=N(μϕ(x),diag⁡σϕ2(x))q_\phi(z\mid x)=\mathcal N(\mu_\phi(x),\operatorname{diag}\sigma^2_\phi(x)) by maximizing the evidence lower bound (ELBO) on log⁡pθ(x)\log p_\theta(x). Introduced by Kingma & Welling (2013) and, independently, Rezende, Mohamed & Wierstra (2014).

A plain autoencoder (see the Autoencoder article) compresses well, but its code space is not a distribution you can sample from. Codes of training points land wherever the encoder puts them, and a random point between them usually decodes to garbage. A VAE fixes this by making the codes probabilistic and pulling every code distribution toward the same prior. The space of zz then gets filled with no holes, and decoding z∼N(0,I)z\sim\mathcal N(0,I) produces new data.

The latent-variable model and its problem

Generate xx in two steps: draw a code z∼p(z)z\sim p(z), then draw x∼pθ(x∣z)x\sim p_\theta(x\mid z), for example N(gθ(z),σx2I)\mathcal N(g_\theta(z),\sigma_x^2I) with a decoder network gθg_\theta. Maximum likelihood wants log⁡pθ(x)=log⁡∫pθ(x∣z)p(z) dz\log p_\theta(x)=\log\int p_\theta(x\mid z)p(z)\,dz. That integral is intractable for a neural decoder, and so is the posterior pθ(z∣x)p_\theta(z\mid x). Sampling zz from the prior to estimate it is hopeless in high dimensions, because almost no prior samples explain a given xx.

The fix is amortized variational inference. Train an encoder qϕ(z∣x)q_\phi(z\mid x) to propose the codes that could have produced xx, and optimize a bound that only needs samples from qq.

Deriving the ELBO

Multiply and divide by qϕ(z∣x)q_\phi(z\mid x) inside the integral, then apply Jensen's inequality (log is concave):

log⁡pθ(x)=log⁡Eqϕ(z∣x) ⁣[pθ(x∣z)p(z)qϕ(z∣x)]  ≥  Eqϕ(z∣x) ⁣[log⁡pθ(x∣z)p(z)qϕ(z∣x)]≡ELBO\log p_\theta(x)=\log\mathbb{E}_{q_\phi(z|x)}\!\left[\frac{p_\theta(x\mid z)p(z)}{q_\phi(z\mid x)}\right]\;\ge\;\mathbb{E}_{q_\phi(z|x)}\!\left[\log\frac{p_\theta(x\mid z)p(z)}{q_\phi(z\mid x)}\right]\equiv\mathrm{ELBO}
qϕ(z∣x)q_\phi(z\mid x)
encoder's approximate posterior, any distribution we can sample from and evaluate
≥\ge
Jensen: log⁡E[Y]≥E[log⁡Y]\log\mathbb{E}[Y]\ge\mathbb{E}[\log Y]

The slack in the inequality has an exact form. Write pθ(x∣z)p(z)=pθ(z∣x) pθ(x)p_\theta(x\mid z)p(z)=p_\theta(z\mid x)\,p_\theta(x) inside the ELBO and collect terms:

log⁡pθ(x)=ELBO(x)+KL(qϕ(z∣x) ∥ pθ(z∣x))\log p_\theta(x)=\mathrm{ELBO}(x)+\mathrm{KL}\big(q_\phi(z\mid x)\,\big\|\,p_\theta(z\mid x)\big)
KL(q∥p(z∣x))≥0\mathrm{KL}(q\|p(z|x))\ge0
how far the encoder is from the true posterior; this is the gap

So maximizing the ELBO over ϕ\phi pulls qq toward the true posterior, and maximizing it over θ\theta raises a lower bound on the likelihood. Regrouping the same expression gives the form actually implemented:

ELBO(x)=Eqϕ(z∣x)[log⁡pθ(x∣z)]⏟reconstruction−KL(qϕ(z∣x) ∥ p(z))⏟regularizer\mathrm{ELBO}(x)=\underbrace{\mathbb{E}_{q_\phi(z|x)}\big[\log p_\theta(x\mid z)\big]}_{\text{reconstruction}}-\underbrace{\mathrm{KL}\big(q_\phi(z\mid x)\,\|\,p(z)\big)}_{\text{regularizer}}
log⁡pθ(x∣z)\log p_\theta(x|z)
for a Gaussian decoder: −∥x−gθ(z)∥2/2σx2-\|x-g_\theta(z)\|^2/2\sigma_x^2 + const, a scaled squared error; for binary data: −-BCE
KL(q∥p(z))\mathrm{KL}(q\|p(z))
keeps each code distribution close to the prior, so the prior is a valid sampler

For a diagonal Gaussian q=N(μ,diag⁡σ2)q=\mathcal N(\mu,\operatorname{diag}\sigma^2) and the prior N(0,I)\mathcal N(0,I) the regularizer is closed-form, so it needs no sampling:

KL(N(μ,diag⁡σ2) ∥ N(0,I))=12∑k=1dz(μk2+σk2−1−log⁡σk2)\mathrm{KL}\big(\mathcal N(\mu,\operatorname{diag}\sigma^2)\,\|\,\mathcal N(0,I)\big)=\frac12\sum_{k=1}^{d_z}\Big(\mu_k^2+\sigma_k^2-1-\log\sigma_k^2\Big)
μk2\mu_k^2
penalizes codes far from the origin
σk2−1−log⁡σk2\sigma_k^2-1-\log\sigma_k^2
≥0\ge0, zero only at σk=1\sigma_k=1: penalizes posteriors much narrower (or wider) than the prior

The reparameterization trick

The reconstruction term is an expectation over z∼qϕ(z∣x)z\sim q_\phi(z\mid x), and we need its gradient with respect to ϕ\phi. We cannot backpropagate through "draw a random sample": the sample is not a differentiable function of μ\mu and σ\sigma. The score-function (REINFORCE) estimator works in principle but has high variance. Kingma & Welling instead move the randomness into an input:

z=μϕ(x)+σϕ(x)⊙ε,ε∼N(0,I)z=\mu_\phi(x)+\sigma_\phi(x)\odot\varepsilon,\qquad \varepsilon\sim\mathcal N(0,I)
ε\varepsilon
parameter-free noise, sampled outside the network
∂z/∂μ=1, ∂z/∂σ=ε\partial z/\partial\mu=1,\ \partial z/\partial\sigma=\varepsilon
ordinary backprop now reaches the encoder

Now ∇ϕ Eε[log⁡pθ(x∣μ+σε)]=Eε[∇ϕlog⁡pθ(x∣μ+σε)]\nabla_\phi\,\mathbb{E}_\varepsilon[\log p_\theta(x\mid\mu+\sigma\varepsilon)]=\mathbb{E}_\varepsilon[\nabla_\phi\log p_\theta(x\mid\mu+\sigma\varepsilon)], and one sample of ε\varepsilon per example gives a low-variance, unbiased gradient. Encoders output log⁡σ2\log\sigma^2 rather than σ\sigma, so that the variance is always positive and the parameterization is numerically stable.

latent z · q(z|x) = N(μ, diag σ²), 1σdashed: prior 1σ, 2σ · ● z = μ + σ⊙ε (chosen x)data x · decoder mean g(z)grey: z-grid · ○ g(z ~ prior) · ● g(μ + σ⊙ε)chosen x = (−1.22, 0.79) → μ = (0.06, −0.76), σ = (0.995, 0.969)KL = ½Σ(μ² + σ² − 1 − log σ²) = 0.002 + 0.287 = 0.289 nats
Adam step 0/3000
−log p(x|z) 23.986KL per dim 0.126, 0.229ELBO −24.341 natsσₓ = 0.2 · 200 points
fig.A real VAE (2-D data, 2-D latent, 24-unit tanh encoder and decoder, Gaussian decoder with σₓ = 0.2) trained live for 3000 Adam steps. Left: each ellipse is q(z|x) at 1σ, against the dashed prior. White dots are reparameterized samples z = μ + σ⊙ε for the chosen point, and their decodings appear on the right. Rose rings decode 150 draws from the prior. At β = 1 the four clusters get separate regions of the prior, but the gaps between them decode to points between the clusters. At β = 0.1 the posteriors shrink to σ ≈ 0.13 and drift apart. At β = 20 they swell toward the prior, and at β = 30 every posterior is N(0, I) and every sample decodes to the data mean.

Worked example

Encoder output μ = (0.5, −1), σ = (0.5, 0.2)

  1. 1Dimension 1: 12(0.25+0.25−1−log⁡0.25)=12(−0.5+1.3863)=0.4431\tfrac12(0.25+0.25-1-\log0.25)=\tfrac12(-0.5+1.3863)=0.4431 nats.
  2. 2Dimension 2: 12(1+0.04−1−log⁡0.04)=12(0.04+3.2189)=1.6294\tfrac12(1+0.04-1-\log0.04)=\tfrac12(0.04+3.2189)=1.6294 nats. The narrow σ2=0.2\sigma_2=0.2 costs far more than the offset μ2=−1\mu_2=-1 alone would (0.50.5).
  3. 3Total KL(q∥p(z))=2.0726\mathrm{KL}(q\|p(z))=2.0726 nats: the information this code carries about xx, measured against the prior.
  4. 4Reparameterized sample with ε=(1.2, −0.4)\varepsilon=(1.2,\,-0.4): z=(0.5+0.5⋅1.2,  −1+0.2⋅(−0.4))=(1.1, −1.08)z=(0.5+0.5\cdot1.2,\;-1+0.2\cdot(-0.4))=(1.1,\,-1.08). Decode zz, score log⁡pθ(x∣z)\log p_\theta(x\mid z), and backprop flows to μ\mu (factor 1) and to σ\sigma (factor ε\varepsilon).
β−E log p(x|z)KL(q‖p(z))ELBOlog p(x) (IS)mean σ of qPrior samples near data
0.1−1.2514.185−2.934−2.0240.1346%
0.3−1.0222.948−1.925−1.3910.2360%
1−0.3882.214−1.827−1.1010.3472%
30.7811.676−2.457−1.0460.4372%
104.4601.126−5.586−1.3550.5854%
2017.4250.306−17.731−4.5080.863%
3023.6290.002−23.631−22.0041.000%
The figure's VAE after 3000 steps at each β (all numbers in nats per point, averaged over the 200 points). log p(x) is an importance-sampling estimate with 500 samples from q. "Prior samples near data" = share of 500 decoded prior draws within 0.4 of a cluster centre.

β = 1 gives the best ELBO, as expected, since that is the objective it optimizes. The gap to the importance-sampled log⁡p(x)\log p(x) is about 0.730.73 nats, an estimate of the average KL(q∥p(z∣x))\mathrm{KL}(q\|p(z|x)) (a slight underestimate, because the importance-sampled value is itself a lower bound). Note that β = 3 has the higher log⁡p(x)\log p(x): a looser bound does not always mean a worse model. Small β buys sharp reconstructions with codes the prior does not cover, so fewer prior samples look like data. Large β makes qq match the prior, until at β = 30 the code carries no information (KL ≈ 0) and the decoder outputs the data mean for every zz. That is posterior collapse.

The collapse point can be predicted. Dividing the β-weighted loss by β shows that a β-VAE with decoder variance σx2\sigma_x^2 is an ordinary VAE with variance βσx2\beta\sigma_x^2. For a linear-Gaussian model a latent direction is used only if the data variance along it exceeds that noise level (Lucas et al., 2019). Here the variances are about 1.01 and 0.98, and σx2=0.04\sigma_x^2=0.04, which predicts collapse near β≈25\beta\approx25. The trained VAE's KL is 0.306 at β = 20 and 0.005 at β = 25.

Training step

  1. 1Encode a batch: (μ,log⁡σ2)=encϕ(x)(\mu,\log\sigma^2)=\mathrm{enc}_\phi(x).
  2. 2Sample ε∼N(0,I)\varepsilon\sim\mathcal N(0,I) and set z=μ+e12log⁡σ2⊙εz=\mu+e^{\frac12\log\sigma^2}\odot\varepsilon (one sample per example is standard).
  3. 3Decode zz and compute the reconstruction term: −log⁡pθ(x∣z)-\log p_\theta(x\mid z), e.g. squared error /2σx2/2\sigma_x^2 or BCE, summed over the dimensions of xx.
  4. 4Add β⋅12∑k(μk2+σk2−1−log⁡σk2)\beta\cdot\frac12\sum_k(\mu_k^2+\sigma_k^2-1-\log\sigma_k^2), with β=1\beta=1 for the true ELBO.
  5. 5Backpropagate through decoder, sample and encoder; update θ\theta and ϕ\phi together.
  6. 6To generate: z∼N(0,I)z\sim\mathcal N(0,I), output gθ(z)g_\theta(z) (usually the mean rather than a noisy sample).
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class VAE(nn.Module):
4 def __init__(self, d=784, h=400, k=20):
5 super().__init__()
6 self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, 2 * k)) # -> (mu, logvar)
7 self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d)) # -> logits of p(x|z)
8
9 def forward(self, x):
10 mu, logvar = self.enc(x).chunk(2, dim=-1)
11 z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu) # reparameterization
12 return self.dec(z), mu, logvar
13
14def neg_elbo(model, x, beta=1.0):
15 logits, mu, logvar = model(x)
16 recon = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / len(x) # -E log p(x|z)
17 kl = 0.5 * (mu.pow(2) + logvar.exp() - 1 - logvar).sum(-1).mean() # KL(q || N(0, I))
18 return recon + beta * kl
19
20@torch.no_grad()
21def sample(model, n, k=20):
22 return torch.sigmoid(model.dec(torch.randn(n, k))) # decode prior draws
A VAE in PyTorch: closed-form KL and the reparameterization trick

β-VAE, posterior collapse and blur

β-VAE (Higgins et al., 2017) uses β>1\beta>1 on the KL term. The stronger pull toward a factorized prior tends to give disentangled latents, with one factor of variation (position, size, rotation) per dimension, at the cost of reconstruction quality. Later work showed that disentanglement is not guaranteed without inductive biases or supervision (Locatello et al., 2019).

Posterior collapse happens when the decoder can model xx without zz. A powerful autoregressive decoder (an LSTM over text, as in Bowman et al., 2016) can do this, and so can large β, or a code dimension that buys too little reconstruction. Then q(z∣x)→p(z)q(z\mid x)\to p(z), the KL term goes to 0, and the latent is ignored. Common remedies: KL annealing (ramp β from 0 to 1), free bits (no penalty below a minimum KL per dimension, Kingma et al. 2016), and weaker decoders.

Blurry samples. A Gaussian decoder with fixed variance is squared error, and squared error is minimized by the average of the plausible outputs. When the code leaves uncertainty (and the KL term ensures it does), the decoder averages, which shows up as blur. The figure shows the low-dimensional version: prior samples that fall between posteriors decode to points between the clusters. GANs, autoregressive decoders, VQ-VAE and diffusion trade this away in different ways. Latent diffusion (Stable Diffusion) keeps a KL-regularized autoencoder only for compression, and the diffusion model in its latent space does the generating.

Use when

  • +A real generative model with a tractable training objective, a likelihood bound you can compare, and a fast encoder for inference.
  • +Smooth, sampleable latent space: interpolation and arithmetic on codes give plausible outputs.
  • +Stable to train (single objective, no adversary), and the building block of latent-diffusion pipelines.

Avoid when

  • −Samples and reconstructions are blurrier than GANs or diffusion with simple Gaussian or Bernoulli decoders.
  • −Posterior collapse with powerful decoders; β, annealing and free bits need tuning.
  • −The diagonal-Gaussian posterior is restrictive: the ELBO gap KL(q‖p(z|x)) can be large (normalizing flows and IWAE tighten it).

Where it's used

Latent diffusion

The KL-regularized autoencoder of Stable Diffusion compresses images 8× per side, so the diffusion model runs on small latents.

Representation and disentanglement

β-VAE and variants learn factorized codes for controllable generation and analysis.

Anomaly detection

Low ELBO, or high reconstruction error under the decoder, flags inputs unlike the training data.

Science

Molecule design in a continuous latent space (Gómez-Bombarelli et al., 2018), single-cell genomics (scVI).

Check yourself

Why is the reparameterization z = μ + σ ⊙ ε needed?

Check yourself

An encoder outputs μ=(0,0)\mu=(0,0), σ=(1,1)\sigma=(1,1) for every input. What is the KL term, and what does it mean?

Remember

  • ✓Model pθ(x)=∫pθ(x∣z) N(z;0,I) dzp_\theta(x)=\int p_\theta(x|z)\,\mathcal N(z;0,I)\,dz; encoder qϕ(z∣x)=N(μ,diag⁡σ2)q_\phi(z|x)=\mathcal N(\mu,\operatorname{diag}\sigma^2).
  • ✓log⁡p(x)=ELBO+KL(q∥p(z∣x))\log p(x)=\mathrm{ELBO}+\mathrm{KL}(q\|p(z|x)), and ELBO=Eqlog⁡p(x∣z)−KL(q∥p(z))\mathrm{ELBO}=\mathbb{E}_q\log p(x|z)-\mathrm{KL}(q\|p(z)).
  • ✓Closed form: KL=12∑k(μk2+σk2−1−log⁡σk2)\mathrm{KL}=\tfrac12\sum_k(\mu_k^2+\sigma_k^2-1-\log\sigma_k^2).
  • ✓Reparameterization z=μ+σ⊙εz=\mu+\sigma\odot\varepsilon makes sampling differentiable (Kingma & Welling, 2013).
  • ✓β trades reconstruction against matching the prior: too large gives posterior collapse; Gaussian decoders average, so samples blur.
← back to Deep Learning roadmap