# Variational autoencoder (VAE)

> A latent-variable generative model trained by maximizing the ELBO: the encoder outputs a Gaussian over codes, and the reparameterization trick makes sampling differentiable.

Source: https://www.mlfoundry.in/learn/variational-autoencoder-vae  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 12 min read · updated 2026-10-11

**Variational autoencoder (VAE):** is a generative model $p_\theta(x)=\int p_\theta(x\mid z)\,p(z)\,dz$ with a simple prior $p(z)=\mathcal N(0,I)$ and a neural **decoder** $p_\theta(x\mid z)$. It is trained jointly with a neural **encoder** $q_\phi(z\mid x)=\mathcal N(\mu_\phi(x),\operatorname{diag}\sigma^2_\phi(x))$ by maximizing the evidence lower bound (ELBO) on $\log p_\theta(x)$. Introduced by Kingma & Welling (2013) and, independently, Rezende, Mohamed & Wierstra (2014).

A plain autoencoder (see the Autoencoder article) compresses well, but its code space is not a distribution you can sample from. Codes of training points land wherever the encoder puts them, and a random point between them usually decodes to garbage. A VAE fixes this by making the codes **probabilistic** and pulling every code distribution toward the same prior. The space of $z$ then gets filled with no holes, and decoding $z\sim\mathcal N(0,I)$ produces new data.

## The latent-variable model and its problem

Generate $x$ in two steps: draw a code $z\sim p(z)$, then draw $x\sim p_\theta(x\mid z)$, for example $\mathcal N(g_\theta(z),\sigma_x^2I)$ with a decoder network $g_\theta$. Maximum likelihood wants $\log p_\theta(x)=\log\int p_\theta(x\mid z)p(z)\,dz$. That integral is intractable for a neural decoder, and so is the posterior $p_\theta(z\mid x)$. Sampling $z$ from the prior to estimate it is hopeless in high dimensions, because almost no prior samples explain a given $x$.

The fix is **amortized variational inference**. Train an encoder $q_\phi(z\mid x)$ to propose the codes that could have produced $x$, and optimize a bound that only needs samples from $q$.

## Deriving the ELBO

Multiply and divide by $q_\phi(z\mid x)$ inside the integral, then apply Jensen's inequality (log is concave):

$$
\log p_\theta(x)=\log\mathbb{E}_{q_\phi(z|x)}\!\left[\frac{p_\theta(x\mid z)p(z)}{q_\phi(z\mid x)}\right]\;\ge\;\mathbb{E}_{q_\phi(z|x)}\!\left[\log\frac{p_\theta(x\mid z)p(z)}{q_\phi(z\mid x)}\right]\equiv\mathrm{ELBO}
$$

- $q_\phi(z\mid x)$: encoder's approximate posterior, any distribution we can sample from and evaluate
- $\ge$: Jensen: $\log\mathbb{E}[Y]\ge\mathbb{E}[\log Y]$

The slack in the inequality has an exact form. Write $p_\theta(x\mid z)p(z)=p_\theta(z\mid x)\,p_\theta(x)$ inside the ELBO and collect terms:

$$
\log p_\theta(x)=\mathrm{ELBO}(x)+\mathrm{KL}\big(q_\phi(z\mid x)\,\big\|\,p_\theta(z\mid x)\big)
$$

- $\mathrm{KL}(q\|p(z|x))\ge0$: how far the encoder is from the true posterior; this is the gap

So maximizing the ELBO over $\phi$ pulls $q$ toward the true posterior, and maximizing it over $\theta$ raises a lower bound on the likelihood. Regrouping the same expression gives the form actually implemented:

$$
\mathrm{ELBO}(x)=\underbrace{\mathbb{E}_{q_\phi(z|x)}\big[\log p_\theta(x\mid z)\big]}_{\text{reconstruction}}-\underbrace{\mathrm{KL}\big(q_\phi(z\mid x)\,\|\,p(z)\big)}_{\text{regularizer}}
$$

- $\log p_\theta(x|z)$: for a Gaussian decoder: $-\|x-g_\theta(z)\|^2/2\sigma_x^2$ + const, a scaled squared error; for binary data: $-$BCE
- $\mathrm{KL}(q\|p(z))$: keeps each code distribution close to the prior, so the prior is a valid sampler

For a diagonal Gaussian $q=\mathcal N(\mu,\operatorname{diag}\sigma^2)$ and the prior $\mathcal N(0,I)$ the regularizer is closed-form, so it needs no sampling:

$$
\mathrm{KL}\big(\mathcal N(\mu,\operatorname{diag}\sigma^2)\,\|\,\mathcal N(0,I)\big)=\frac12\sum_{k=1}^{d_z}\Big(\mu_k^2+\sigma_k^2-1-\log\sigma_k^2\Big)
$$

- $\mu_k^2$: penalizes codes far from the origin
- $\sigma_k^2-1-\log\sigma_k^2$: $\ge0$, zero only at $\sigma_k=1$: penalizes posteriors much narrower (or wider) than the prior

## The reparameterization trick

The reconstruction term is an expectation over $z\sim q_\phi(z\mid x)$, and we need its gradient with respect to $\phi$. We cannot backpropagate through "draw a random sample": the sample is not a differentiable function of $\mu$ and $\sigma$. The score-function (REINFORCE) estimator works in principle but has high variance. Kingma & Welling instead **move the randomness into an input**:

$$
z=\mu_\phi(x)+\sigma_\phi(x)\odot\varepsilon,\qquad \varepsilon\sim\mathcal N(0,I)
$$

- $\varepsilon$: parameter-free noise, sampled outside the network
- $\partial z/\partial\mu=1,\ \partial z/\partial\sigma=\varepsilon$: ordinary backprop now reaches the encoder

Now $\nabla_\phi\,\mathbb{E}_\varepsilon[\log p_\theta(x\mid\mu+\sigma\varepsilon)]=\mathbb{E}_\varepsilon[\nabla_\phi\log p_\theta(x\mid\mu+\sigma\varepsilon)]$, and one sample of $\varepsilon$ per example gives a low-variance, unbiased gradient. Encoders output $\log\sigma^2$ rather than $\sigma$, so that the variance is always positive and the parameterization is numerically stable.

> Figure: A real VAE (2-D data, 2-D latent, 24-unit tanh encoder and decoder, Gaussian decoder with σₓ = 0.2) trained live for 3000 Adam steps. Left: each ellipse is q(z|x) at 1σ, against the dashed prior. White dots are reparameterized samples z = μ + σ⊙ε for the chosen point, and their decodings appear on the right. Rose rings decode 150 draws from the prior. At β = 1 the four clusters get separate regions of the prior, but the gaps between them decode to points between the clusters. At β = 0.1 the posteriors shrink to σ ≈ 0.13 and drift apart. At β = 20 they swell toward the prior, and at β = 30 every posterior is N(0, I) and every sample decodes to the data mean.

## Worked example

**Encoder output μ = (0.5, −1), σ = (0.5, 0.2)**

1. Dimension 1: $\tfrac12(0.25+0.25-1-\log0.25)=\tfrac12(-0.5+1.3863)=0.4431$ nats.
2. Dimension 2: $\tfrac12(1+0.04-1-\log0.04)=\tfrac12(0.04+3.2189)=1.6294$ nats. The narrow $\sigma_2=0.2$ costs far more than the offset $\mu_2=-1$ alone would ($0.5$).
3. Total $\mathrm{KL}(q\|p(z))=2.0726$ nats: the information this code carries about $x$, measured against the prior.
4. Reparameterized sample with $\varepsilon=(1.2,\,-0.4)$: $z=(0.5+0.5\cdot1.2,\;-1+0.2\cdot(-0.4))=(1.1,\,-1.08)$. Decode $z$, score $\log p_\theta(x\mid z)$, and backprop flows to $\mu$ (factor 1) and to $\sigma$ (factor $\varepsilon$).

The figure's VAE after 3000 steps at each β (all numbers in nats per point, averaged over the 200 points). log p(x) is an importance-sampling estimate with 500 samples from q. "Prior samples near data" = share of 500 decoded prior draws within 0.4 of a cluster centre.

| β | −E log p(x\|z) | KL(q‖p(z)) | ELBO | log p(x) (IS) | mean σ of q | Prior samples near data |
| --- | --- | --- | --- | --- | --- | --- |
| 0.1 | −1.251 | 4.185 | −2.934 | −2.024 | 0.13 | 46% |
| 0.3 | −1.022 | 2.948 | −1.925 | −1.391 | 0.23 | 60% |
| 1 | −0.388 | 2.214 | −1.827 | −1.101 | 0.34 | 72% |
| 3 | 0.781 | 1.676 | −2.457 | −1.046 | 0.43 | 72% |
| 10 | 4.460 | 1.126 | −5.586 | −1.355 | 0.58 | 54% |
| 20 | 17.425 | 0.306 | −17.731 | −4.508 | 0.86 | 3% |
| 30 | 23.629 | 0.002 | −23.631 | −22.004 | 1.00 | 0% |

β = 1 gives the best ELBO, as expected, since that is the objective it optimizes. The gap to the importance-sampled $\log p(x)$ is about $0.73$ nats, an estimate of the average $\mathrm{KL}(q\|p(z|x))$ (a slight underestimate, because the importance-sampled value is itself a lower bound). Note that β = 3 has the higher $\log p(x)$: a looser bound does not always mean a worse model. Small β buys sharp reconstructions with codes the prior does not cover, so fewer prior samples look like data. Large β makes $q$ match the prior, until at β = 30 the code carries no information (KL ≈ 0) and the decoder outputs the data mean for every $z$. That is **posterior collapse**.

The collapse point can be predicted. Dividing the β-weighted loss by β shows that a β-VAE with decoder variance $\sigma_x^2$ is an ordinary VAE with variance $\beta\sigma_x^2$. For a linear-Gaussian model a latent direction is used only if the data variance along it exceeds that noise level (Lucas et al., 2019). Here the variances are about 1.01 and 0.98, and $\sigma_x^2=0.04$, which predicts collapse near $\beta\approx25$. The trained VAE's KL is 0.306 at β = 20 and 0.005 at β = 25.

**Training step**

1. Encode a batch: $(\mu,\log\sigma^2)=\mathrm{enc}_\phi(x)$.
2. Sample $\varepsilon\sim\mathcal N(0,I)$ and set $z=\mu+e^{\frac12\log\sigma^2}\odot\varepsilon$ (one sample per example is standard).
3. Decode $z$ and compute the reconstruction term: $-\log p_\theta(x\mid z)$, e.g. squared error $/2\sigma_x^2$ or BCE, summed over the dimensions of $x$.
4. Add $\beta\cdot\frac12\sum_k(\mu_k^2+\sigma_k^2-1-\log\sigma_k^2)$, with $\beta=1$ for the true ELBO.
5. Backpropagate through decoder, sample and encoder; update $\theta$ and $\phi$ together.
6. To generate: $z\sim\mathcal N(0,I)$, output $g_\theta(z)$ (usually the mean rather than a noisy sample).

```python
import torch, torch.nn as nn, torch.nn.functional as F

class VAE(nn.Module):
    def __init__(self, d=784, h=400, k=20):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, 2 * k))   # -> (mu, logvar)
        self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d))       # -> logits of p(x|z)

    def forward(self, x):
        mu, logvar = self.enc(x).chunk(2, dim=-1)
        z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu)    # reparameterization
        return self.dec(z), mu, logvar

def neg_elbo(model, x, beta=1.0):
    logits, mu, logvar = model(x)
    recon = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / len(x)  # -E log p(x|z)
    kl = 0.5 * (mu.pow(2) + logvar.exp() - 1 - logvar).sum(-1).mean()               # KL(q || N(0, I))
    return recon + beta * kl

@torch.no_grad()
def sample(model, n, k=20):
    return torch.sigmoid(model.dec(torch.randn(n, k)))           # decode prior draws
```

A VAE in PyTorch: closed-form KL and the reparameterization trick

## β-VAE, posterior collapse and blur

**β-VAE** (Higgins et al., 2017) uses $\beta>1$ on the KL term. The stronger pull toward a factorized prior tends to give **disentangled** latents, with one factor of variation (position, size, rotation) per dimension, at the cost of reconstruction quality. Later work showed that disentanglement is not guaranteed without inductive biases or supervision (Locatello et al., 2019).

**Posterior collapse** happens when the decoder can model $x$ without $z$. A powerful autoregressive decoder (an LSTM over text, as in Bowman et al., 2016) can do this, and so can large β, or a code dimension that buys too little reconstruction. Then $q(z\mid x)\to p(z)$, the KL term goes to 0, and the latent is ignored. Common remedies: **KL annealing** (ramp β from 0 to 1), **free bits** (no penalty below a minimum KL per dimension, Kingma et al. 2016), and weaker decoders.

**Blurry samples.** A Gaussian decoder with fixed variance is squared error, and squared error is minimized by the **average** of the plausible outputs. When the code leaves uncertainty (and the KL term ensures it does), the decoder averages, which shows up as blur. The figure shows the low-dimensional version: prior samples that fall between posteriors decode to points between the clusters. GANs, autoregressive decoders, VQ-VAE and diffusion trade this away in different ways. Latent diffusion (Stable Diffusion) keeps a KL-regularized autoencoder only for compression, and the diffusion model in its latent space does the generating.

> **Sum, don't average, the reconstruction:** $\log p(x\mid z)$ is a sum over all pixels. Using `F.mse_loss` with its default mean over 784 pixels quietly divides the reconstruction term by 784. That is equivalent to a β of 784 on the KL term, and it is a classic cause of collapsed, blurry VAEs. Sum over data dimensions and average over the batch, or set β deliberately.

> **The KL term is a code length:** $\mathrm{KL}(q(z|x)\|p(z))$ is the expected number of nats needed to tell the decoder which $z$ was drawn, beyond what the prior already says. The ELBO is then (minus) a two-part description length: bits for the code plus bits for $x$ given the code. β sets the exchange rate between them, which is why β-VAE is often analysed as a rate–distortion trade-off.

**Strengths**

- A real generative model with a tractable training objective, a likelihood bound you can compare, and a fast encoder for inference.
- Smooth, sampleable latent space: interpolation and arithmetic on codes give plausible outputs.
- Stable to train (single objective, no adversary), and the building block of latent-diffusion pipelines.

**Limitations**

- Samples and reconstructions are blurrier than GANs or diffusion with simple Gaussian or Bernoulli decoders.
- Posterior collapse with powerful decoders; β, annealing and free bits need tuning.
- The diagonal-Gaussian posterior is restrictive: the ELBO gap KL(q‖p(z|x)) can be large (normalizing flows and IWAE tighten it).

- **Latent diffusion:** The KL-regularized autoencoder of Stable Diffusion compresses images 8× per side, so the diffusion model runs on small latents.
- **Representation and disentanglement:** β-VAE and variants learn factorized codes for controllable generation and analysis.
- **Anomaly detection:** Low ELBO, or high reconstruction error under the decoder, flags inputs unlike the training data.
- **Science:** Molecule design in a continuous latent space (Gómez-Bombarelli et al., 2018), single-cell genomics (scVI).

**Question:** Why is the reparameterization z = μ + σ ⊙ ε needed?

- A. To make the sample a differentiable function of the encoder outputs, so backprop can reach the encoder
- B. To make the KL term computable in closed form
- C. To make z exactly equal to μ at test time
- D. To guarantee the posterior is Gaussian

**Answer:** A. To make the sample a differentiable function of the encoder outputs, so backprop can reach the encoder — Sampling $z\sim\mathcal N(\mu,\sigma^2)$ directly has no gradient with respect to $\mu,\sigma$. Writing $z=\mu+\sigma\varepsilon$ with parameter-free $\varepsilon$ gives $\partial z/\partial\mu=1$ and $\partial z/\partial\sigma=\varepsilon$. The KL is closed-form either way.

**Question:** An encoder outputs $\mu=(0,0)$, $\sigma=(1,1)$ for every input. What is the KL term, and what does it mean?

- A. 0 nats: every posterior equals the prior, so z carries no information about x (posterior collapse)
- B. 1 nat per dimension: the code is maximally informative
- C. Infinite, because σ = 1 is too wide
- D. It depends on the decoder

**Answer:** A. 0 nats: every posterior equals the prior, so z carries no information about x (posterior collapse) — $\tfrac12(0+1-1-\log1)=0$ per dimension. The decoder then sees $z\sim\mathcal N(0,I)$ regardless of $x$, so the best it can do is output the data mean. This is the β = 30 row of the table.

**Key points**

- Model $p_\theta(x)=\int p_\theta(x|z)\,\mathcal N(z;0,I)\,dz$; encoder $q_\phi(z|x)=\mathcal N(\mu,\operatorname{diag}\sigma^2)$.
- $\log p(x)=\mathrm{ELBO}+\mathrm{KL}(q\|p(z|x))$, and $\mathrm{ELBO}=\mathbb{E}_q\log p(x|z)-\mathrm{KL}(q\|p(z))$.
- Closed form: $\mathrm{KL}=\tfrac12\sum_k(\mu_k^2+\sigma_k^2-1-\log\sigma_k^2)$.
- Reparameterization $z=\mu+\sigma\odot\varepsilon$ makes sampling differentiable (Kingma & Welling, 2013).
- β trades reconstruction against matching the prior: too large gives posterior collapse; Gaussian decoders average, so samples blur.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
