# Latent diffusion models

> A latent diffusion model (Rombach et al., CVPR 2022) first trains an autoencoder that compresses images by a factor f per side into a latent z = E(x), then trains a diffusion model on z, conditioned through cross-attention on τ_θ(y). Per-step compute falls by about f²; f = 4–8 keeps quality.

Source: https://www.mlfoundry.in/learn/latent-diffusion-models  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 14 min read · updated 2026-10-12

**Latent diffusion model (LDM):** A two-stage generative model from Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR 2022). Stage 1 is an autoencoder $\mathcal E,\mathcal D$ trained once, with perceptual and adversarial losses, that maps an image $x\in\mathbb R^{H\times W\times3}$ to a latent $z=\mathcal E(x)\in\mathbb R^{h\times w\times c}$ with $h=H/f$, $w=W/f$. Stage 2 is a diffusion model, usually a U-Net, trained to denoise $z$ instead of $x$. Conditioning (text, class, layout, a low-resolution image) enters through **cross-attention** from the U-Net's features to a learned encoding $\tau_\theta(y)$. Sampling runs the diffusion in latent space and decodes once: $\tilde x=\mathcal D(z_0)$.

Pixel-space diffusion models (see **Denoising diffusion probabilistic models (DDPM)**) reached top image quality by 2021, but at great cost. Every training and sampling step evaluates the network on the full pixel grid, and the paper notes that the strongest models took 150 to 1000 V100 days to train. Rombach et al. argued that most of those bits are **perceptually irrelevant detail**. A likelihood-based model spends much of its capacity on imperceptible high-frequency content, which they called *perceptual compression*, and only a small part on the *semantic compression* that matters. So they split the work. An autoencoder removes the imperceptible detail once (perceptual compression), and the diffusion model learns the semantic and conceptual composition in a space that is spatially smaller but keeps the 2-D layout. Earlier two-stage models such as VQGAN with a transformer (see **Vector-quantized VAE (VQ-VAE)** and **Autoregressive image generation**) needed aggressive compression to make an autoregressive prior affordable. A convolutional diffusion prior works with mild compression, so the latent can stay detailed.

## Stage 1: perceptual compression

$$
\mathcal L_{\mathrm{AE}}=\min_{\mathcal E,\mathcal D}\max_{\psi}\Big(\mathcal L_{\mathrm{rec}}\big(x,\mathcal D(\mathcal E(x))\big)-\mathcal L_{\mathrm{adv}}\big(\mathcal D(\mathcal E(x))\big)+\log D_\psi(x)+\mathcal L_{\mathrm{reg}}(x;\mathcal E,\mathcal D)\Big)
$$

- $\mathcal L_{\mathrm{rec}}$: pixel $\ell_1$ plus an LPIPS perceptual loss (see **Perceptual loss**), so the decoder keeps what looks right rather than what is pixel-exact
- $D_\psi$: a patch-based discriminator (see **Adversarial loss**); it forces realistic local texture and avoids the blur of $\ell_2$ reconstructions
- $\mathcal L_{\mathrm{reg}}$: **KL-reg**: a slight KL penalty pulling $q_{\mathcal E}(z\mid x)$ toward $\mathcal N(0,I)$, as in a VAE but with a tiny weight (see **Variational autoencoder (VAE)**); or **VQ-reg**: a vector-quantisation layer in the decoder, as in VQGAN
- $f=H/h=2^m$: the spatial downsampling factor; the paper trains $f\in\{1,2,4,8,16,32\}$ (LDM-$f$; LDM-1 is pixel space)

The regulariser only keeps the latent from having arbitrary scale and variance. It is not meant to make $z$ a sample from a prior, because the diffusion model is the prior. With KL-reg the KL weight is very small (about $10^{-6}$ in the released configs), so $z$ is close to a deterministic code. With VQ-reg the diffusion model works on the continuous pre-quantisation features and the quantiser is "absorbed" into the decoder. Because the latents of a KL-reg autoencoder are not unit-variance, LDM rescales them by one scalar, estimated from the standard deviation of the first training batch, before diffusion. Stable Diffusion's $0.18215$ is that scalar (see **Stable Diffusion**).

## Stage 2: diffusion in latent space

$$
\mathcal L_{\mathrm{LDM}}=\mathbb E_{\mathcal E(x),\,y,\,\epsilon\sim\mathcal N(0,I),\,t}\Big[\big\|\epsilon-\epsilon_\theta\big(z_t,\,t,\,\tau_\theta(y)\big)\big\|_2^2\Big],\qquad z_t=\sqrt{\bar\alpha_t}\,\mathcal E(x)+\sqrt{1-\bar\alpha_t}\,\epsilon
$$

- $\epsilon_\theta$: a time-conditional U-Net on $h\times w\times c$ latents: convolutions keep the 2-D inductive bias, unlike a 1-D token prior
- $\tau_\theta(y)$: a domain-specific encoder of the condition $y$, trained jointly with $\epsilon_\theta$: a transformer over BERT-tokenised text in the paper, CLIP's text encoder in Stable Diffusion
- $\mathcal E(x)$: computed with the frozen encoder; $\mathcal D$ is used only to decode samples

The objective is the ordinary noise-prediction loss (see **Diffusion noise-prediction loss**), so any diffusion sampler applies unchanged. The paper used DDIM (see **Denoising diffusion implicit models (DDIM)**) for most results. The U-Net is fully convolutional, so a model trained on $64\times64$ latents can be sampled on larger latent grids for larger images, as in the paper's $1024$-pixel-wide landscapes. Results are good but not always coherent at large sizes.

## Conditioning by cross-attention

$$
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt d}\Big)V,\qquad Q=W_Q^{(i)}\varphi_i(z_t),\quad K=W_K^{(i)}\tau_\theta(y),\quad V=W_V^{(i)}\tau_\theta(y)
$$

- $\varphi_i(z_t)\in\mathbb R^{N\times d_\epsilon^i}$: the U-Net's intermediate feature map at layer $i$, flattened so each of the $N=h_i w_i$ spatial positions is a query
- $\tau_\theta(y)\in\mathbb R^{M\times d_\tau}$: $M$ condition tokens, e.g. 77 text tokens; they supply keys and values
- $W^{(i)}$: per-layer projections; the block is self-attention, then cross-attention, then a feed-forward layer, each with a residual connection

Each image position reads from the condition tokens. The attention matrix is $N\times M$, not $N\times N$, so cross-attention is cheap next to the U-Net's self-attention at the same resolution (see **Cross-attention**). Spatially aligned conditions such as semantic maps, low-resolution images for super-resolution, or masked images for inpainting are simpler. They are **concatenated** to $z_t$ as extra input channels.

**Findings and results in the paper**

- **Choice of f:** LDM-1 and LDM-2 train slowly; LDM-32 compresses too much and caps quality. LDM-4 to LDM-16 balance efficiency and fidelity; after 2M training steps on class-conditional ImageNet the FID gap between pixel-space LDM-1 and LDM-8 was 38
- **Unconditional, 256²:** FID 5.11 on CelebA-HQ (then state of the art), 4.98 on FFHQ, 4.02 on LSUN-Churches, 2.95 on LSUN-Bedrooms
- **Class-conditional ImageNet 256²:** LDM-4-G with classifier-free guidance (scale 1.5): FID 3.60, Inception score 247.67, with fewer parameters and less compute than ADM
- **Text-to-image:** LDM-KL-8-G, 1.45B parameters, trained on LAION-400M with a transformer text encoder: zero-shot MS-COCO 256² FID 12.63 with guidance 1.5
- **Other tasks:** state of the art in inpainting; competitive $4\times$ super-resolution and semantic-map-to-image synthesis, all with the same recipe

> Figure: A 64×64 toy image passes through a fixed stand-in for stage 1, a block DCT that keeps the c lowest frequencies of each f×f block. The latent is (64/f)×(64/f)×c. With the Stable Diffusion shape (f = 8, c = 4) the latent is 8×8×4, 256 of 4096 values (16× fewer), and the reconstruction has PSNR 20.2 dB (MSE 0.00955). At c = 4 the PSNR falls with f: lossless at f = 1 and 2 (four coefficients cover a 2×2 block), then 22.5 dB at f = 4, 20.2 at f = 8, 18.4 at f = 16 and 17.4 at f = 32. With c = 16 it is 22.7 dB at f = 8 and 20.6 at f = 16. The right chart shows what the second stage gains. At 512×512 one full-resolution self-attention layer has 68.7G scores per head in pixel space (a 128 GiB fp16 matrix). At f = 4 it has 268M, at f = 8 16.8M (32 MiB), and at f = 16 1.0M. Convolution cost per step falls as 1/f² (1/64 at f = 8). A learned KL autoencoder with LPIPS and adversarial losses keeps far more detail than the DCT at the same latent shape. That is why f = 4–8 costs little perceived quality.

## Worked example: what f buys at 512×512

Latent size and one self-attention layer's cost at 512×512×3, c = 4 latent channels (c = 3 at f = 1); toy PSNR from the figure (computed in Node)

| f | latent | values | pixel values per latent value | self-attn scores / head | toy PSNR, c = 4 |
| --- | --- | --- | --- | --- | --- |
| 1 | 512×512×3 | 786,432 | 1 | 6.87 × 10¹⁰ | lossless |
| 2 | 256×256×4 | 262,144 | 3 | 4.29 × 10⁹ | lossless |
| 4 | 128×128×4 | 65,536 | 12 | 2.68 × 10⁸ | 22.5 dB |
| 8 | 64×64×4 | 16,384 | 48 | 1.68 × 10⁷ | 20.2 dB |
| 16 | 32×32×4 | 4,096 | 192 | 1.05 × 10⁶ | 18.4 dB |
| 32 | 16×16×4 | 1,024 | 768 | 6.55 × 10⁴ | 17.4 dB |

**Reading the f = 8 row**

1. $512/8=64$, so $z\in\mathbb R^{64\times64\times4}$: $16{,}384$ numbers against $786{,}432$, a factor $48=8^2\cdot3/4$.
2. Convolutional layers at a fixed width cost in proportion to positions: $64^2/512^2=1/64$ of pixel space per step.
3. Self-attention over all $4096$ latent positions: $4096^2=16{,}777{,}216$ scores per head, 32 MiB in fp16. Over $262{,}144$ pixels it would be $6.87\times10^{10}$ scores, $4096\times$ more. Pixel models therefore restrict attention to low resolutions.
4. Cross-attention to 77 text tokens at the same resolution: $4096\times77=315{,}392$ scores per head, 53 times fewer than the self-attention.
5. Training also gets cheaper: $\mathcal E(x)$ can be precomputed once per image, and the decoder runs only once per sample, not once per step.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

torch.manual_seed(0)

class Autoencoder(nn.Module):                         # first stage: f = 8, c = 4 (KL-reg, trained beforehand)
    def __init__(self, c=4, w=64):
        super().__init__()
        self.enc = nn.Sequential(nn.Conv2d(3, w, 3, 2, 1), nn.SiLU(), nn.Conv2d(w, w, 3, 2, 1), nn.SiLU(),
                                 nn.Conv2d(w, w, 3, 2, 1), nn.SiLU(), nn.Conv2d(w, 2 * c, 3, 1, 1))
        self.dec = nn.Sequential(nn.Conv2d(c, w, 3, 1, 1), nn.SiLU(), nn.Upsample(scale_factor=8), nn.Conv2d(w, 3, 3, 1, 1))
    def encode(self, x):                              # returns a sample of q(z | x) = N(mu, sigma^2)
        mu, logvar = self.enc(x).chunk(2, dim=1)
        return mu + (0.5 * logvar).exp() * torch.randn_like(mu)
    def decode(self, z):
        return self.dec(z)

class CrossAttention(nn.Module):                      # Q from the latent, K and V from tau_theta(y)
    def __init__(self, d, d_ctx, heads=4):
        super().__init__()
        self.h = heads
        self.q, self.k, self.v = nn.Linear(d, d, bias=False), nn.Linear(d_ctx, d, bias=False), nn.Linear(d_ctx, d, bias=False)
        self.o = nn.Linear(d, d)
    def forward(self, x, ctx):                        # x (B, N, d), ctx (B, M, d_ctx)
        B, N, d = x.shape
        split = lambda t: t.view(B, -1, self.h, d // self.h).transpose(1, 2)
        out = F.scaled_dot_product_attention(split(self.q(x)), split(self.k(ctx)), split(self.v(ctx)))
        return self.o(out.transpose(1, 2).reshape(B, N, d))

class Denoiser(nn.Module):                            # eps_theta(z_t, t, tau_theta(y)), a one-level "U-Net"
    def __init__(self, c=4, d=128, d_ctx=64):
        super().__init__()
        self.inp, self.out = nn.Conv2d(c, d, 3, 1, 1), nn.Conv2d(d, c, 3, 1, 1)
        self.temb = nn.Sequential(nn.Linear(d, d), nn.SiLU(), nn.Linear(d, d))
        self.res = nn.Sequential(nn.GroupNorm(8, d), nn.SiLU(), nn.Conv2d(d, d, 3, 1, 1))
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.self_attn = nn.MultiheadAttention(d, 4, batch_first=True)
        self.cross = CrossAttention(d, d_ctx)
    def forward(self, z, t, ctx):
        half = 64
        freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)
        temb = self.temb(torch.cat([torch.sin(t[:, None] * freqs), torch.cos(t[:, None] * freqs)], 1))
        h = self.inp(z) + temb[:, :, None, None]
        h = h + self.res(h)
        B, d, H, W = h.shape
        s = h.flatten(2).transpose(1, 2)              # (B, H*W, d): every latent position is a token
        s = s + self.self_attn(self.n1(s), self.n1(s), self.n1(s), need_weights=False)[0]
        s = s + self.cross(self.n2(s), ctx)
        return self.out(s.transpose(1, 2).view(B, d, H, W))

T = 1000
betas = torch.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T) ** 2
abar = torch.cumprod(1 - betas, 0)

ae, eps_model = Autoencoder(), Denoiser()
ae.requires_grad_(False)                              # stage 1 is frozen while the diffusion model trains
tau = nn.Embedding(1000, 64)                          # tau_theta: a stand-in text encoder (token embeddings)
opt = torch.optim.AdamW(list(eps_model.parameters()) + list(tau.parameters()), lr=1e-4)

x = torch.randn(2, 3, 256, 256)                       # images in [-1, 1] in practice
tokens = torch.randint(0, 1000, (2, 77))
with torch.no_grad():
    z0 = ae.encode(x)
scale = 1 / z0.std()                                  # SD v1 fixed this once: 0.18215
z0 = z0 * scale
t = torch.randint(0, T, (2,))
noise = torch.randn_like(z0)
a = abar[t].view(-1, 1, 1, 1)
z_t = a.sqrt() * z0 + (1 - a).sqrt() * noise
loss = F.mse_loss(eps_model(z_t, t.float(), tau(tokens)), noise)
opt.zero_grad(); loss.backward(); opt.step()
print(x.shape, "->", z0.shape)                        # torch.Size([2, 3, 256, 256]) -> torch.Size([2, 4, 32, 32])
print(x[0].numel() // z0[0].numel())                  # 48 pixel values per latent value
print(ae.decode(z0 / scale).shape)                    # torch.Size([2, 3, 256, 256])
```

One LDM training step: frozen f = 8 autoencoder, latent rescaling, and a denoiser with self- and cross-attention over latent positions (toy sizes; shapes checked with PyTorch 2.x on CPU)

> **Forgetting the latent scale factor:** The diffusion schedule assumes data of roughly unit variance. Raw KL-autoencoder latents are not (their standard deviation is about $1/0.18215\approx5.5$ for Stable Diffusion's VAE). If you train or sample without multiplying by the scale factor, or decode without dividing by it, the signal-to-noise ratio at every $t$ is wrong. Training loses most of its useful noise levels, and decoded samples are washed out or saturated. A new autoencoder needs its own factor.

> **The autoencoder sets the ceiling:** The diffusion model can only produce what $\mathcal D$ can decode. Fine text, small faces and exact textures that the autoencoder cannot reconstruct cannot be generated either, however good stage 2 is. That is why later systems kept $f=8$ but raised the latent channels (SD3 uses 16), and why reconstruction quality (PSNR, rFID) of the first stage is reported alongside sample FID.

> **Two separate trainings:** Stage 1 is trained once and frozen, then reused for many stage-2 models and tasks. This makes experiments cheap, but the autoencoder is never optimised for what the diffusion model finds easy to model. A latent that reconstructs well is not automatically easy to generate. The KL/VQ regularisation and moderate $f$ are what keep it smooth enough.

**Strengths**

- About $f^2$ less compute per denoising step; training and sampling at 512² on single GPUs
- Perceptual and adversarial losses in stage 1 remove imperceptible detail the diffusion model would otherwise model
- Cross-attention gives one general conditioning mechanism for text, classes and layouts; concatenation handles spatial conditions
- Convolutional latent U-Net generalises to larger latent grids; the frozen autoencoder is reused across models

**Limitations**

- Output quality is capped by the autoencoder (fine text, small details); artefacts of the decoder appear in every sample
- Large $f$ (16–32) loses detail; small $f$ (1–2) loses most of the speed-up
- Two-stage training and a latent scale factor to keep consistent
- The likelihood is of latents, not images, so likelihood comparisons with pixel models do not apply

**Question:** A 768×768 image is generated by an LDM with f = 8 and c = 4. What is the latent shape, and how many positions does a full self-attention layer at that resolution attend over?

- A. 96×96×4, 9,216 positions
- B. 64×64×4, 4,096 positions
- C. 96×96×3, 9,216 positions
- D. 192×192×4, 36,864 positions

**Answer:** A. 96×96×4, 9,216 positions — $768/8=96$, so $z\in\mathbb R^{96\times96\times4}$ and $96^2=9{,}216$ positions, giving about $8.5\times10^7$ attention scores per head. Pixel space would have $768^2=589{,}824$ positions, 64 times more, and $4096$ times more attention scores.

**Question:** Why does LDM regularise its autoencoder only slightly (tiny KL weight or a VQ layer) instead of training a full VAE whose latent is close to N(0, I)?

- A. Because the KL term cannot be computed for images
- B. Because the diffusion model is the prior over z; the regulariser only has to prevent high-variance, arbitrarily scaled latents while keeping reconstructions sharp
- C. Because VQ layers are faster
- D. Because cross-attention requires discrete latents

**Answer:** B. Because the diffusion model is the prior over z; the regulariser only has to prevent high-variance, arbitrarily scaled latents while keeping reconstructions sharp — A strong KL term would trade reconstruction quality for a simple prior, but LDM does not sample $z$ from $\mathcal N(0,I)$. It samples $z$ with the diffusion model. A light regulariser keeps the latent well-scaled and smooth. The remaining scale is fixed by one rescaling factor.

**Key points**

- LDM: autoencoder $z=\mathcal E(x)\in\mathbb R^{H/f\times W/f\times c}$ (perceptual + adversarial loss, KL- or VQ-reg), then diffusion on $z$ with $\|\epsilon-\epsilon_\theta(z_t,t,\tau_\theta(y))\|^2$; sample, then decode $\mathcal D(z_0)$.
- Downsampling $f$ cuts per-step cost by about $f^2$ (attention by $f^4$); LDM-4 to LDM-8 (up to 16) keep quality, LDM-1/2 are slow, LDM-32 loses detail.
- Conditioning: cross-attention with $Q$ from U-Net features and $K,V$ from $\tau_\theta(y)$; spatial conditions are concatenated to $z_t$.
- Latents are rescaled to unit variance by one scalar (0.18215 in Stable Diffusion) before diffusion.
- Results (CVPR 2022): CelebA-HQ FID 5.11, ImageNet class-conditional FID 3.60, text-to-image MS-COCO FID 12.63 with a 1.45B model.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
