Latent diffusion models

A latent diffusion model (Rombach et al., CVPR 2022) first trains an autoencoder that compresses images by a factor f per side into a latent z = E(x), then trains a diffusion model on z, conditioned through cross-attention on τ_θ(y). Per-step compute falls by about f²; f = 4–8 keeps quality.

14 min read901 wordsadvanced0 viewsupdated 2026-10-12

Definition

Latent diffusion model (LDM) A two-stage generative model from Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR 2022). Stage 1 is an autoencoder E,D\mathcal E,\mathcal D trained once, with perceptual and adversarial losses, that maps an image x∈RH×W×3x\in\mathbb R^{H\times W\times3} to a latent z=E(x)∈Rh×w×cz=\mathcal E(x)\in\mathbb R^{h\times w\times c} with h=H/fh=H/f, w=W/fw=W/f. Stage 2 is a diffusion model, usually a U-Net, trained to denoise zz instead of xx. Conditioning (text, class, layout, a low-resolution image) enters through cross-attention from the U-Net's features to a learned encoding τθ(y)\tau_\theta(y). Sampling runs the diffusion in latent space and decodes once: x~=D(z0)\tilde x=\mathcal D(z_0).

Pixel-space diffusion models (see Denoising diffusion probabilistic models (DDPM)) reached top image quality by 2021, but at great cost. Every training and sampling step evaluates the network on the full pixel grid, and the paper notes that the strongest models took 150 to 1000 V100 days to train. Rombach et al. argued that most of those bits are perceptually irrelevant detail. A likelihood-based model spends much of its capacity on imperceptible high-frequency content, which they called perceptual compression, and only a small part on the semantic compression that matters. So they split the work. An autoencoder removes the imperceptible detail once (perceptual compression), and the diffusion model learns the semantic and conceptual composition in a space that is spatially smaller but keeps the 2-D layout. Earlier two-stage models such as VQGAN with a transformer (see Vector-quantized VAE (VQ-VAE) and Autoregressive image generation) needed aggressive compression to make an autoregressive prior affordable. A convolutional diffusion prior works with mild compression, so the latent can stay detailed.

Stage 1: perceptual compression

LAE=min⁡E,Dmax⁡ψ(Lrec(x,D(E(x)))−Ladv(D(E(x)))+log⁡Dψ(x)+Lreg(x;E,D))\mathcal L_{\mathrm{AE}}=\min_{\mathcal E,\mathcal D}\max_{\psi}\Big(\mathcal L_{\mathrm{rec}}\big(x,\mathcal D(\mathcal E(x))\big)-\mathcal L_{\mathrm{adv}}\big(\mathcal D(\mathcal E(x))\big)+\log D_\psi(x)+\mathcal L_{\mathrm{reg}}(x;\mathcal E,\mathcal D)\Big)
Lrec\mathcal L_{\mathrm{rec}}
pixel ℓ1\ell_1 plus an LPIPS perceptual loss (see Perceptual loss), so the decoder keeps what looks right rather than what is pixel-exact
DψD_\psi
a patch-based discriminator (see Adversarial loss); it forces realistic local texture and avoids the blur of ℓ2\ell_2 reconstructions
Lreg\mathcal L_{\mathrm{reg}}
KL-reg: a slight KL penalty pulling qE(z∣x)q_{\mathcal E}(z\mid x) toward N(0,I)\mathcal N(0,I), as in a VAE but with a tiny weight (see Variational autoencoder (VAE)); or VQ-reg: a vector-quantisation layer in the decoder, as in VQGAN
f=H/h=2mf=H/h=2^m
the spatial downsampling factor; the paper trains f∈{1,2,4,8,16,32}f\in\{1,2,4,8,16,32\} (LDM-ff; LDM-1 is pixel space)

The regulariser only keeps the latent from having arbitrary scale and variance. It is not meant to make zz a sample from a prior, because the diffusion model is the prior. With KL-reg the KL weight is very small (about 10−610^{-6} in the released configs), so zz is close to a deterministic code. With VQ-reg the diffusion model works on the continuous pre-quantisation features and the quantiser is "absorbed" into the decoder. Because the latents of a KL-reg autoencoder are not unit-variance, LDM rescales them by one scalar, estimated from the standard deviation of the first training batch, before diffusion. Stable Diffusion's 0.182150.18215 is that scalar (see Stable Diffusion).

Stage 2: diffusion in latent space

LLDM=EE(x), y, ϵ∼N(0,I), t[∥ϵ−ϵθ(zt, t, τθ(y))∥22],zt=αˉt E(x)+1−αˉt ϵ\mathcal L_{\mathrm{LDM}}=\mathbb E_{\mathcal E(x),\,y,\,\epsilon\sim\mathcal N(0,I),\,t}\Big[\big\|\epsilon-\epsilon_\theta\big(z_t,\,t,\,\tau_\theta(y)\big)\big\|_2^2\Big],\qquad z_t=\sqrt{\bar\alpha_t}\,\mathcal E(x)+\sqrt{1-\bar\alpha_t}\,\epsilon
ϵθ\epsilon_\theta
a time-conditional U-Net on h×w×ch\times w\times c latents: convolutions keep the 2-D inductive bias, unlike a 1-D token prior
τθ(y)\tau_\theta(y)
a domain-specific encoder of the condition yy, trained jointly with ϵθ\epsilon_\theta: a transformer over BERT-tokenised text in the paper, CLIP's text encoder in Stable Diffusion
E(x)\mathcal E(x)
computed with the frozen encoder; D\mathcal D is used only to decode samples

The objective is the ordinary noise-prediction loss (see Diffusion noise-prediction loss), so any diffusion sampler applies unchanged. The paper used DDIM (see Denoising diffusion implicit models (DDIM)) for most results. The U-Net is fully convolutional, so a model trained on 64×6464\times64 latents can be sampled on larger latent grids for larger images, as in the paper's 10241024-pixel-wide landscapes. Results are good but not always coherent at large sizes.

Conditioning by cross-attention

Attention(Q,K,V)=softmax ⁣(QK⊤d)V,Q=WQ(i)φi(zt),K=WK(i)τθ(y),V=WV(i)τθ(y)\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt d}\Big)V,\qquad Q=W_Q^{(i)}\varphi_i(z_t),\quad K=W_K^{(i)}\tau_\theta(y),\quad V=W_V^{(i)}\tau_\theta(y)
φi(zt)∈RN×dϵi\varphi_i(z_t)\in\mathbb R^{N\times d_\epsilon^i}
the U-Net's intermediate feature map at layer ii, flattened so each of the N=hiwiN=h_i w_i spatial positions is a query
τθ(y)∈RM×dτ\tau_\theta(y)\in\mathbb R^{M\times d_\tau}
MM condition tokens, e.g. 77 text tokens; they supply keys and values
W(i)W^{(i)}
per-layer projections; the block is self-attention, then cross-attention, then a feed-forward layer, each with a residual connection

Each image position reads from the condition tokens. The attention matrix is N×MN\times M, not N×NN\times N, so cross-attention is cheap next to the U-Net's self-attention at the same resolution (see Cross-attention). Spatially aligned conditions such as semantic maps, low-resolution images for super-resolution, or masked images for inpainting are simpler. They are concatenated to ztz_t as extra input channels.

Findings and results in the paper

Choice of f
LDM-1 and LDM-2 train slowly; LDM-32 compresses too much and caps quality. LDM-4 to LDM-16 balance efficiency and fidelity; after 2M training steps on class-conditional ImageNet the FID gap between pixel-space LDM-1 and LDM-8 was 38
Unconditional, 256²
FID 5.11 on CelebA-HQ (then state of the art), 4.98 on FFHQ, 4.02 on LSUN-Churches, 2.95 on LSUN-Bedrooms
Class-conditional ImageNet 256²
LDM-4-G with classifier-free guidance (scale 1.5): FID 3.60, Inception score 247.67, with fewer parameters and less compute than ADM
Text-to-image
LDM-KL-8-G, 1.45B parameters, trained on LAION-400M with a transformer text encoder: zero-shot MS-COCO 256² FID 12.63 with guidance 1.5
Other tasks
state of the art in inpainting; competitive 4×4\times super-resolution and semantic-map-to-image synthesis, all with the same recipe
stage 1: x (64×64) → z = E(x) (8×8×4) → x̃ = D(z) · stage 2 diffuses z, not xinput xD(E(x)), PSNR 20.2 dBz channel 0 (block mean), 8×8latent shape8×8×4values kept256 of 4096compression16.0×recon MSE0.00955reconstruction PSNR (dB) vs f, c = 420301∞2∞422.5820.21618.43217.4attention scores / head / layer at 512² (log)1M1G168.7G24.3G4268.4M816.8M161.0M3265.5K512×512×3 → latent 64×64×4: 16.4K values (48 pixel values per latent value)conv FLOPs ∝ positions = 1/64 of pixel space · attention ∝ positions² = 16.8M scoresfp16 attention matrix at full latent resolution: 32.0 MiB
factor f
image size

The autoencoder here is a fixed block DCT that keeps the c lowest frequencies of each f×f block (capped at f²). A learned KL- or VQ-regularised autoencoder with perceptual and adversarial losses reconstructs far better at the same latent shape.

fig.A 64×64 toy image passes through a fixed stand-in for stage 1, a block DCT that keeps the c lowest frequencies of each f×f block. The latent is (64/f)×(64/f)×c. With the Stable Diffusion shape (f = 8, c = 4) the latent is 8×8×4, 256 of 4096 values (16× fewer), and the reconstruction has PSNR 20.2 dB (MSE 0.00955). At c = 4 the PSNR falls with f: lossless at f = 1 and 2 (four coefficients cover a 2×2 block), then 22.5 dB at f = 4, 20.2 at f = 8, 18.4 at f = 16 and 17.4 at f = 32. With c = 16 it is 22.7 dB at f = 8 and 20.6 at f = 16. The right chart shows what the second stage gains. At 512×512 one full-resolution self-attention layer has 68.7G scores per head in pixel space (a 128 GiB fp16 matrix). At f = 4 it has 268M, at f = 8 16.8M (32 MiB), and at f = 16 1.0M. Convolution cost per step falls as 1/f² (1/64 at f = 8). A learned KL autoencoder with LPIPS and adversarial losses keeps far more detail than the DCT at the same latent shape. That is why f = 4–8 costs little perceived quality.

Worked example: what f buys at 512×512

flatentvaluespixel values per latent valueself-attn scores / headtoy PSNR, c = 4
1512×512×3786,43216.87 × 10¹⁰lossless
2256×256×4262,14434.29 × 10⁹lossless
4128×128×465,536122.68 × 10⁸22.5 dB
864×64×416,384481.68 × 10⁷20.2 dB
1632×32×44,0961921.05 × 10⁶18.4 dB
3216×16×41,0247686.55 × 10⁴17.4 dB
Latent size and one self-attention layer's cost at 512×512×3, c = 4 latent channels (c = 3 at f = 1); toy PSNR from the figure (computed in Node)

Reading the f = 8 row

  1. 1512/8=64512/8=64, so z∈R64×64×4z\in\mathbb R^{64\times64\times4}: 16,38416{,}384 numbers against 786,432786{,}432, a factor 48=82⋅3/448=8^2\cdot3/4.
  2. 2Convolutional layers at a fixed width cost in proportion to positions: 642/5122=1/6464^2/512^2=1/64 of pixel space per step.
  3. 3Self-attention over all 40964096 latent positions: 40962=16,777,2164096^2=16{,}777{,}216 scores per head, 32 MiB in fp16. Over 262,144262{,}144 pixels it would be 6.87×10106.87\times10^{10} scores, 4096×4096\times more. Pixel models therefore restrict attention to low resolutions.
  4. 4Cross-attention to 77 text tokens at the same resolution: 4096×77=315,3924096\times77=315{,}392 scores per head, 53 times fewer than the self-attention.
  5. 5Training also gets cheaper: E(x)\mathcal E(x) can be precomputed once per image, and the decoder runs only once per sample, not once per step.
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3torch.manual_seed(0)
4
5class Autoencoder(nn.Module): # first stage: f = 8, c = 4 (KL-reg, trained beforehand)
6 def __init__(self, c=4, w=64):
7 super().__init__()
8 self.enc = nn.Sequential(nn.Conv2d(3, w, 3, 2, 1), nn.SiLU(), nn.Conv2d(w, w, 3, 2, 1), nn.SiLU(),
9 nn.Conv2d(w, w, 3, 2, 1), nn.SiLU(), nn.Conv2d(w, 2 * c, 3, 1, 1))
10 self.dec = nn.Sequential(nn.Conv2d(c, w, 3, 1, 1), nn.SiLU(), nn.Upsample(scale_factor=8), nn.Conv2d(w, 3, 3, 1, 1))
11 def encode(self, x): # returns a sample of q(z | x) = N(mu, sigma^2)
12 mu, logvar = self.enc(x).chunk(2, dim=1)
13 return mu + (0.5 * logvar).exp() * torch.randn_like(mu)
14 def decode(self, z):
15 return self.dec(z)
16
17class CrossAttention(nn.Module): # Q from the latent, K and V from tau_theta(y)
18 def __init__(self, d, d_ctx, heads=4):
19 super().__init__()
20 self.h = heads
21 self.q, self.k, self.v = nn.Linear(d, d, bias=False), nn.Linear(d_ctx, d, bias=False), nn.Linear(d_ctx, d, bias=False)
22 self.o = nn.Linear(d, d)
23 def forward(self, x, ctx): # x (B, N, d), ctx (B, M, d_ctx)
24 B, N, d = x.shape
25 split = lambda t: t.view(B, -1, self.h, d // self.h).transpose(1, 2)
26 out = F.scaled_dot_product_attention(split(self.q(x)), split(self.k(ctx)), split(self.v(ctx)))
27 return self.o(out.transpose(1, 2).reshape(B, N, d))
28
29class Denoiser(nn.Module): # eps_theta(z_t, t, tau_theta(y)), a one-level "U-Net"
30 def __init__(self, c=4, d=128, d_ctx=64):
31 super().__init__()
32 self.inp, self.out = nn.Conv2d(c, d, 3, 1, 1), nn.Conv2d(d, c, 3, 1, 1)
33 self.temb = nn.Sequential(nn.Linear(d, d), nn.SiLU(), nn.Linear(d, d))
34 self.res = nn.Sequential(nn.GroupNorm(8, d), nn.SiLU(), nn.Conv2d(d, d, 3, 1, 1))
35 self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
36 self.self_attn = nn.MultiheadAttention(d, 4, batch_first=True)
37 self.cross = CrossAttention(d, d_ctx)
38 def forward(self, z, t, ctx):
39 half = 64
40 freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)
41 temb = self.temb(torch.cat([torch.sin(t[:, None] * freqs), torch.cos(t[:, None] * freqs)], 1))
42 h = self.inp(z) + temb[:, :, None, None]
43 h = h + self.res(h)
44 B, d, H, W = h.shape
45 s = h.flatten(2).transpose(1, 2) # (B, H*W, d): every latent position is a token
46 s = s + self.self_attn(self.n1(s), self.n1(s), self.n1(s), need_weights=False)[0]
47 s = s + self.cross(self.n2(s), ctx)
48 return self.out(s.transpose(1, 2).view(B, d, H, W))
49
50T = 1000
51betas = torch.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T) ** 2
52abar = torch.cumprod(1 - betas, 0)
53
54ae, eps_model = Autoencoder(), Denoiser()
55ae.requires_grad_(False) # stage 1 is frozen while the diffusion model trains
56tau = nn.Embedding(1000, 64) # tau_theta: a stand-in text encoder (token embeddings)
57opt = torch.optim.AdamW(list(eps_model.parameters()) + list(tau.parameters()), lr=1e-4)
58
59x = torch.randn(2, 3, 256, 256) # images in [-1, 1] in practice
60tokens = torch.randint(0, 1000, (2, 77))
61with torch.no_grad():
62 z0 = ae.encode(x)
63scale = 1 / z0.std() # SD v1 fixed this once: 0.18215
64z0 = z0 * scale
65t = torch.randint(0, T, (2,))
66noise = torch.randn_like(z0)
67a = abar[t].view(-1, 1, 1, 1)
68z_t = a.sqrt() * z0 + (1 - a).sqrt() * noise
69loss = F.mse_loss(eps_model(z_t, t.float(), tau(tokens)), noise)
70opt.zero_grad(); loss.backward(); opt.step()
71print(x.shape, "->", z0.shape) # torch.Size([2, 3, 256, 256]) -> torch.Size([2, 4, 32, 32])
72print(x[0].numel() // z0[0].numel()) # 48 pixel values per latent value
73print(ae.decode(z0 / scale).shape) # torch.Size([2, 3, 256, 256])
One LDM training step: frozen f = 8 autoencoder, latent rescaling, and a denoiser with self- and cross-attention over latent positions (toy sizes; shapes checked with PyTorch 2.x on CPU)

Use when

  • +About f2f^2 less compute per denoising step; training and sampling at 512² on single GPUs
  • +Perceptual and adversarial losses in stage 1 remove imperceptible detail the diffusion model would otherwise model
  • +Cross-attention gives one general conditioning mechanism for text, classes and layouts; concatenation handles spatial conditions
  • +Convolutional latent U-Net generalises to larger latent grids; the frozen autoencoder is reused across models

Avoid when

  • −Output quality is capped by the autoencoder (fine text, small details); artefacts of the decoder appear in every sample
  • −Large ff (16–32) loses detail; small ff (1–2) loses most of the speed-up
  • −Two-stage training and a latent scale factor to keep consistent
  • −The likelihood is of latents, not images, so likelihood comparisons with pixel models do not apply

Check yourself

A 768×768 image is generated by an LDM with f = 8 and c = 4. What is the latent shape, and how many positions does a full self-attention layer at that resolution attend over?

Check yourself

Why does LDM regularise its autoencoder only slightly (tiny KL weight or a VQ layer) instead of training a full VAE whose latent is close to N(0, I)?

Remember

  • ✓LDM: autoencoder z=E(x)∈RH/f×W/f×cz=\mathcal E(x)\in\mathbb R^{H/f\times W/f\times c} (perceptual + adversarial loss, KL- or VQ-reg), then diffusion on zz with ∥ϵ−ϵθ(zt,t,τθ(y))∥2\|\epsilon-\epsilon_\theta(z_t,t,\tau_\theta(y))\|^2; sample, then decode D(z0)\mathcal D(z_0).
  • ✓Downsampling ff cuts per-step cost by about f2f^2 (attention by f4f^4); LDM-4 to LDM-8 (up to 16) keep quality, LDM-1/2 are slow, LDM-32 loses detail.
  • ✓Conditioning: cross-attention with QQ from U-Net features and K,VK,V from τθ(y)\tau_\theta(y); spatial conditions are concatenated to ztz_t.
  • ✓Latents are rescaled to unit variance by one scalar (0.18215 in Stable Diffusion) before diffusion.
  • ✓Results (CVPR 2022): CelebA-HQ FID 5.11, ImageNet class-conditional FID 3.60, text-to-image MS-COCO FID 12.63 with a 1.45B model.
← back to Deep Learning roadmap