Definition
Latent diffusion model (LDM) A two-stage generative model from Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR 2022). Stage 1 is an autoencoder trained once, with perceptual and adversarial losses, that maps an image to a latent with , . Stage 2 is a diffusion model, usually a U-Net, trained to denoise instead of . Conditioning (text, class, layout, a low-resolution image) enters through cross-attention from the U-Net's features to a learned encoding . Sampling runs the diffusion in latent space and decodes once: .
Pixel-space diffusion models (see Denoising diffusion probabilistic models (DDPM)) reached top image quality by 2021, but at great cost. Every training and sampling step evaluates the network on the full pixel grid, and the paper notes that the strongest models took 150 to 1000 V100 days to train. Rombach et al. argued that most of those bits are perceptually irrelevant detail. A likelihood-based model spends much of its capacity on imperceptible high-frequency content, which they called perceptual compression, and only a small part on the semantic compression that matters. So they split the work. An autoencoder removes the imperceptible detail once (perceptual compression), and the diffusion model learns the semantic and conceptual composition in a space that is spatially smaller but keeps the 2-D layout. Earlier two-stage models such as VQGAN with a transformer (see Vector-quantized VAE (VQ-VAE) and Autoregressive image generation) needed aggressive compression to make an autoregressive prior affordable. A convolutional diffusion prior works with mild compression, so the latent can stay detailed.
Stage 1: perceptual compression
- pixel plus an LPIPS perceptual loss (see Perceptual loss), so the decoder keeps what looks right rather than what is pixel-exact
- a patch-based discriminator (see Adversarial loss); it forces realistic local texture and avoids the blur of reconstructions
- KL-reg: a slight KL penalty pulling toward , as in a VAE but with a tiny weight (see Variational autoencoder (VAE)); or VQ-reg: a vector-quantisation layer in the decoder, as in VQGAN
- the spatial downsampling factor; the paper trains (LDM-; LDM-1 is pixel space)
The regulariser only keeps the latent from having arbitrary scale and variance. It is not meant to make a sample from a prior, because the diffusion model is the prior. With KL-reg the KL weight is very small (about in the released configs), so is close to a deterministic code. With VQ-reg the diffusion model works on the continuous pre-quantisation features and the quantiser is "absorbed" into the decoder. Because the latents of a KL-reg autoencoder are not unit-variance, LDM rescales them by one scalar, estimated from the standard deviation of the first training batch, before diffusion. Stable Diffusion's is that scalar (see Stable Diffusion).
Stage 2: diffusion in latent space
- a time-conditional U-Net on latents: convolutions keep the 2-D inductive bias, unlike a 1-D token prior
- a domain-specific encoder of the condition , trained jointly with : a transformer over BERT-tokenised text in the paper, CLIP's text encoder in Stable Diffusion
- computed with the frozen encoder; is used only to decode samples
The objective is the ordinary noise-prediction loss (see Diffusion noise-prediction loss), so any diffusion sampler applies unchanged. The paper used DDIM (see Denoising diffusion implicit models (DDIM)) for most results. The U-Net is fully convolutional, so a model trained on latents can be sampled on larger latent grids for larger images, as in the paper's -pixel-wide landscapes. Results are good but not always coherent at large sizes.
Conditioning by cross-attention
- the U-Net's intermediate feature map at layer , flattened so each of the spatial positions is a query
- condition tokens, e.g. 77 text tokens; they supply keys and values
- per-layer projections; the block is self-attention, then cross-attention, then a feed-forward layer, each with a residual connection
Each image position reads from the condition tokens. The attention matrix is , not , so cross-attention is cheap next to the U-Net's self-attention at the same resolution (see Cross-attention). Spatially aligned conditions such as semantic maps, low-resolution images for super-resolution, or masked images for inpainting are simpler. They are concatenated to as extra input channels.
Findings and results in the paper
- Choice of f
- LDM-1 and LDM-2 train slowly; LDM-32 compresses too much and caps quality. LDM-4 to LDM-16 balance efficiency and fidelity; after 2M training steps on class-conditional ImageNet the FID gap between pixel-space LDM-1 and LDM-8 was 38
- Unconditional, 256²
- FID 5.11 on CelebA-HQ (then state of the art), 4.98 on FFHQ, 4.02 on LSUN-Churches, 2.95 on LSUN-Bedrooms
- Class-conditional ImageNet 256²
- LDM-4-G with classifier-free guidance (scale 1.5): FID 3.60, Inception score 247.67, with fewer parameters and less compute than ADM
- Text-to-image
- LDM-KL-8-G, 1.45B parameters, trained on LAION-400M with a transformer text encoder: zero-shot MS-COCO 256² FID 12.63 with guidance 1.5
- Other tasks
- state of the art in inpainting; competitive super-resolution and semantic-map-to-image synthesis, all with the same recipe
The autoencoder here is a fixed block DCT that keeps the c lowest frequencies of each f×f block (capped at f²). A learned KL- or VQ-regularised autoencoder with perceptual and adversarial losses reconstructs far better at the same latent shape.
Worked example: what f buys at 512×512
| f | latent | values | pixel values per latent value | self-attn scores / head | toy PSNR, c = 4 |
|---|---|---|---|---|---|
| 1 | 512×512×3 | 786,432 | 1 | 6.87 × 10¹⁰ | lossless |
| 2 | 256×256×4 | 262,144 | 3 | 4.29 × 10⁹ | lossless |
| 4 | 128×128×4 | 65,536 | 12 | 2.68 × 10⁸ | 22.5 dB |
| 8 | 64×64×4 | 16,384 | 48 | 1.68 × 10⁷ | 20.2 dB |
| 16 | 32×32×4 | 4,096 | 192 | 1.05 × 10⁶ | 18.4 dB |
| 32 | 16×16×4 | 1,024 | 768 | 6.55 × 10⁴ | 17.4 dB |
Reading the f = 8 row
- 1, so : numbers against , a factor .
- 2Convolutional layers at a fixed width cost in proportion to positions: of pixel space per step.
- 3Self-attention over all latent positions: scores per head, 32 MiB in fp16. Over pixels it would be scores, more. Pixel models therefore restrict attention to low resolutions.
- 4Cross-attention to 77 text tokens at the same resolution: scores per head, 53 times fewer than the self-attention.
- 5Training also gets cheaper: can be precomputed once per image, and the decoder runs only once per sample, not once per step.
1import math, torch, torch.nn as nn, torch.nn.functional as F23torch.manual_seed(0)45class Autoencoder(nn.Module): # first stage: f = 8, c = 4 (KL-reg, trained beforehand)6 def __init__(self, c=4, w=64):7 super().__init__()8 self.enc = nn.Sequential(nn.Conv2d(3, w, 3, 2, 1), nn.SiLU(), nn.Conv2d(w, w, 3, 2, 1), nn.SiLU(),9 nn.Conv2d(w, w, 3, 2, 1), nn.SiLU(), nn.Conv2d(w, 2 * c, 3, 1, 1))10 self.dec = nn.Sequential(nn.Conv2d(c, w, 3, 1, 1), nn.SiLU(), nn.Upsample(scale_factor=8), nn.Conv2d(w, 3, 3, 1, 1))11 def encode(self, x): # returns a sample of q(z | x) = N(mu, sigma^2)12 mu, logvar = self.enc(x).chunk(2, dim=1)13 return mu + (0.5 * logvar).exp() * torch.randn_like(mu)14 def decode(self, z):15 return self.dec(z)1617class CrossAttention(nn.Module): # Q from the latent, K and V from tau_theta(y)18 def __init__(self, d, d_ctx, heads=4):19 super().__init__()20 self.h = heads21 self.q, self.k, self.v = nn.Linear(d, d, bias=False), nn.Linear(d_ctx, d, bias=False), nn.Linear(d_ctx, d, bias=False)22 self.o = nn.Linear(d, d)23 def forward(self, x, ctx): # x (B, N, d), ctx (B, M, d_ctx)24 B, N, d = x.shape25 split = lambda t: t.view(B, -1, self.h, d // self.h).transpose(1, 2)26 out = F.scaled_dot_product_attention(split(self.q(x)), split(self.k(ctx)), split(self.v(ctx)))27 return self.o(out.transpose(1, 2).reshape(B, N, d))2829class Denoiser(nn.Module): # eps_theta(z_t, t, tau_theta(y)), a one-level "U-Net"30 def __init__(self, c=4, d=128, d_ctx=64):31 super().__init__()32 self.inp, self.out = nn.Conv2d(c, d, 3, 1, 1), nn.Conv2d(d, c, 3, 1, 1)33 self.temb = nn.Sequential(nn.Linear(d, d), nn.SiLU(), nn.Linear(d, d))34 self.res = nn.Sequential(nn.GroupNorm(8, d), nn.SiLU(), nn.Conv2d(d, d, 3, 1, 1))35 self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)36 self.self_attn = nn.MultiheadAttention(d, 4, batch_first=True)37 self.cross = CrossAttention(d, d_ctx)38 def forward(self, z, t, ctx):39 half = 6440 freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)41 temb = self.temb(torch.cat([torch.sin(t[:, None] * freqs), torch.cos(t[:, None] * freqs)], 1))42 h = self.inp(z) + temb[:, :, None, None]43 h = h + self.res(h)44 B, d, H, W = h.shape45 s = h.flatten(2).transpose(1, 2) # (B, H*W, d): every latent position is a token46 s = s + self.self_attn(self.n1(s), self.n1(s), self.n1(s), need_weights=False)[0]47 s = s + self.cross(self.n2(s), ctx)48 return self.out(s.transpose(1, 2).view(B, d, H, W))4950T = 100051betas = torch.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T) ** 252abar = torch.cumprod(1 - betas, 0)5354ae, eps_model = Autoencoder(), Denoiser()55ae.requires_grad_(False) # stage 1 is frozen while the diffusion model trains56tau = nn.Embedding(1000, 64) # tau_theta: a stand-in text encoder (token embeddings)57opt = torch.optim.AdamW(list(eps_model.parameters()) + list(tau.parameters()), lr=1e-4)5859x = torch.randn(2, 3, 256, 256) # images in [-1, 1] in practice60tokens = torch.randint(0, 1000, (2, 77))61with torch.no_grad():62 z0 = ae.encode(x)63scale = 1 / z0.std() # SD v1 fixed this once: 0.1821564z0 = z0 * scale65t = torch.randint(0, T, (2,))66noise = torch.randn_like(z0)67a = abar[t].view(-1, 1, 1, 1)68z_t = a.sqrt() * z0 + (1 - a).sqrt() * noise69loss = F.mse_loss(eps_model(z_t, t.float(), tau(tokens)), noise)70opt.zero_grad(); loss.backward(); opt.step()71print(x.shape, "->", z0.shape) # torch.Size([2, 3, 256, 256]) -> torch.Size([2, 4, 32, 32])72print(x[0].numel() // z0[0].numel()) # 48 pixel values per latent value73print(ae.decode(z0 / scale).shape) # torch.Size([2, 3, 256, 256])Use when
- +About less compute per denoising step; training and sampling at 512² on single GPUs
- +Perceptual and adversarial losses in stage 1 remove imperceptible detail the diffusion model would otherwise model
- +Cross-attention gives one general conditioning mechanism for text, classes and layouts; concatenation handles spatial conditions
- +Convolutional latent U-Net generalises to larger latent grids; the frozen autoencoder is reused across models
Avoid when
- −Output quality is capped by the autoencoder (fine text, small details); artefacts of the decoder appear in every sample
- −Large (16–32) loses detail; small (1–2) loses most of the speed-up
- −Two-stage training and a latent scale factor to keep consistent
- −The likelihood is of latents, not images, so likelihood comparisons with pixel models do not apply
Check yourself
A 768×768 image is generated by an LDM with f = 8 and c = 4. What is the latent shape, and how many positions does a full self-attention layer at that resolution attend over?
Check yourself
Why does LDM regularise its autoencoder only slightly (tiny KL weight or a VQ layer) instead of training a full VAE whose latent is close to N(0, I)?
Remember
- ✓LDM: autoencoder (perceptual + adversarial loss, KL- or VQ-reg), then diffusion on with ; sample, then decode .
- ✓Downsampling cuts per-step cost by about (attention by ); LDM-4 to LDM-8 (up to 16) keep quality, LDM-1/2 are slow, LDM-32 loses detail.
- ✓Conditioning: cross-attention with from U-Net features and from ; spatial conditions are concatenated to .
- ✓Latents are rescaled to unit variance by one scalar (0.18215 in Stable Diffusion) before diffusion.
- ✓Results (CVPR 2022): CelebA-HQ FID 5.11, ImageNet class-conditional FID 3.60, text-to-image MS-COCO FID 12.63 with a 1.45B model.