Stable Diffusion

Stable Diffusion is the openly released latent diffusion model: an f = 8 VAE maps 512×512 images to 64×64×4 latents (scaled by 0.18215), and an 860M U-Net denoises them with cross-attention to a frozen CLIP ViT-L/14 text encoder. It samples with classifier-free guidance (scale ~7.5) in 20–50 steps.

15 min read1115 wordsadvanced0 viewsupdated 2026-10-12

Definition

Stable Diffusion A family of text-to-image latent diffusion models whose weights were released publicly, starting in August 2022 by CompVis (LMU Munich), Stability AI and Runway, under the CreativeML OpenRAIL-M licence. Version 1 is the recipe of Latent diffusion models at scale. A KL-regularised autoencoder with downsampling factor f=8f=8 maps a 512×512×3512\times512\times3 image to a 64×64×464\times64\times4 latent. An 860M-parameter U-Net predicts the noise in that latent, conditioned through cross-attention on the 77-token hidden states of a frozen CLIP ViT-L/14 text encoder (123M parameters). It was trained on subsets of LAION-5B, and it samples with classifier-free guidance and fast solvers such as PNDM/PLMS, DDIM or DPM-Solver.

Rombach et al.'s CVPR 2022 paper had shown a 1.45B text-to-image LDM trained on LAION-400M with its own transformer text encoder. Stable Diffusion kept the architecture, replaced the text encoder by OpenAI's pretrained CLIP ViT-L/14 (kept frozen), and trained on far more data at 512×512. Because the whole model, about 1.07B parameters, fits in under 10 GB of GPU memory at inference, it ran on consumer GPUs. The open weights made it the base for an ecosystem of fine-tunes, LoRA adapters, ControlNet and img2img and inpainting tools. Later versions changed the text encoder (SD 2), scaled the U-Net (SDXL), and finally replaced the U-Net and the diffusion objective (SD 3).

The three networks of SD v1

SD v1.x components (model card and v1-inference.yaml)

VAE (stage 1)
KL-regularised autoencoder, f=8f=8, c=4c=4, about 84M parameters, LPIPS + patch-GAN training; 5122×3→642×4512^2\times3\to64^2\times4, 48 pixel values per latent value
Latent scale
z=0.18215⋅E(x)z=0.18215\cdot\mathcal E(x) before diffusion and x=D(z/0.18215)x=\mathcal D(z/0.18215) after; the factor gives the latents roughly unit variance
U-Net (stage 2)
860M parameters; base width 320 channels, multipliers (1, 2, 4, 4), 2 ResNet blocks per level; transformer blocks (self-attention, cross-attention, feed-forward, 8 heads) at the 64², 32² and 16² latent resolutions; context dimension 768
Text encoder
frozen CLIP ViT-L/14 text transformer, 123M parameters; the prompt is padded or truncated to 77 tokens and the final hidden states τ(y)∈R77×768\tau(y)\in\mathbb R^{77\times768} are the cross-attention keys and values
Noise schedule
T=1000T=1000, "scaled linear" βt\beta_t: βt\sqrt{\beta_t} linear from 0.00085\sqrt{0.00085} to 0.012\sqrt{0.012}; ϵ\epsilon-prediction
Training
LAION-5B subsets: first laion2B-en at 256², then high-resolution and aesthetic-filtered subsets at 512²; 32×8 A100 GPUs, AdamW, batch 2048, learning rate warmed up to 10−410^{-4} over 10,000 steps, about 150,000 A100-hours; v1-3 and later drop the text 10% of the time for classifier-free guidance

Training is the LDM objective ∥ϵ−ϵθ(zt,t,τ(y))∥2\|\epsilon-\epsilon_\theta(z_t,t,\tau(y))\|^2 with the text encoder frozen, on latents of LAION images and their alt-text captions. Dropping the caption, so that τ(∅)\tau(\varnothing) is the encoding of the empty prompt, teaches the same U-Net an unconditional model. That is what classifier-free guidance needs.

Classifier-free guidance

ϵ~θ(zt,t,y)=ϵθ(zt,t,∅)+w (ϵθ(zt,t,y)−ϵθ(zt,t,∅))⟺∇log⁡p~t(z∣y)=∇log⁡pt(z∣y)+(w−1) ∇log⁡pt(y∣z)\tilde\epsilon_\theta(z_t,t,y)=\epsilon_\theta(z_t,t,\varnothing)+w\,\big(\epsilon_\theta(z_t,t,y)-\epsilon_\theta(z_t,t,\varnothing)\big)\quad\Longleftrightarrow\quad \nabla\log\tilde p_t(z\mid y)=\nabla\log p_t(z\mid y)+(w-1)\,\nabla\log p_t(y\mid z)
ww
guidance scale (guidance_scale in diffusers, 7.5 by default for SD); w=1w=1 is the plain conditional model, w=0w=0 the unconditional one
∅\varnothing
the empty prompt; replacing it by a negative prompt pushes samples away from that text
pt(y∣z)p_t(y\mid z)
an implicit classifier, p(y∣z)∝p(z∣y)/p(z)p(y\mid z)\propto p(z\mid y)/p(z); guidance raises it to the power w−1w-1

Ho and Salimans (NeurIPS 2021 workshop, "Classifier-Free Diffusion Guidance") introduced this as a replacement for classifier guidance: no separate noisy-image classifier, just two evaluations of the same network. In practice the conditional and unconditional inputs are batched, so a 50-step sample costs 100 U-Net evaluations in 50 batched calls. Larger ww gives images that match the prompt better and have more contrast, but are less diverse. Very large ww oversaturates colours, because the guided prediction z^0\hat z_0 leaves the range of real latents. With the text dropped only 10% of the time and a weak text–image coupling, w=1w=1 alone produces images that follow the prompt poorly, which is why SD is almost always used with ww around 5 to 10.

Sampling, img2img and inpainting

z^0=zt−1−αˉt ϵ~αˉt,zt′=αˉt′ z^0+1−αˉt′ ϵ~(DDIM, η=0, t′<t)\hat z_0=\frac{z_t-\sqrt{1-\bar\alpha_t}\,\tilde\epsilon}{\sqrt{\bar\alpha_t}},\qquad z_{t'}=\sqrt{\bar\alpha_{t'}}\,\hat z_0+\sqrt{1-\bar\alpha_{t'}}\,\tilde\epsilon\qquad(\text{DDIM},\ \eta=0,\ t'<t)
z^0\hat z_0
the current guess of the clean latent; decoding it at intermediate steps gives the familiar "preview" images
t′t'
the next timestep of the solver's schedule; with 50 steps and diffusers' "leading" spacing the timesteps are 981, 961, …, 21, 1

Any sampler for an ϵ\epsilon-prediction model works, and the released scripts and libraries exposed many. PLMS/PNDM (pseudo-numerical methods: the original repository's example command used PLMS with 50 steps, and PNDM is the default scheduler of the diffusers v1 pipelines), DDIM, Euler and Euler-ancestral, DPM-Solver++ (good results in 20–25 steps) and Karras-style variants all solve the same probability-flow ODE or reverse SDE (see Score-based generative models and Denoising diffusion implicit models (DDIM)). In diffusers a call looks like StableDiffusionPipeline.from_pretrained(...)(prompt, num_inference_steps=50, guidance_scale=7.5).

img2img is SDEdit (Meng et al., ICLR 2022) in latent space. Encode an input image to zinitz^{\mathrm{init}}, noise it to an intermediate timestep, zt=αˉt zinit+1−αˉt ϵz_t=\sqrt{\bar\alpha_t}\,z^{\mathrm{init}}+\sqrt{1-\bar\alpha_t}\,\epsilon, and run only the remaining steps with the new prompt. The strength s∈[0,1]s\in[0,1] sets how far back to go. With SS solver steps, ⌊sS⌋\lfloor sS\rfloor steps are run, starting at the matching point of the schedule. Low strength keeps the layout and colours of the input, and strength 1 ignores it almost entirely. Inpainting can be done the same way by re-imposing the known region at every step. The dedicated SD inpainting checkpoint instead has a U-Net with 9 input channels: the 4 noisy latent channels, the 4 channels of the masked image's latent, and 1 downsampled mask. It is fine-tuned so that it sees the context directly.

versiondenoisertext encoder(s)native resolutionnotes
SD 1.x (2022)U-Net, 860MCLIP ViT-L/14 (frozen)512²v1-1 … v1-4 by CompVis/Stability, v1-5 by Runway
SD 2.x (Nov–Dec 2022)U-Net, similar sizeOpenCLIP ViT-H/14 (penultimate layer)512² and 768²768² model uses v-prediction; also depth-to-image and ×4 upscaler
SDXL (Podell et al. 2023)U-Net, 2.6BOpenCLIP ViT-bigG/14 + CLIP ViT-L/141024²size and crop conditioning, multi-aspect training, optional refiner model
SD 3 (Esser et al., ICML 2024)MMDiT transformer, up to 8BCLIP-L + OpenCLIP bigG + T5-XXL1024²rectified-flow objective, 16-channel VAE, joint text–image attention
Main Stable Diffusion generations (sizes as published; SD3 sizes are of the family)

SD 3 is a break with the LDM U-Net. Its MMDiT keeps separate weights for text tokens and image-patch tokens but lets both attend jointly in each block, so text representations are updated too. It is trained with the Rectified flow objective (see Flow matching), with timesteps sampled from a logit-normal distribution that emphasises intermediate noise levels.

2-D toy latent · 3 prompts × 2 modes (std 0.18) · exact ε, prompt adherence λ = 0.5“a cat”“a dog”“a car”[−2.4, 2.4]² · rings: 1 and 2 stdprompt“a dog”, w = 7.5 (SD default)modetxt2img, x_T ~ N(0, I)DDIM steps run50 of 50: t = 981 … 1ᾱ at start → signal kept0.0058 → √ᾱ = 0.076U-Net evaluations100 (cond + uncond)on prompt99.4% of 160split over the prompt's 2 modes2 / 157spread around mode0.87 × datavs w: on prompt · spread (1 = data)1012357.51015ε_w = ε(x_t, t, ∅) + w·(ε(x_t, t, c) − ε(x_t, t, ∅)) · x̂₀ = (x_t − √(1−ᾱ_t)·ε_w)/√ᾱ_tDDIM: x_prev = √ᾱ_prev·x̂₀ + √(1−ᾱ_prev)·ε_w · scaled-linear β 0.00085 → 0.012, ᾱ_999 = 0.00466w > 1 tilts p_θ(x | c) by p_θ(c | x)^(w−1): more on-prompt, less variety
prompt

The conditional model is p_θ(x | c) = 0.5·p(x | c) + 0.5·p(x), so w = 1 leaves many off-prompt samples. Schedule, timestep spacing and img2img start step follow SD v1 with the diffusers DDIM scheduler.

fig.A 2-D stand-in for SD's latent space. Three prompts each own two Gaussian modes (std 0.18). The toy model follows the prompt only halfway, p_θ(x | c) = 0.5·p(x | c) + 0.5·p(x), and its ε is exact. Sampling uses SD v1's schedule (scaled-linear β 0.00085 → 0.012, ᾱ₉₉₉ = 0.00466) and diffusers' DDIM with 50 steps from t = 981 to 1, 160 samples, seed 1. For “a dog”, the share of samples on the prompt is 29.4% at w = 0, 60.6% at w = 1, 95.6% at w = 3 and 99.4% at w = 7.5, at the cost of 100 instead of 50 U-Net evaluations. Diversity goes the other way. At w = 1 the two dog modes get 50 and 47 samples, at w = 7.5 only 2 and 157, because guidance pushes away from the cat and car prompts and the dog mode nearest them almost vanishes. At w = 15 all 160 samples sit in one mode, spread 1.56× wider than the data. img2img from cat latents with w = 7.5: strength 0.2 runs 10 steps from t = 181 (√ᾱ = 0.885 of the input kept) and leaves 70.6% of samples as cats. Strength 0.5 runs 25 steps from t = 481 and reaches 95.0% dogs, and strength 0.75 runs 37 steps from t = 721 (ᾱ = 0.0705) and reaches 98.8%.

Worked example: one guided DDIM step and img2img strength

strengthsteps runfirst timestepᾱ_t√ᾱ_t (input kept)√(1 − ᾱ_t) (noise)
0.5254810.30230.5500.835
0.75377210.07050.2660.964
0.8 (diffusers default)407810.04350.2080.978
1.0509810.00580.0760.997
Where img2img starts with 50 DDIM steps (SD v1 schedule, computed in Node and PyTorch)

Two latent coordinates through the first step of a 50-step run

  1. 1At t=981t=981, αˉ981=0.005776\bar\alpha_{981}=0.005776. Let zt=(1.0, −0.5)z_t=(1.0,\,-0.5), ϵθ(zt,t,∅)=(0.2, −0.1)\epsilon_\theta(z_t,t,\varnothing)=(0.2,\,-0.1) and ϵθ(zt,t,y)=(0.5, 0.3)\epsilon_\theta(z_t,t,y)=(0.5,\,0.3).
  2. 2Guidance with w=7.5w=7.5: ϵ~=(0.2+7.5⋅0.3, −0.1+7.5⋅0.4)=(2.45, 2.90)\tilde\epsilon=(0.2+7.5\cdot0.3,\ -0.1+7.5\cdot0.4)=(2.45,\ 2.90). The prompt-specific difference is amplified 7.5 times.
  3. 3Clean-latent guess: z^0=(zt−1−0.005776 ϵ~)/0.005776=(−18.99, −44.63)\hat z_0=(z_t-\sqrt{1-0.005776}\,\tilde\epsilon)/\sqrt{0.005776}=(-18.99,\,-44.63), far outside the unit-variance range of real latents (it is (6.60, −10.52)(6.60,\,-10.52) with w=1w=1). Early guesses are crude; this overshoot is what becomes oversaturation if ww is too large.
  4. 4DDIM to t′=961t'=961 (αˉ961=0.007282\bar\alpha_{961}=0.007282): zt′=0.007282 z^0+0.992718 ϵ~=(0.821, −0.919)z_{t'}=\sqrt{0.007282}\,\hat z_0+\sqrt{0.992718}\,\tilde\epsilon=(0.821,\,-0.919). The latent itself moves only a little per step.
  5. 5After the last step, decode x=D(z0/0.18215)x=\mathcal D(z_0/0.18215), i.e. multiply the latent by 5.495.49 first: 64×64×4→512×512×364\times64\times4\to512\times512\times3.
python
1import torch
2
3T = 1000
4betas = torch.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T, dtype=torch.float64) ** 2 # "scaled_linear"
5abar = torch.cumprod(1 - betas, 0)
6VAE_SCALE = 0.18215 # SD v1/v2 latent scaling factor
7
8def ddim_timesteps(steps): # diffusers DDIM, "leading" spacing, steps_offset = 1
9 return (torch.arange(steps) * (T // steps) + 1).flip(0)
10
11@torch.no_grad()
12def sample(unet, cond, uncond, steps=50, w=7.5, init_latents=None, strength=0.8, generator=None):
13 """cond / uncond: (B, 77, 768) CLIP ViT-L/14 hidden states of the prompt and of "" (or a negative prompt)."""
14 ts = ddim_timesteps(steps)
15 B = cond.shape[0]
16 noise = torch.randn(B, 4, 64, 64, generator=generator, dtype=torch.float64)
17 if init_latents is None: # txt2img: x_T ~ N(0, I)
18 x = noise
19 else: # img2img (SDEdit): noise the image latents to ts[0]
20 n_run = min(int(steps * strength), steps)
21 ts = ts[steps - n_run:]
22 a = abar[ts[0]]
23 x = a.sqrt() * init_latents + (1 - a).sqrt() * noise
24 for i, t in enumerate(ts):
25 e_u, e_c = unet(torch.cat([x, x]), t, torch.cat([uncond, cond])).chunk(2) # one batched call
26 e = e_u + w * (e_c - e_u) # classifier-free guidance
27 a_t = abar[t]
28 a_prev = abar[ts[i + 1]] if i + 1 < len(ts) else abar[0] # set_alpha_to_one = False
29 x0 = (x - (1 - a_t).sqrt() * e) / a_t.sqrt()
30 x = a_prev.sqrt() * x0 + (1 - a_prev).sqrt() * e # DDIM, eta = 0
31 return x / VAE_SCALE # pass to the VAE decoder -> (B, 3, 512, 512)
32
33calls = []
34def toy_unet(x, t, ctx): # stand-in for the 860M-parameter U-Net
35 calls.append(x.shape[0])
36 return 0.1 * x + ctx.mean((1, 2)).view(-1, 1, 1, 1)
37
38g = torch.Generator().manual_seed(0)
39cond, uncond = torch.randn(1, 77, 768, dtype=torch.float64), torch.zeros(1, 77, 768, dtype=torch.float64)
40print(ddim_timesteps(50)[:3].tolist(), ddim_timesteps(50)[-1].item()) # [981, 961, 941] 1
41z = sample(toy_unet, cond, uncond, generator=g)
42print(z.shape, len(calls), sum(calls)) # torch.Size([1, 4, 64, 64]) 50 100
43calls.clear()
44z = sample(toy_unet, cond, uncond, init_latents=torch.randn(1, 4, 64, 64, dtype=torch.float64), strength=0.75, generator=g)
45print(len(calls), f"{abar[721].item():.4f}") # 37 0.0705 (37 steps from t = 721)
46print(f"{abar[999].item():.5f}") # 0.00466: the last training step is not pure noise
The SD v1 sampling loop: scaled-linear schedule, diffusers-style DDIM timesteps, batched classifier-free guidance and img2img strength (a stand-in U-Net; run with PyTorch on CPU)

Use when

  • +Open weights; about 1B parameters in total, so it runs and fine-tunes on consumer GPUs
  • +One model for txt2img, img2img, inpainting (with the 9-channel variant) and, through add-ons, depth, pose or edge control
  • +Any ε-prediction sampler applies; 20–50 steps with modern solvers
  • +Modular: VAE, text encoder, U-Net and scheduler can be swapped or fine-tuned separately

Avoid when

  • −CLIP ViT-L/14 text conditioning handles counting, spatial relations and rendered text poorly (improved in SDXL and SD 3 with larger and T5 encoders)
  • −Quality capped by the f = 8, 4-channel VAE: small faces, hands and text are hard
  • −Needs classifier-free guidance (two evaluations per step) and has a non-zero terminal SNR
  • −Trained on web-scraped LAION data: copyright, consent and bias issues, and a safety filter that is easy to remove

Check yourself

With 50 DDIM steps and img2img strength 0.8, what does the diffusers pipeline do?

Check yourself

In SD v1, what is the role of the factor 0.18215?

Remember

  • ✓SD v1 = LDM: f=8f=8 KL-VAE (5122×3→642×4512^2\times3\to64^2\times4, latents ×0.18215\times0.18215), 860M U-Net with cross-attention to frozen CLIP ViT-L/14 states τ(y)∈R77×768\tau(y)\in\mathbb R^{77\times768} (123M), trained on LAION-5B subsets.
  • ✓Classifier-free guidance: ϵ~=ϵ(zt,∅)+w(ϵ(zt,y)−ϵ(zt,∅))\tilde\epsilon=\epsilon(z_t,\varnothing)+w(\epsilon(z_t,y)-\epsilon(z_t,\varnothing)), w≈7.5w\approx7.5; enabled by dropping the caption 10% of the time; costs two U-Net passes per step.
  • ✓Schedule: T=1000T=1000, scaled-linear β\beta from 0.00085 to 0.012, ϵ\epsilon-prediction; samplers PLMS, DDIM, Euler, DPM-Solver++ in 20–50 steps.
  • ✓img2img: noise the input latent to the timestep at fraction strength of the schedule and run ⌊strength⋅S⌋\lfloor\text{strength}\cdot S\rfloor steps; inpainting model: 9-channel U-Net input (latent, masked-image latent, mask).
  • ✓SD 2: OpenCLIP ViT-H/14, 768² v-prediction; SDXL: 2.6B U-Net, two text encoders, 1024²; SD 3: MMDiT with rectified flow, three text encoders, 16-channel VAE.
← back to Deep Learning roadmap