Definition
Stable Diffusion A family of text-to-image latent diffusion models whose weights were released publicly, starting in August 2022 by CompVis (LMU Munich), Stability AI and Runway, under the CreativeML OpenRAIL-M licence. Version 1 is the recipe of Latent diffusion models at scale. A KL-regularised autoencoder with downsampling factor maps a image to a latent. An 860M-parameter U-Net predicts the noise in that latent, conditioned through cross-attention on the 77-token hidden states of a frozen CLIP ViT-L/14 text encoder (123M parameters). It was trained on subsets of LAION-5B, and it samples with classifier-free guidance and fast solvers such as PNDM/PLMS, DDIM or DPM-Solver.
Rombach et al.'s CVPR 2022 paper had shown a 1.45B text-to-image LDM trained on LAION-400M with its own transformer text encoder. Stable Diffusion kept the architecture, replaced the text encoder by OpenAI's pretrained CLIP ViT-L/14 (kept frozen), and trained on far more data at 512×512. Because the whole model, about 1.07B parameters, fits in under 10 GB of GPU memory at inference, it ran on consumer GPUs. The open weights made it the base for an ecosystem of fine-tunes, LoRA adapters, ControlNet and img2img and inpainting tools. Later versions changed the text encoder (SD 2), scaled the U-Net (SDXL), and finally replaced the U-Net and the diffusion objective (SD 3).
The three networks of SD v1
SD v1.x components (model card and v1-inference.yaml)
- VAE (stage 1)
- KL-regularised autoencoder, , , about 84M parameters, LPIPS + patch-GAN training; , 48 pixel values per latent value
- Latent scale
- before diffusion and after; the factor gives the latents roughly unit variance
- U-Net (stage 2)
- 860M parameters; base width 320 channels, multipliers (1, 2, 4, 4), 2 ResNet blocks per level; transformer blocks (self-attention, cross-attention, feed-forward, 8 heads) at the 64², 32² and 16² latent resolutions; context dimension 768
- Text encoder
- frozen CLIP ViT-L/14 text transformer, 123M parameters; the prompt is padded or truncated to 77 tokens and the final hidden states are the cross-attention keys and values
- Noise schedule
- , "scaled linear" : linear from to ; -prediction
- Training
- LAION-5B subsets: first laion2B-en at 256², then high-resolution and aesthetic-filtered subsets at 512²; 32×8 A100 GPUs, AdamW, batch 2048, learning rate warmed up to over 10,000 steps, about 150,000 A100-hours; v1-3 and later drop the text 10% of the time for classifier-free guidance
Training is the LDM objective with the text encoder frozen, on latents of LAION images and their alt-text captions. Dropping the caption, so that is the encoding of the empty prompt, teaches the same U-Net an unconditional model. That is what classifier-free guidance needs.
Classifier-free guidance
- guidance scale (
guidance_scalein diffusers, 7.5 by default for SD); is the plain conditional model, the unconditional one - the empty prompt; replacing it by a negative prompt pushes samples away from that text
- an implicit classifier, ; guidance raises it to the power
Ho and Salimans (NeurIPS 2021 workshop, "Classifier-Free Diffusion Guidance") introduced this as a replacement for classifier guidance: no separate noisy-image classifier, just two evaluations of the same network. In practice the conditional and unconditional inputs are batched, so a 50-step sample costs 100 U-Net evaluations in 50 batched calls. Larger gives images that match the prompt better and have more contrast, but are less diverse. Very large oversaturates colours, because the guided prediction leaves the range of real latents. With the text dropped only 10% of the time and a weak text–image coupling, alone produces images that follow the prompt poorly, which is why SD is almost always used with around 5 to 10.
Sampling, img2img and inpainting
- the current guess of the clean latent; decoding it at intermediate steps gives the familiar "preview" images
- the next timestep of the solver's schedule; with 50 steps and diffusers' "leading" spacing the timesteps are 981, 961, …, 21, 1
Any sampler for an -prediction model works, and the released scripts and libraries exposed many. PLMS/PNDM (pseudo-numerical methods: the original repository's example command used PLMS with 50 steps, and PNDM is the default scheduler of the diffusers v1 pipelines), DDIM, Euler and Euler-ancestral, DPM-Solver++ (good results in 20–25 steps) and Karras-style variants all solve the same probability-flow ODE or reverse SDE (see Score-based generative models and Denoising diffusion implicit models (DDIM)). In diffusers a call looks like StableDiffusionPipeline.from_pretrained(...)(prompt, num_inference_steps=50, guidance_scale=7.5).
img2img is SDEdit (Meng et al., ICLR 2022) in latent space. Encode an input image to , noise it to an intermediate timestep, , and run only the remaining steps with the new prompt. The strength sets how far back to go. With solver steps, steps are run, starting at the matching point of the schedule. Low strength keeps the layout and colours of the input, and strength 1 ignores it almost entirely. Inpainting can be done the same way by re-imposing the known region at every step. The dedicated SD inpainting checkpoint instead has a U-Net with 9 input channels: the 4 noisy latent channels, the 4 channels of the masked image's latent, and 1 downsampled mask. It is fine-tuned so that it sees the context directly.
| version | denoiser | text encoder(s) | native resolution | notes |
|---|---|---|---|---|
| SD 1.x (2022) | U-Net, 860M | CLIP ViT-L/14 (frozen) | 512² | v1-1 … v1-4 by CompVis/Stability, v1-5 by Runway |
| SD 2.x (Nov–Dec 2022) | U-Net, similar size | OpenCLIP ViT-H/14 (penultimate layer) | 512² and 768² | 768² model uses v-prediction; also depth-to-image and ×4 upscaler |
| SDXL (Podell et al. 2023) | U-Net, 2.6B | OpenCLIP ViT-bigG/14 + CLIP ViT-L/14 | 1024² | size and crop conditioning, multi-aspect training, optional refiner model |
| SD 3 (Esser et al., ICML 2024) | MMDiT transformer, up to 8B | CLIP-L + OpenCLIP bigG + T5-XXL | 1024² | rectified-flow objective, 16-channel VAE, joint text–image attention |
SD 3 is a break with the LDM U-Net. Its MMDiT keeps separate weights for text tokens and image-patch tokens but lets both attend jointly in each block, so text representations are updated too. It is trained with the Rectified flow objective (see Flow matching), with timesteps sampled from a logit-normal distribution that emphasises intermediate noise levels.
The conditional model is p_θ(x | c) = 0.5·p(x | c) + 0.5·p(x), so w = 1 leaves many off-prompt samples. Schedule, timestep spacing and img2img start step follow SD v1 with the diffusers DDIM scheduler.
Worked example: one guided DDIM step and img2img strength
| strength | steps run | first timestep | ᾱ_t | √ᾱ_t (input kept) | √(1 − ᾱ_t) (noise) |
|---|---|---|---|---|---|
| 0.5 | 25 | 481 | 0.3023 | 0.550 | 0.835 |
| 0.75 | 37 | 721 | 0.0705 | 0.266 | 0.964 |
| 0.8 (diffusers default) | 40 | 781 | 0.0435 | 0.208 | 0.978 |
| 1.0 | 50 | 981 | 0.0058 | 0.076 | 0.997 |
Two latent coordinates through the first step of a 50-step run
- 1At , . Let , and .
- 2Guidance with : . The prompt-specific difference is amplified 7.5 times.
- 3Clean-latent guess: , far outside the unit-variance range of real latents (it is with ). Early guesses are crude; this overshoot is what becomes oversaturation if is too large.
- 4DDIM to (): . The latent itself moves only a little per step.
- 5After the last step, decode , i.e. multiply the latent by first: .
1import torch23T = 10004betas = torch.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T, dtype=torch.float64) ** 2 # "scaled_linear"5abar = torch.cumprod(1 - betas, 0)6VAE_SCALE = 0.18215 # SD v1/v2 latent scaling factor78def ddim_timesteps(steps): # diffusers DDIM, "leading" spacing, steps_offset = 19 return (torch.arange(steps) * (T // steps) + 1).flip(0)1011@torch.no_grad()12def sample(unet, cond, uncond, steps=50, w=7.5, init_latents=None, strength=0.8, generator=None):13 """cond / uncond: (B, 77, 768) CLIP ViT-L/14 hidden states of the prompt and of "" (or a negative prompt)."""14 ts = ddim_timesteps(steps)15 B = cond.shape[0]16 noise = torch.randn(B, 4, 64, 64, generator=generator, dtype=torch.float64)17 if init_latents is None: # txt2img: x_T ~ N(0, I)18 x = noise19 else: # img2img (SDEdit): noise the image latents to ts[0]20 n_run = min(int(steps * strength), steps)21 ts = ts[steps - n_run:]22 a = abar[ts[0]]23 x = a.sqrt() * init_latents + (1 - a).sqrt() * noise24 for i, t in enumerate(ts):25 e_u, e_c = unet(torch.cat([x, x]), t, torch.cat([uncond, cond])).chunk(2) # one batched call26 e = e_u + w * (e_c - e_u) # classifier-free guidance27 a_t = abar[t]28 a_prev = abar[ts[i + 1]] if i + 1 < len(ts) else abar[0] # set_alpha_to_one = False29 x0 = (x - (1 - a_t).sqrt() * e) / a_t.sqrt()30 x = a_prev.sqrt() * x0 + (1 - a_prev).sqrt() * e # DDIM, eta = 031 return x / VAE_SCALE # pass to the VAE decoder -> (B, 3, 512, 512)3233calls = []34def toy_unet(x, t, ctx): # stand-in for the 860M-parameter U-Net35 calls.append(x.shape[0])36 return 0.1 * x + ctx.mean((1, 2)).view(-1, 1, 1, 1)3738g = torch.Generator().manual_seed(0)39cond, uncond = torch.randn(1, 77, 768, dtype=torch.float64), torch.zeros(1, 77, 768, dtype=torch.float64)40print(ddim_timesteps(50)[:3].tolist(), ddim_timesteps(50)[-1].item()) # [981, 961, 941] 141z = sample(toy_unet, cond, uncond, generator=g)42print(z.shape, len(calls), sum(calls)) # torch.Size([1, 4, 64, 64]) 50 10043calls.clear()44z = sample(toy_unet, cond, uncond, init_latents=torch.randn(1, 4, 64, 64, dtype=torch.float64), strength=0.75, generator=g)45print(len(calls), f"{abar[721].item():.4f}") # 37 0.0705 (37 steps from t = 721)46print(f"{abar[999].item():.5f}") # 0.00466: the last training step is not pure noiseUse when
- +Open weights; about 1B parameters in total, so it runs and fine-tunes on consumer GPUs
- +One model for txt2img, img2img, inpainting (with the 9-channel variant) and, through add-ons, depth, pose or edge control
- +Any ε-prediction sampler applies; 20–50 steps with modern solvers
- +Modular: VAE, text encoder, U-Net and scheduler can be swapped or fine-tuned separately
Avoid when
- −CLIP ViT-L/14 text conditioning handles counting, spatial relations and rendered text poorly (improved in SDXL and SD 3 with larger and T5 encoders)
- −Quality capped by the f = 8, 4-channel VAE: small faces, hands and text are hard
- −Needs classifier-free guidance (two evaluations per step) and has a non-zero terminal SNR
- −Trained on web-scraped LAION data: copyright, consent and bias issues, and a safety filter that is easy to remove
Check yourself
With 50 DDIM steps and img2img strength 0.8, what does the diffusers pipeline do?
Check yourself
In SD v1, what is the role of the factor 0.18215?
Remember
- ✓SD v1 = LDM: KL-VAE (, latents ), 860M U-Net with cross-attention to frozen CLIP ViT-L/14 states (123M), trained on LAION-5B subsets.
- ✓Classifier-free guidance: , ; enabled by dropping the caption 10% of the time; costs two U-Net passes per step.
- ✓Schedule: , scaled-linear from 0.00085 to 0.012, -prediction; samplers PLMS, DDIM, Euler, DPM-Solver++ in 20–50 steps.
- ✓img2img: noise the input latent to the timestep at fraction strength of the schedule and run steps; inpainting model: 9-channel U-Net input (latent, masked-image latent, mask).
- ✓SD 2: OpenCLIP ViT-H/14, 768² v-prediction; SDXL: 2.6B U-Net, two text encoders, 1024²; SD 3: MMDiT with rectified flow, three text encoders, 16-channel VAE.