# Stable Diffusion

> Stable Diffusion is the openly released latent diffusion model: an f = 8 VAE maps 512×512 images to 64×64×4 latents (scaled by 0.18215), and an 860M U-Net denoises them with cross-attention to a frozen CLIP ViT-L/14 text encoder. It samples with classifier-free guidance (scale ~7.5) in 20–50 steps.

Source: https://www.mlfoundry.in/learn/stable-diffusion  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**Stable Diffusion:** A family of text-to-image **latent diffusion models** whose weights were released publicly, starting in August 2022 by CompVis (LMU Munich), Stability AI and Runway, under the CreativeML OpenRAIL-M licence. Version 1 is the recipe of **Latent diffusion models** at scale. A KL-regularised autoencoder with downsampling factor $f=8$ maps a $512\times512\times3$ image to a $64\times64\times4$ latent. An 860M-parameter U-Net predicts the noise in that latent, conditioned through cross-attention on the 77-token hidden states of a frozen CLIP ViT-L/14 text encoder (123M parameters). It was trained on subsets of LAION-5B, and it samples with **classifier-free guidance** and fast solvers such as PNDM/PLMS, DDIM or DPM-Solver.

Rombach et al.'s CVPR 2022 paper had shown a 1.45B text-to-image LDM trained on LAION-400M with its own transformer text encoder. Stable Diffusion kept the architecture, replaced the text encoder by OpenAI's pretrained CLIP ViT-L/14 (kept frozen), and trained on far more data at 512×512. Because the whole model, about 1.07B parameters, fits in under 10 GB of GPU memory at inference, it ran on consumer GPUs. The open weights made it the base for an ecosystem of fine-tunes, LoRA adapters, ControlNet and img2img and inpainting tools. Later versions changed the text encoder (SD 2), scaled the U-Net (SDXL), and finally replaced the U-Net and the diffusion objective (SD 3).

## The three networks of SD v1

**SD v1.x components (model card and v1-inference.yaml)**

- **VAE (stage 1):** KL-regularised autoencoder, $f=8$, $c=4$, about 84M parameters, LPIPS + patch-GAN training; $512^2\times3\to64^2\times4$, 48 pixel values per latent value
- **Latent scale:** $z=0.18215\cdot\mathcal E(x)$ before diffusion and $x=\mathcal D(z/0.18215)$ after; the factor gives the latents roughly unit variance
- **U-Net (stage 2):** 860M parameters; base width 320 channels, multipliers (1, 2, 4, 4), 2 ResNet blocks per level; transformer blocks (self-attention, cross-attention, feed-forward, 8 heads) at the 64², 32² and 16² latent resolutions; context dimension 768
- **Text encoder:** frozen CLIP ViT-L/14 text transformer, 123M parameters; the prompt is padded or truncated to 77 tokens and the final hidden states $\tau(y)\in\mathbb R^{77\times768}$ are the cross-attention keys and values
- **Noise schedule:** $T=1000$, "scaled linear" $\beta_t$: $\sqrt{\beta_t}$ linear from $\sqrt{0.00085}$ to $\sqrt{0.012}$; $\epsilon$-prediction
- **Training:** LAION-5B subsets: first laion2B-en at 256², then high-resolution and aesthetic-filtered subsets at 512²; 32×8 A100 GPUs, AdamW, batch 2048, learning rate warmed up to $10^{-4}$ over 10,000 steps, about 150,000 A100-hours; v1-3 and later drop the text 10% of the time for classifier-free guidance

Training is the LDM objective $\|\epsilon-\epsilon_\theta(z_t,t,\tau(y))\|^2$ with the text encoder frozen, on latents of LAION images and their alt-text captions. Dropping the caption, so that $\tau(\varnothing)$ is the encoding of the empty prompt, teaches the same U-Net an **unconditional** model. That is what classifier-free guidance needs.

## Classifier-free guidance

$$
\tilde\epsilon_\theta(z_t,t,y)=\epsilon_\theta(z_t,t,\varnothing)+w\,\big(\epsilon_\theta(z_t,t,y)-\epsilon_\theta(z_t,t,\varnothing)\big)\quad\Longleftrightarrow\quad \nabla\log\tilde p_t(z\mid y)=\nabla\log p_t(z\mid y)+(w-1)\,\nabla\log p_t(y\mid z)
$$

- $w$: guidance scale (`guidance_scale` in diffusers, 7.5 by default for SD); $w=1$ is the plain conditional model, $w=0$ the unconditional one
- $\varnothing$: the empty prompt; replacing it by a **negative prompt** pushes samples away from that text
- $p_t(y\mid z)$: an implicit classifier, $p(y\mid z)\propto p(z\mid y)/p(z)$; guidance raises it to the power $w-1$

Ho and Salimans (NeurIPS 2021 workshop, "Classifier-Free Diffusion Guidance") introduced this as a replacement for classifier guidance: no separate noisy-image classifier, just two evaluations of the same network. In practice the conditional and unconditional inputs are batched, so a 50-step sample costs 100 U-Net evaluations in 50 batched calls. Larger $w$ gives images that match the prompt better and have more contrast, but are less diverse. Very large $w$ oversaturates colours, because the guided prediction $\hat z_0$ leaves the range of real latents. With the text dropped only 10% of the time and a weak text–image coupling, $w=1$ alone produces images that follow the prompt poorly, which is why SD is almost always used with $w$ around 5 to 10.

## Sampling, img2img and inpainting

$$
\hat z_0=\frac{z_t-\sqrt{1-\bar\alpha_t}\,\tilde\epsilon}{\sqrt{\bar\alpha_t}},\qquad z_{t'}=\sqrt{\bar\alpha_{t'}}\,\hat z_0+\sqrt{1-\bar\alpha_{t'}}\,\tilde\epsilon\qquad(\text{DDIM},\ \eta=0,\ t'<t)
$$

- $\hat z_0$: the current guess of the clean latent; decoding it at intermediate steps gives the familiar "preview" images
- $t'$: the next timestep of the solver's schedule; with 50 steps and diffusers' "leading" spacing the timesteps are 981, 961, …, 21, 1

Any sampler for an $\epsilon$-prediction model works, and the released scripts and libraries exposed many. PLMS/PNDM (pseudo-numerical methods: the original repository's example command used PLMS with 50 steps, and PNDM is the default scheduler of the diffusers v1 pipelines), DDIM, Euler and Euler-ancestral, DPM-Solver++ (good results in 20–25 steps) and Karras-style variants all solve the same probability-flow ODE or reverse SDE (see **Score-based generative models** and **Denoising diffusion implicit models (DDIM)**). In diffusers a call looks like `StableDiffusionPipeline.from_pretrained(...)(prompt, num_inference_steps=50, guidance_scale=7.5)`.

**img2img** is SDEdit (Meng et al., ICLR 2022) in latent space. Encode an input image to $z^{\mathrm{init}}$, noise it to an intermediate timestep, $z_t=\sqrt{\bar\alpha_t}\,z^{\mathrm{init}}+\sqrt{1-\bar\alpha_t}\,\epsilon$, and run only the remaining steps with the new prompt. The **strength** $s\in[0,1]$ sets how far back to go. With $S$ solver steps, $\lfloor sS\rfloor$ steps are run, starting at the matching point of the schedule. Low strength keeps the layout and colours of the input, and strength 1 ignores it almost entirely. **Inpainting** can be done the same way by re-imposing the known region at every step. The dedicated SD inpainting checkpoint instead has a U-Net with 9 input channels: the 4 noisy latent channels, the 4 channels of the masked image's latent, and 1 downsampled mask. It is fine-tuned so that it sees the context directly.

Main Stable Diffusion generations (sizes as published; SD3 sizes are of the family)

| version | denoiser | text encoder(s) | native resolution | notes |
| --- | --- | --- | --- | --- |
| SD 1.x (2022) | U-Net, 860M | CLIP ViT-L/14 (frozen) | 512² | v1-1 … v1-4 by CompVis/Stability, v1-5 by Runway |
| SD 2.x (Nov–Dec 2022) | U-Net, similar size | OpenCLIP ViT-H/14 (penultimate layer) | 512² and 768² | 768² model uses v-prediction; also depth-to-image and ×4 upscaler |
| SDXL (Podell et al. 2023) | U-Net, 2.6B | OpenCLIP ViT-bigG/14 + CLIP ViT-L/14 | 1024² | size and crop conditioning, multi-aspect training, optional refiner model |
| SD 3 (Esser et al., ICML 2024) | MMDiT transformer, up to 8B | CLIP-L + OpenCLIP bigG + T5-XXL | 1024² | rectified-flow objective, 16-channel VAE, joint text–image attention |

SD 3 is a break with the LDM U-Net. Its **MMDiT** keeps separate weights for text tokens and image-patch tokens but lets both attend jointly in each block, so text representations are updated too. It is trained with the **Rectified flow** objective (see **Flow matching**), with timesteps sampled from a logit-normal distribution that emphasises intermediate noise levels.

> Figure: A 2-D stand-in for SD's latent space. Three prompts each own two Gaussian modes (std 0.18). The toy model follows the prompt only halfway, p_θ(x | c) = 0.5·p(x | c) + 0.5·p(x), and its ε is exact. Sampling uses SD v1's schedule (scaled-linear β 0.00085 → 0.012, ᾱ₉₉₉ = 0.00466) and diffusers' DDIM with 50 steps from t = 981 to 1, 160 samples, seed 1. For “a dog”, the share of samples on the prompt is 29.4% at w = 0, 60.6% at w = 1, 95.6% at w = 3 and 99.4% at w = 7.5, at the cost of 100 instead of 50 U-Net evaluations. Diversity goes the other way. At w = 1 the two dog modes get 50 and 47 samples, at w = 7.5 only 2 and 157, because guidance pushes away from the cat and car prompts and the dog mode nearest them almost vanishes. At w = 15 all 160 samples sit in one mode, spread 1.56× wider than the data. img2img from cat latents with w = 7.5: strength 0.2 runs 10 steps from t = 181 (√ᾱ = 0.885 of the input kept) and leaves 70.6% of samples as cats. Strength 0.5 runs 25 steps from t = 481 and reaches 95.0% dogs, and strength 0.75 runs 37 steps from t = 721 (ᾱ = 0.0705) and reaches 98.8%.

## Worked example: one guided DDIM step and img2img strength

Where img2img starts with 50 DDIM steps (SD v1 schedule, computed in Node and PyTorch)

| strength | steps run | first timestep | ᾱ_t | √ᾱ_t (input kept) | √(1 − ᾱ_t) (noise) |
| --- | --- | --- | --- | --- | --- |
| 0.5 | 25 | 481 | 0.3023 | 0.550 | 0.835 |
| 0.75 | 37 | 721 | 0.0705 | 0.266 | 0.964 |
| 0.8 (diffusers default) | 40 | 781 | 0.0435 | 0.208 | 0.978 |
| 1.0 | 50 | 981 | 0.0058 | 0.076 | 0.997 |

**Two latent coordinates through the first step of a 50-step run**

1. At $t=981$, $\bar\alpha_{981}=0.005776$. Let $z_t=(1.0,\,-0.5)$, $\epsilon_\theta(z_t,t,\varnothing)=(0.2,\,-0.1)$ and $\epsilon_\theta(z_t,t,y)=(0.5,\,0.3)$.
2. Guidance with $w=7.5$: $\tilde\epsilon=(0.2+7.5\cdot0.3,\ -0.1+7.5\cdot0.4)=(2.45,\ 2.90)$. The prompt-specific difference is amplified 7.5 times.
3. Clean-latent guess: $\hat z_0=(z_t-\sqrt{1-0.005776}\,\tilde\epsilon)/\sqrt{0.005776}=(-18.99,\,-44.63)$, far outside the unit-variance range of real latents (it is $(6.60,\,-10.52)$ with $w=1$). Early guesses are crude; this overshoot is what becomes oversaturation if $w$ is too large.
4. DDIM to $t'=961$ ($\bar\alpha_{961}=0.007282$): $z_{t'}=\sqrt{0.007282}\,\hat z_0+\sqrt{0.992718}\,\tilde\epsilon=(0.821,\,-0.919)$. The latent itself moves only a little per step.
5. After the last step, decode $x=\mathcal D(z_0/0.18215)$, i.e. multiply the latent by $5.49$ first: $64\times64\times4\to512\times512\times3$.

```python
import torch

T = 1000
betas = torch.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T, dtype=torch.float64) ** 2   # "scaled_linear"
abar = torch.cumprod(1 - betas, 0)
VAE_SCALE = 0.18215                                   # SD v1/v2 latent scaling factor

def ddim_timesteps(steps):                            # diffusers DDIM, "leading" spacing, steps_offset = 1
    return (torch.arange(steps) * (T // steps) + 1).flip(0)

@torch.no_grad()
def sample(unet, cond, uncond, steps=50, w=7.5, init_latents=None, strength=0.8, generator=None):
    """cond / uncond: (B, 77, 768) CLIP ViT-L/14 hidden states of the prompt and of "" (or a negative prompt)."""
    ts = ddim_timesteps(steps)
    B = cond.shape[0]
    noise = torch.randn(B, 4, 64, 64, generator=generator, dtype=torch.float64)
    if init_latents is None:                          # txt2img: x_T ~ N(0, I)
        x = noise
    else:                                             # img2img (SDEdit): noise the image latents to ts[0]
        n_run = min(int(steps * strength), steps)
        ts = ts[steps - n_run:]
        a = abar[ts[0]]
        x = a.sqrt() * init_latents + (1 - a).sqrt() * noise
    for i, t in enumerate(ts):
        e_u, e_c = unet(torch.cat([x, x]), t, torch.cat([uncond, cond])).chunk(2)   # one batched call
        e = e_u + w * (e_c - e_u)                     # classifier-free guidance
        a_t = abar[t]
        a_prev = abar[ts[i + 1]] if i + 1 < len(ts) else abar[0]   # set_alpha_to_one = False
        x0 = (x - (1 - a_t).sqrt() * e) / a_t.sqrt()
        x = a_prev.sqrt() * x0 + (1 - a_prev).sqrt() * e          # DDIM, eta = 0
    return x / VAE_SCALE                              # pass to the VAE decoder -> (B, 3, 512, 512)

calls = []
def toy_unet(x, t, ctx):                              # stand-in for the 860M-parameter U-Net
    calls.append(x.shape[0])
    return 0.1 * x + ctx.mean((1, 2)).view(-1, 1, 1, 1)

g = torch.Generator().manual_seed(0)
cond, uncond = torch.randn(1, 77, 768, dtype=torch.float64), torch.zeros(1, 77, 768, dtype=torch.float64)
print(ddim_timesteps(50)[:3].tolist(), ddim_timesteps(50)[-1].item())   # [981, 961, 941] 1
z = sample(toy_unet, cond, uncond, generator=g)
print(z.shape, len(calls), sum(calls))                # torch.Size([1, 4, 64, 64]) 50 100
calls.clear()
z = sample(toy_unet, cond, uncond, init_latents=torch.randn(1, 4, 64, 64, dtype=torch.float64), strength=0.75, generator=g)
print(len(calls), f"{abar[721].item():.4f}")          # 37 0.0705   (37 steps from t = 721)
print(f"{abar[999].item():.5f}")                      # 0.00466: the last training step is not pure noise
```

The SD v1 sampling loop: scaled-linear schedule, diffusers-style DDIM timesteps, batched classifier-free guidance and img2img strength (a stand-in U-Net; run with PyTorch on CPU)

> **The schedule never reaches pure noise:** With SD's scaled-linear schedule $\bar\alpha_{999}=0.00466$, so the training data at the last step still has signal-to-noise ratio $0.0047$ (about 7% of the latent's amplitude, $\sqrt{\bar\alpha}=0.068$). Sampling starts from pure $\mathcal N(0,I)$, a slight train–test mismatch. Lin et al. ("Common Diffusion Noise Schedules and Sample Steps are Flawed", WACV 2024) traced SD's difficulty with very dark or very bright images to it. The "leading" spacing adds a second quirk. With 10 steps the first timestep is 901, so the sampler never starts from the noisiest level at all.

> **Guidance scale is a trade, not a quality knob:** Raising $w$ improves prompt adherence and lowers diversity. In the figure the share on the prompt goes from 60.6% at $w=1$ to 99.4% at $w=7.5$, while one of the two dog modes falls from 50 samples to 2. Past roughly 10–15, images oversaturate and lose detail. FID against real images is typically best at small $w$ (the LDM paper used 1.5) even though people prefer the look at higher $w$.

> **Why the latent space made this release possible:** At $f=8$ the U-Net sees a $64\times64$ grid, 4096 positions. Self-attention there costs 16.8M scores per head, against $6.9\times10^{10}$ in pixel space. Training on hundreds of millions of image–text pairs at 512² and sampling in seconds on a gaming GPU both depend on this compression (see **Latent diffusion models**). The VAE runs once per image, not once per step.

> **Swapping parts:** Each component can be replaced on its own. Fine-tuned VAE decoders (e.g. the improved "ft-MSE" decoder) sharpen faces and text without retraining the U-Net, because the latent space is unchanged. Changing the text encoder (SD 2) or the latent channels (SD 3) breaks compatibility: LoRAs, embeddings and ControlNets trained for SD 1.x do not transfer.

**Strengths**

- Open weights; about 1B parameters in total, so it runs and fine-tunes on consumer GPUs
- One model for txt2img, img2img, inpainting (with the 9-channel variant) and, through add-ons, depth, pose or edge control
- Any ε-prediction sampler applies; 20–50 steps with modern solvers
- Modular: VAE, text encoder, U-Net and scheduler can be swapped or fine-tuned separately

**Limitations**

- CLIP ViT-L/14 text conditioning handles counting, spatial relations and rendered text poorly (improved in SDXL and SD 3 with larger and T5 encoders)
- Quality capped by the f = 8, 4-channel VAE: small faces, hands and text are hard
- Needs classifier-free guidance (two evaluations per step) and has a non-zero terminal SNR
- Trained on web-scraped LAION data: copyright, consent and bias issues, and a safety filter that is easy to remove

**Question:** With 50 DDIM steps and img2img strength 0.8, what does the diffusers pipeline do?

- A. Runs all 50 steps from pure noise and blends 80% of the result with the input
- B. Noises the input latent to the 11th timestep of the schedule (t = 781) and runs the remaining 40 steps
- C. Runs 8 steps
- D. Multiplies the input latent by 0.8 and runs 50 steps

**Answer:** B. Noises the input latent to the 11th timestep of the schedule (t = 781) and runs the remaining 40 steps — $\lfloor0.8\cdot50\rfloor=40$ steps are run, starting at index $50-40=10$ of $[981,961,\dots,1]$, which is $t=781$. There $\sqrt{\bar\alpha_{781}}=0.208$ of the input latent survives and $\sqrt{1-\bar\alpha}=0.978$ is fresh noise, enough to change content while keeping the coarse layout.

**Question:** In SD v1, what is the role of the factor 0.18215?

- A. It is the classifier-free guidance scale divided by 41
- B. It is the learning rate
- C. It rescales the VAE latents to roughly unit variance before diffusion (and is undone before decoding)
- D. It is the fraction of captions dropped during training

**Answer:** C. It rescales the VAE latents to roughly unit variance before diffusion (and is undone before decoding) — LDM rescales KL-autoencoder latents by one scalar estimated from data so the noise schedule's signal-to-noise ratios are meaningful. For SD's VAE it is 0.18215: encode $z=0.18215\,\mathcal E(x)$, decode $\mathcal D(z/0.18215)$. Captions were dropped 10% of the time, and guidance is about 7.5.

**Key points**

- SD v1 = LDM: $f=8$ KL-VAE ($512^2\times3\to64^2\times4$, latents $\times0.18215$), 860M U-Net with cross-attention to frozen CLIP ViT-L/14 states $\tau(y)\in\mathbb R^{77\times768}$ (123M), trained on LAION-5B subsets.
- Classifier-free guidance: $\tilde\epsilon=\epsilon(z_t,\varnothing)+w(\epsilon(z_t,y)-\epsilon(z_t,\varnothing))$, $w\approx7.5$; enabled by dropping the caption 10% of the time; costs two U-Net passes per step.
- Schedule: $T=1000$, scaled-linear $\beta$ from 0.00085 to 0.012, $\epsilon$-prediction; samplers PLMS, DDIM, Euler, DPM-Solver++ in 20–50 steps.
- img2img: noise the input latent to the timestep at fraction strength of the schedule and run $\lfloor\text{strength}\cdot S\rfloor$ steps; inpainting model: 9-channel U-Net input (latent, masked-image latent, mask).
- SD 2: OpenCLIP ViT-H/14, 768² v-prediction; SDXL: 2.6B U-Net, two text encoders, 1024²; SD 3: MMDiT with rectified flow, three text encoders, 16-channel VAE.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
