# Text-to-image generation

> A text-to-image call encodes the prompt, draws seeded latent noise and runs 20–50 denoising steps, each mixing a conditional and a negative-prompt prediction by classifier-free guidance (w ≈ 5–8), then decodes the latent to pixels.

Source: https://www.mlfoundry.in/learn/text-to-image-generation  
Section: Generative AI › Multimodal Generative AI  
Level: core · 8 min read · updated 2026-10-21

**Text-to-image generation:** produces an image from a natural-language prompt. Current open systems are conditional **latent diffusion models**: a frozen text encoder turns the prompt into embeddings, a denoising network refines random latent noise over a fixed number of steps while attending to those embeddings, and a VAE decoder maps the result to pixels. Almost all of them steer the denoiser with **classifier-free guidance** (Ho and Salimans, "Classifier-Free Diffusion Guidance", NeurIPS 2021 workshop, arXiv 2022).

This article covers the generation call as a practitioner sees it: the prompt, the negative prompt, the seed, the step count, the scheduler and the guidance scale, and what each one costs. How the denoiser is trained and why the sampler works are covered in **Diffusion models**, **Latent diffusion models** and **Stable Diffusion**. Editing an existing image is in **Image editing and inpainting**, and spatial control in **ControlNet**.

In a product the model sits behind a request with a handful of parameters. Every one of them changes the output, so reproducing an image requires all of them: the same weights, prompt, negative prompt, seed, size, step count, scheduler and guidance scale. Most GPU stacks are deterministic for a fixed setup, but changing the GPU type or library version can change pixels slightly.

## The pipeline

1. **Tokenize and encode.** Stable Diffusion v1 uses CLIP's BPE tokenizer and pads or truncates the prompt to 77 tokens (75 of text plus begin and end markers). Its frozen CLIP ViT-L/14 text encoder returns a $77\times768$ matrix of hidden states, not a single vector (see **CLIP**).
2. **Seed the noise.** A seeded random generator fills the initial latent $z_T\sim\mathcal N(0,I)$. At $512\times512$ it is $4\times64\times64$, 48 times fewer values than the RGB image.
3. **Denoise.** For each of $N$ timesteps the U-Net (or, in newer models, a diffusion transformer) predicts the noise in $z_t$, attending to the text through **Cross-attention**. Guidance combines a conditional and an unconditional prediction, and the scheduler turns the result into $z_{t-\Delta}$.
4. **Decode.** The VAE decoder maps $z_0$ (divided by SD's latent scale 0.18215) back to $512\times512\times3$ pixels. Safety checkers or watermarking may run afterwards.

> Figure: One Stable Diffusion v1 call at 512 × 512 with 30 steps and guidance 7.5. The prompt and negative prompt become two 77 × 768 embedding matrices. Seed 42 fixes the 4 × 64 × 64 starting noise. Each of the 30 loop iterations evaluates the U-Net on a batch of two (negative and positive conditioning), mixes the predictions with classifier-free guidance and takes one scheduler step, so the call costs 60 U-Net evaluations and one VAE decode.

## Guidance and the negative prompt

During training the caption is replaced by the empty prompt a small fraction of the time (10% for SD v1.3 onwards), so one network learns both $\epsilon_\theta(z_t,t,c)$ and $\epsilon_\theta(z_t,t,\varnothing)$. At sampling time the two are extrapolated:

$$
\tilde\epsilon=\epsilon_\theta(z_t,t,c_{\text{neg}})+w\,\big(\epsilon_\theta(z_t,t,c)-\epsilon_\theta(z_t,t,c_{\text{neg}})\big),\qquad c_{\text{neg}}=\varnothing\ \text{by default}
$$

- $c$: embedding of the prompt
- $c_{\text{neg}}$: embedding of the negative prompt; with none given, the embedding of the empty string $\varnothing$
- $w$: guidance scale (`guidance_scale` in diffusers, default 7.5 for SD v1); $w=1$ is plain conditional sampling and needs only one evaluation per step

So a **negative prompt** is not a filter. It replaces the empty prompt as the reference point, and guidance pushes every step away from it along $\epsilon_c-\epsilon_{\text{neg}}$. That is why "blurry, low quality, extra fingers" can help, and why a negative prompt that shares words with the prompt weakens the prompt. Larger $w$ raises prompt adherence and contrast and lowers diversity. Very large $w$ oversaturates colours because the guided prediction leaves the range of real latents. Imagen (Saharia et al., 2022) added **dynamic thresholding** to clip the predicted image and make large weights usable.

## Steps and schedulers

The scheduler (the "sampler" in UIs) decides which timesteps are visited and how a noise prediction becomes the next latent. The deterministic DDIM update, which the figure below uses, is:

$$
\hat z_0=\frac{z_t-\sqrt{1-\bar\alpha_t}\,\tilde\epsilon}{\sqrt{\bar\alpha_t}},\qquad z_{t'}=\sqrt{\bar\alpha_{t'}}\,\hat z_0+\sqrt{1-\bar\alpha_{t'}}\,\tilde\epsilon
$$

- $\bar\alpha_t$: fraction of signal variance left at timestep $t$; SD v1 uses a scaled-linear schedule with $\beta$ from 0.00085 to 0.012 over 1,000 steps
- $t'$: the next timestep; 50 steps visit $t=981,961,\dots,1$ in diffusers' leading spacing
- $\hat z_0$: the current guess of the clean latent; previews in UIs decode this

Deterministic ODE solvers (DDIM, Euler, DPM-Solver++, Heun) map one seed to nearly the same image at 25 or at 100 steps; more steps only refine it. **Ancestral** samplers (Euler a, DDPM) add fresh noise at every step, so the image keeps changing with the step count. DPM-Solver++ (Lu et al., 2022) gives good results in about 20–25 steps. Distilled models need far fewer: **Consistency models** and adversarial distillation (SDXL-Turbo, Sauer et al., 2023) sample in 1–4 steps and usually run without guidance.

**Reference systems (as published)**

- **Stable Diffusion v1 (2022):** 860M-parameter U-Net, 64 × 64 × 4 latent for 512 × 512, frozen CLIP ViT-L/14 text encoder (77 tokens × 768)
- **SDXL (Podell et al., 2023):** 2.6B-parameter U-Net, two text encoders (OpenCLIP ViT-bigG/14 and CLIP ViT-L/14), 1024 × 1024 base resolution, optional refiner model
- **Imagen (Saharia et al., 2022):** frozen T5-XXL text encoder; scaling the text encoder helped more than scaling the image U-Net; zero-shot MS-COCO FID 7.27
- **DALL·E 2 (Ramesh et al., 2022):** a prior maps the CLIP text embedding to a CLIP image embedding, and a diffusion decoder generates from it; MS-COCO FID 10.39
- **diffusers defaults:** `StableDiffusionPipeline`: `num_inference_steps=50`, `guidance_scale=7.5`; guidance is skipped when `guidance_scale` ≤ 1

> Figure: A toy 2-D latent space with four kinds of image, an exact noise predictor that follows the prompt only halfway, SD v1's schedule and 64 seeds. At the defaults (prompt “a dog”, w = 7.5, 50 DDIM steps, no negative prompt) all 64 samples are dogs, but 27 are blurry (57.8% sharp), at 100 U-Net evaluations. Add the negative prompt “blurry” and 63 of 64 are sharp dogs (98.4%), with spread 1.35× the data's; at w = 15 the spread reaches 2.18× (oversaturation). At w = 1 only 68.8% are dogs, for 50 evaluations. Seeds move 0.011 on average between 50 and 100 DDIM steps (0.084 at 10 steps), but 0.485 with the ancestral sampler.

The figure separates the knobs. The **seed** picks the starting point, so the same seed always lands in the same place. **Guidance** decides how hard the sample is pulled towards the prompt. The **negative prompt** changes the direction of that pull, here away from the blurry half of the space. **Steps** change little once a deterministic solver has converged, while an ancestral sampler lands elsewhere for every step count.

## Worked example

At one step, let the negative-prompt prediction be $\epsilon_{\text{neg}}=(0.10,-0.20)$ and the prompt prediction $\epsilon_c=(0.30,0.20)$ on two latent coordinates. The difference is $(0.20,0.40)$, and with $w=7.5$: $\tilde\epsilon=(0.10+7.5\cdot0.20,\ -0.20+7.5\cdot0.40)=(1.60,\ 2.80)$. With $w=1$ it would be $\epsilon_c=(0.30,0.20)$. Cost depends on steps and on whether guidance runs:

Denoiser evaluations per image (the VAE decode is extra and runs once)

| Setting | Steps | Guidance | Evaluations |
| --- | --- | --- | --- |
| SD v1 diffusers default | 50 | w = 7.5 | 50 × 2 = 100 |
| DPM-Solver++ | 25 | w = 7.5 | 25 × 2 = 50 |
| DPM-Solver++, no guidance | 25 | w = 1 | 25 |
| distilled 1-step model | 1 | none | 1 |

```python
import torch
from diffusers import AutoPipelineForText2Image, DPMSolverMultistepScheduler

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
).to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

g = torch.Generator("cuda").manual_seed(42)        # fixes the initial latent noise
image = pipe(
    prompt="a corgi in a yellow raincoat, watercolor",
    negative_prompt="blurry, low quality",          # replaces the empty prompt in CFG
    num_inference_steps=25,
    guidance_scale=7.0,
    height=1024, width=1024,                        # SDXL's training resolution
    generator=g,
).images[0]
image.save("corgi_seed42.png")                     # log every argument with the file
```

A reproducible SDXL call with Hugging Face diffusers (needs a GPU and the downloaded weights; not run here)

> **Silent truncation and off-size generation:** CLIP-based models see only the first 77 tokens; anything after that is dropped (diffusers logs a warning, many UIs do not). Asking for a size far from the training resolution, such as $1024\times1024$ from SD v1.5, often duplicates subjects or breaks composition, because the denoiser never saw that latent size. Generate near the native size and upscale afterwards.

> **Prompting the encoder, not a person:** The text encoder was trained on alt-text captions, so short, concrete descriptions of subject, medium, style and lighting work better than instructions ("draw me..."). Word order matters too: CLIP's text encoder uses a causal attention mask, so each token's hidden state depends only on the tokens before it. A personal subject or style that words cannot describe is better taught with **DreamBooth** or **Textual inversion**. Measure prompt adherence with **CLIP score** and image quality with **FID and Inception Score**.

**Question:** A user adds the negative prompt “dog” to the prompt “a cat”. What does the sampler actually do?

- A. Rejects finished images that contain a dog
- B. Uses ε(z, t, “dog”) instead of ε(z, t, ∅) as the guidance reference, pushing each step away from dog-like latents
- C. Subtracts the word “dog” from the prompt tokens
- D. Doubles the number of denoising steps

**Answer:** B. Uses ε(z, t, “dog”) instead of ε(z, t, ∅) as the guidance reference, pushing each step away from dog-like latents — The negative prompt replaces the empty-prompt branch of classifier-free guidance: $\tilde\epsilon=\epsilon_{\text{neg}}+w(\epsilon_c-\epsilon_{\text{neg}})$. The cost stays at two evaluations per step.

**Question:** You found a good image at seed 7 with 30 steps of Euler a (ancestral). You rerun it at 60 steps for more detail. What happens?

- A. The same image with finer detail
- B. Usually a noticeably different image, because ancestral samplers inject fresh noise at every step
- C. An error, because the seed only supports 30 steps
- D. The same image, but twice as fast

**Answer:** B. Usually a noticeably different image, because ancestral samplers inject fresh noise at every step — Ancestral samplers draw new noise each step, so changing the step count changes the noise sequence. Deterministic solvers (DDIM, DPM-Solver++, Euler) converge to nearly the same image. In the figure the mean shift between 50 and 100 steps is 0.011 for DDIM and 0.485 for the ancestral sampler.

**Key points**

- Pipeline: tokenize (77 tokens for CLIP) → frozen text encoder → seeded latent noise → N denoising steps with cross-attention → VAE decode.
- Classifier-free guidance: $\tilde\epsilon=\epsilon_{\text{neg}}+w(\epsilon_c-\epsilon_{\text{neg}})$; typical $w$ 5–8, two evaluations per step.
- The negative prompt replaces the empty prompt as the guidance reference; it is a direction, not a filter.
- Deterministic solvers make a seed stable across step counts; ancestral samplers do not.
- Reproducibility needs weights, prompt, negative prompt, seed, size, steps, scheduler and guidance scale together.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
