Text-to-image generation

A text-to-image call encodes the prompt, draws seeded latent noise and runs 20–50 denoising steps, each mixing a conditional and a negative-prompt prediction by classifier-free guidance (w ≈ 5–8), then decodes the latent to pixels.

8 min read910 wordscore0 viewsupdated 2026-10-21

Definition

Text-to-image generation produces an image from a natural-language prompt. Current open systems are conditional latent diffusion models: a frozen text encoder turns the prompt into embeddings, a denoising network refines random latent noise over a fixed number of steps while attending to those embeddings, and a VAE decoder maps the result to pixels. Almost all of them steer the denoiser with classifier-free guidance (Ho and Salimans, "Classifier-Free Diffusion Guidance", NeurIPS 2021 workshop, arXiv 2022).

This article covers the generation call as a practitioner sees it: the prompt, the negative prompt, the seed, the step count, the scheduler and the guidance scale, and what each one costs. How the denoiser is trained and why the sampler works are covered in Diffusion models, Latent diffusion models and Stable Diffusion. Editing an existing image is in Image editing and inpainting, and spatial control in ControlNet.

In a product the model sits behind a request with a handful of parameters. Every one of them changes the output, so reproducing an image requires all of them: the same weights, prompt, negative prompt, seed, size, step count, scheduler and guidance scale. Most GPU stacks are deterministic for a fixed setup, but changing the GPU type or library version can change pixels slightly.

The pipeline

  1. 1Tokenize and encode. Stable Diffusion v1 uses CLIP's BPE tokenizer and pads or truncates the prompt to 77 tokens (75 of text plus begin and end markers). Its frozen CLIP ViT-L/14 text encoder returns a 77×76877\times768 matrix of hidden states, not a single vector (see CLIP).
  2. 2Seed the noise. A seeded random generator fills the initial latent zT∼N(0,I)z_T\sim\mathcal N(0,I). At 512×512512\times512 it is 4×64×644\times64\times64, 48 times fewer values than the RGB image.
  3. 3Denoise. For each of NN timesteps the U-Net (or, in newer models, a diffusion transformer) predicts the noise in ztz_t, attending to the text through Cross-attention. Guidance combines a conditional and an unconditional prediction, and the scheduler turns the result into zt−Δz_{t-\Delta}.
  4. 4Decode. The VAE decoder maps z0z_0 (divided by SD's latent scale 0.18215) back to 512×512×3512\times512\times3 pixels. Safety checkers or watermarking may run afterwards.
One call: 512 × 512, 30 steps, guidance 7.5, seed 42 (Stable Diffusion v1 components)prompta corgi in a yellow raincoatnegative promptblurry, low qualitytokenizerCLIP BPEBOS + text + EOSpadded to 77text encoderCLIP ViT-L/14123M paramsfrozenc77 × 768 hidden statesc_neg77 × 768 (replaces ∅)seed 42z_T ~ N(0, I)4 × 64 × 64 latentdenoising loop × 30: t = 958, 925, …, 1z_t4 × 64 × 64U-Net ε_θ(z_t, t, ·)batch of 2: c_neg, cguidance (CFG)ε_neg + 7.5(ε_c − ε_neg)scheduler stepz_t → z_(t−33), e.g. DDIMcross-attnVAE decoder×8 upsamplez_0 ÷ 0.18215output: 512 × 512 × 3 = 786,432 valueslatent: 16,384 values (48× fewer)cost: 30 steps × 2 branches = 60 U-Net evaluations in 30 batched calls, then 1 VAE decodesame weights + prompt + seed + steps + scheduler + size → same latent noise and the same image (up to GPU nondeterminism)
fig.One Stable Diffusion v1 call at 512 × 512 with 30 steps and guidance 7.5. The prompt and negative prompt become two 77 × 768 embedding matrices. Seed 42 fixes the 4 × 64 × 64 starting noise. Each of the 30 loop iterations evaluates the U-Net on a batch of two (negative and positive conditioning), mixes the predictions with classifier-free guidance and takes one scheduler step, so the call costs 60 U-Net evaluations and one VAE decode.

Guidance and the negative prompt

During training the caption is replaced by the empty prompt a small fraction of the time (10% for SD v1.3 onwards), so one network learns both ϵθ(zt,t,c)\epsilon_\theta(z_t,t,c) and ϵθ(zt,t,∅)\epsilon_\theta(z_t,t,\varnothing). At sampling time the two are extrapolated:

ϵ~=ϵθ(zt,t,cneg)+w (ϵθ(zt,t,c)−ϵθ(zt,t,cneg)),cneg=∅ by default\tilde\epsilon=\epsilon_\theta(z_t,t,c_{\text{neg}})+w\,\big(\epsilon_\theta(z_t,t,c)-\epsilon_\theta(z_t,t,c_{\text{neg}})\big),\qquad c_{\text{neg}}=\varnothing\ \text{by default}
cc
embedding of the prompt
cnegc_{\text{neg}}
embedding of the negative prompt; with none given, the embedding of the empty string ∅\varnothing
ww
guidance scale (guidance_scale in diffusers, default 7.5 for SD v1); w=1w=1 is plain conditional sampling and needs only one evaluation per step

So a negative prompt is not a filter. It replaces the empty prompt as the reference point, and guidance pushes every step away from it along ϵc−ϵneg\epsilon_c-\epsilon_{\text{neg}}. That is why "blurry, low quality, extra fingers" can help, and why a negative prompt that shares words with the prompt weakens the prompt. Larger ww raises prompt adherence and contrast and lowers diversity. Very large ww oversaturates colours because the guided prediction leaves the range of real latents. Imagen (Saharia et al., 2022) added dynamic thresholding to clip the predicted image and make large weights usable.

Steps and schedulers

The scheduler (the "sampler" in UIs) decides which timesteps are visited and how a noise prediction becomes the next latent. The deterministic DDIM update, which the figure below uses, is:

z^0=zt−1−αˉt ϵ~αˉt,zt′=αˉt′ z^0+1−αˉt′ ϵ~\hat z_0=\frac{z_t-\sqrt{1-\bar\alpha_t}\,\tilde\epsilon}{\sqrt{\bar\alpha_t}},\qquad z_{t'}=\sqrt{\bar\alpha_{t'}}\,\hat z_0+\sqrt{1-\bar\alpha_{t'}}\,\tilde\epsilon
αˉt\bar\alpha_t
fraction of signal variance left at timestep tt; SD v1 uses a scaled-linear schedule with β\beta from 0.00085 to 0.012 over 1,000 steps
t′t'
the next timestep; 50 steps visit t=981,961,…,1t=981,961,\dots,1 in diffusers' leading spacing
z^0\hat z_0
the current guess of the clean latent; previews in UIs decode this

Deterministic ODE solvers (DDIM, Euler, DPM-Solver++, Heun) map one seed to nearly the same image at 25 or at 100 steps; more steps only refine it. Ancestral samplers (Euler a, DDPM) add fresh noise at every step, so the image keeps changing with the step count. DPM-Solver++ (Lu et al., 2022) gives good results in about 20–25 steps. Distilled models need far fewer: Consistency models and adversarial distillation (SDXL-Turbo, Sauer et al., 2023) sample in 1–4 steps and usually run without guidance.

Reference systems (as published)

Stable Diffusion v1 (2022)
860M-parameter U-Net, 64 × 64 × 4 latent for 512 × 512, frozen CLIP ViT-L/14 text encoder (77 tokens × 768)
SDXL (Podell et al., 2023)
2.6B-parameter U-Net, two text encoders (OpenCLIP ViT-bigG/14 and CLIP ViT-L/14), 1024 × 1024 base resolution, optional refiner model
Imagen (Saharia et al., 2022)
frozen T5-XXL text encoder; scaling the text encoder helped more than scaling the image U-Net; zero-shot MS-COCO FID 7.27
DALL·E 2 (Ramesh et al., 2022)
a prior maps the CLIP text embedding to a CLIP image embedding, and a diffusion decoder generates from it; MS-COCO FID 10.39
diffusers defaults
StableDiffusionPipeline: num_inference_steps=50, guidance_scale=7.5; guidance is skipped when guidance_scale ≤ 1
prompt “a dog” · w = 7.5 · 50 steps · DDIMdog, sharpdog, blurrycat, sharpcat, blurryline: seed 1, sample 0, from noise (○) to image (◎)where 64 seeds landdog, sharp37dog, blurry27cat, sharp0cat, blurry0on prompt (dog)100.0%sharp dog57.8%spread / data spread0.95U-Net evaluations100mean shift vs 100 steps0.011ε̃ = ε_neg + w·(ε_c − ε_neg), ε_neg = ε(∅)model: p_θ(x | c) = ½ p(x | c) + ½ p(x)SD v1 schedule, exact ε, same seedsspread = RMS distance to mode / (σ√2)
negative promptscheduler
fig.A toy 2-D latent space with four kinds of image, an exact noise predictor that follows the prompt only halfway, SD v1's schedule and 64 seeds. At the defaults (prompt “a dog”, w = 7.5, 50 DDIM steps, no negative prompt) all 64 samples are dogs, but 27 are blurry (57.8% sharp), at 100 U-Net evaluations. Add the negative prompt “blurry” and 63 of 64 are sharp dogs (98.4%), with spread 1.35× the data's; at w = 15 the spread reaches 2.18× (oversaturation). At w = 1 only 68.8% are dogs, for 50 evaluations. Seeds move 0.011 on average between 50 and 100 DDIM steps (0.084 at 10 steps), but 0.485 with the ancestral sampler.

The figure separates the knobs. The seed picks the starting point, so the same seed always lands in the same place. Guidance decides how hard the sample is pulled towards the prompt. The negative prompt changes the direction of that pull, here away from the blurry half of the space. Steps change little once a deterministic solver has converged, while an ancestral sampler lands elsewhere for every step count.

Worked example

At one step, let the negative-prompt prediction be ϵneg=(0.10,−0.20)\epsilon_{\text{neg}}=(0.10,-0.20) and the prompt prediction ϵc=(0.30,0.20)\epsilon_c=(0.30,0.20) on two latent coordinates. The difference is (0.20,0.40)(0.20,0.40), and with w=7.5w=7.5: ϵ~=(0.10+7.5⋅0.20, −0.20+7.5⋅0.40)=(1.60, 2.80)\tilde\epsilon=(0.10+7.5\cdot0.20,\ -0.20+7.5\cdot0.40)=(1.60,\ 2.80). With w=1w=1 it would be ϵc=(0.30,0.20)\epsilon_c=(0.30,0.20). Cost depends on steps and on whether guidance runs:

SettingStepsGuidanceEvaluations
SD v1 diffusers default50w = 7.550 × 2 = 100
DPM-Solver++25w = 7.525 × 2 = 50
DPM-Solver++, no guidance25w = 125
distilled 1-step model1none1
Denoiser evaluations per image (the VAE decode is extra and runs once)
python
1import torch
2from diffusers import AutoPipelineForText2Image, DPMSolverMultistepScheduler
3
4pipe = AutoPipelineForText2Image.from_pretrained(
5 "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
6).to("cuda")
7pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
8
9g = torch.Generator("cuda").manual_seed(42) # fixes the initial latent noise
10image = pipe(
11 prompt="a corgi in a yellow raincoat, watercolor",
12 negative_prompt="blurry, low quality", # replaces the empty prompt in CFG
13 num_inference_steps=25,
14 guidance_scale=7.0,
15 height=1024, width=1024, # SDXL's training resolution
16 generator=g,
17).images[0]
18image.save("corgi_seed42.png") # log every argument with the file
A reproducible SDXL call with Hugging Face diffusers (needs a GPU and the downloaded weights; not run here)

Check yourself

A user adds the negative prompt “dog” to the prompt “a cat”. What does the sampler actually do?

Check yourself

You found a good image at seed 7 with 30 steps of Euler a (ancestral). You rerun it at 60 steps for more detail. What happens?

Remember

  • ✓Pipeline: tokenize (77 tokens for CLIP) → frozen text encoder → seeded latent noise → N denoising steps with cross-attention → VAE decode.
  • ✓Classifier-free guidance: ϵ~=ϵneg+w(ϵc−ϵneg)\tilde\epsilon=\epsilon_{\text{neg}}+w(\epsilon_c-\epsilon_{\text{neg}}); typical ww 5–8, two evaluations per step.
  • ✓The negative prompt replaces the empty prompt as the guidance reference; it is a direction, not a filter.
  • ✓Deterministic solvers make a seed stable across step counts; ancestral samplers do not.
  • ✓Reproducibility needs weights, prompt, negative prompt, seed, size, steps, scheduler and guidance scale together.
← back to Generative AI roadmap