Definition
Text-to-image generation produces an image from a natural-language prompt. Current open systems are conditional latent diffusion models: a frozen text encoder turns the prompt into embeddings, a denoising network refines random latent noise over a fixed number of steps while attending to those embeddings, and a VAE decoder maps the result to pixels. Almost all of them steer the denoiser with classifier-free guidance (Ho and Salimans, "Classifier-Free Diffusion Guidance", NeurIPS 2021 workshop, arXiv 2022).
This article covers the generation call as a practitioner sees it: the prompt, the negative prompt, the seed, the step count, the scheduler and the guidance scale, and what each one costs. How the denoiser is trained and why the sampler works are covered in Diffusion models, Latent diffusion models and Stable Diffusion. Editing an existing image is in Image editing and inpainting, and spatial control in ControlNet.
In a product the model sits behind a request with a handful of parameters. Every one of them changes the output, so reproducing an image requires all of them: the same weights, prompt, negative prompt, seed, size, step count, scheduler and guidance scale. Most GPU stacks are deterministic for a fixed setup, but changing the GPU type or library version can change pixels slightly.
The pipeline
- 1Tokenize and encode. Stable Diffusion v1 uses CLIP's BPE tokenizer and pads or truncates the prompt to 77 tokens (75 of text plus begin and end markers). Its frozen CLIP ViT-L/14 text encoder returns a matrix of hidden states, not a single vector (see CLIP).
- 2Seed the noise. A seeded random generator fills the initial latent . At it is , 48 times fewer values than the RGB image.
- 3Denoise. For each of timesteps the U-Net (or, in newer models, a diffusion transformer) predicts the noise in , attending to the text through Cross-attention. Guidance combines a conditional and an unconditional prediction, and the scheduler turns the result into .
- 4Decode. The VAE decoder maps (divided by SD's latent scale 0.18215) back to pixels. Safety checkers or watermarking may run afterwards.
Guidance and the negative prompt
During training the caption is replaced by the empty prompt a small fraction of the time (10% for SD v1.3 onwards), so one network learns both and . At sampling time the two are extrapolated:
- embedding of the prompt
- embedding of the negative prompt; with none given, the embedding of the empty string
- guidance scale (
guidance_scalein diffusers, default 7.5 for SD v1); is plain conditional sampling and needs only one evaluation per step
So a negative prompt is not a filter. It replaces the empty prompt as the reference point, and guidance pushes every step away from it along . That is why "blurry, low quality, extra fingers" can help, and why a negative prompt that shares words with the prompt weakens the prompt. Larger raises prompt adherence and contrast and lowers diversity. Very large oversaturates colours because the guided prediction leaves the range of real latents. Imagen (Saharia et al., 2022) added dynamic thresholding to clip the predicted image and make large weights usable.
Steps and schedulers
The scheduler (the "sampler" in UIs) decides which timesteps are visited and how a noise prediction becomes the next latent. The deterministic DDIM update, which the figure below uses, is:
- fraction of signal variance left at timestep ; SD v1 uses a scaled-linear schedule with from 0.00085 to 0.012 over 1,000 steps
- the next timestep; 50 steps visit in diffusers' leading spacing
- the current guess of the clean latent; previews in UIs decode this
Deterministic ODE solvers (DDIM, Euler, DPM-Solver++, Heun) map one seed to nearly the same image at 25 or at 100 steps; more steps only refine it. Ancestral samplers (Euler a, DDPM) add fresh noise at every step, so the image keeps changing with the step count. DPM-Solver++ (Lu et al., 2022) gives good results in about 20–25 steps. Distilled models need far fewer: Consistency models and adversarial distillation (SDXL-Turbo, Sauer et al., 2023) sample in 1–4 steps and usually run without guidance.
Reference systems (as published)
- Stable Diffusion v1 (2022)
- 860M-parameter U-Net, 64 × 64 × 4 latent for 512 × 512, frozen CLIP ViT-L/14 text encoder (77 tokens × 768)
- SDXL (Podell et al., 2023)
- 2.6B-parameter U-Net, two text encoders (OpenCLIP ViT-bigG/14 and CLIP ViT-L/14), 1024 × 1024 base resolution, optional refiner model
- Imagen (Saharia et al., 2022)
- frozen T5-XXL text encoder; scaling the text encoder helped more than scaling the image U-Net; zero-shot MS-COCO FID 7.27
- DALL·E 2 (Ramesh et al., 2022)
- a prior maps the CLIP text embedding to a CLIP image embedding, and a diffusion decoder generates from it; MS-COCO FID 10.39
- diffusers defaults
StableDiffusionPipeline:num_inference_steps=50,guidance_scale=7.5; guidance is skipped whenguidance_scale≤ 1
The figure separates the knobs. The seed picks the starting point, so the same seed always lands in the same place. Guidance decides how hard the sample is pulled towards the prompt. The negative prompt changes the direction of that pull, here away from the blurry half of the space. Steps change little once a deterministic solver has converged, while an ancestral sampler lands elsewhere for every step count.
Worked example
At one step, let the negative-prompt prediction be and the prompt prediction on two latent coordinates. The difference is , and with : . With it would be . Cost depends on steps and on whether guidance runs:
| Setting | Steps | Guidance | Evaluations |
|---|---|---|---|
| SD v1 diffusers default | 50 | w = 7.5 | 50 × 2 = 100 |
| DPM-Solver++ | 25 | w = 7.5 | 25 × 2 = 50 |
| DPM-Solver++, no guidance | 25 | w = 1 | 25 |
| distilled 1-step model | 1 | none | 1 |
1import torch2from diffusers import AutoPipelineForText2Image, DPMSolverMultistepScheduler34pipe = AutoPipelineForText2Image.from_pretrained(5 "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float166).to("cuda")7pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)89g = torch.Generator("cuda").manual_seed(42) # fixes the initial latent noise10image = pipe(11 prompt="a corgi in a yellow raincoat, watercolor",12 negative_prompt="blurry, low quality", # replaces the empty prompt in CFG13 num_inference_steps=25,14 guidance_scale=7.0,15 height=1024, width=1024, # SDXL's training resolution16 generator=g,17).images[0]18image.save("corgi_seed42.png") # log every argument with the fileCheck yourself
A user adds the negative prompt “dog” to the prompt “a cat”. What does the sampler actually do?
Check yourself
You found a good image at seed 7 with 30 steps of Euler a (ancestral). You rerun it at 60 steps for more detail. What happens?
Remember
- ✓Pipeline: tokenize (77 tokens for CLIP) → frozen text encoder → seeded latent noise → N denoising steps with cross-attention → VAE decode.
- ✓Classifier-free guidance: ; typical 5–8, two evaluations per step.
- ✓The negative prompt replaces the empty prompt as the guidance reference; it is a direction, not a filter.
- ✓Deterministic solvers make a seed stable across step counts; ancestral samplers do not.
- ✓Reproducibility needs weights, prompt, negative prompt, seed, size, steps, scheduler and guidance scale together.