Definition
DreamBooth is a method for subject-driven generation: given 3–5 photos of one specific subject (a dog, a backpack, a person), it fine-tunes a pretrained text-to-image diffusion model so that a short identifier token renders that subject in any new prompt. It was introduced by Ruiz, Li, Jampani, Pritch, Rubinstein and Aberman (Google Research and Boston University) in "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" (arXiv August 2022, CVPR 2023), on Imagen and on Stable Diffusion.
A base model can draw "a corgi on the beach" but not your corgi. Prompting cannot fix this: no text description pins down a particular face, coat pattern or logo. DreamBooth solves the problem by changing the weights. Every training photo gets the same caption, "a [V] dog": a unique identifier [V] plus the subject's coarse class noun. After training, "a [V] dog in the snow" keeps the identity and follows the rest of the prompt.
In a product it sits in the customization step: a user uploads a handful of photos, a short fine-tuning job runs, and the result is either a full checkpoint or, often, a small adapter (DreamBooth combined with LoRA). Its cheaper sibling, Textual inversion, learns only a new word embedding and leaves the model frozen. Fine-tuning vs RAG vs prompting covers when any weight change is worth it.
Identifier and class noun
The identifier should have a weak prior: the model should not already associate it with anything. A common word such as "red" or "unique" must first be unlearned, and its usual meaning gets damaged along the way. A random string such as "xxy5syt00" is no better, because the tokenizer splits it into ordinary subwords that do have priors. The paper picks rare tokens from the vocabulary instead: for Imagen's T5-XXL tokenizer, a sequence of 1–3 tokens drawn from IDs in the range 5,000–10,000, decoded back to text. Stable Diffusion users settled on "sks". The class noun ("dog", "backpack") lets the model reuse what it already knows about the class, which speeds up training and improves fidelity.
The loss
Plain fine-tuning on 3–5 images has two failure modes. Language drift: the class word itself is overwritten, so "a dog" starts producing the subject. Reduced output diversity: poses and backgrounds collapse onto the few training views. DreamBooth adds a class-specific prior-preservation loss. Before training, the frozen model generates images from the prompt "a dog". Fine-tuning then trains on both sets at once:
- a subject photo and the embedding of "a [V] dog"
- an image the frozen model generated for "a dog", and that prompt's embedding
- the denoiser being fine-tuned (all weights ), predicting the clean image from the noised one
- noise schedule and per-timestep weight, as in Diffusion models
- prior weight; the paper uses
The second term is self-distillation: it pins the model's output for the class prompt to what the original model produced. The subject can then live in the identifier alone. Stable Diffusion code predicts the noise rather than . The two forms are equivalent up to the weight , so implementations use the mean squared error on for both terms.
Settings and results reported in the paper
- Training data
- 3–5 subject images; about 1,000 class images generated by the frozen model (fewer also work)
- Schedule
- about 1,000 iterations, λ = 1; learning rate 10⁻⁵ (Imagen) or 5 × 10⁻⁶ (Stable Diffusion)
- Cost
- about 5 minutes on one TPUv4 (Imagen) or one NVIDIA A100 (Stable Diffusion)
- Imagen detail
- the 64 × 64 base model and the 64 → 256 super-resolution model are both fine-tuned, so fine details of the subject survive upsampling
- Benchmark
- 30 subjects (21 objects, 9 live subjects such as pets) × 25 prompts; subject fidelity by DINO and CLIP-I similarity to the real photos, prompt fidelity by CLIP-T
| Method | DINO | CLIP-I | CLIP-T |
|---|---|---|---|
| Real images | 0.774 | 0.885 | — |
| DreamBooth (Imagen) | 0.696 | 0.812 | 0.306 |
| DreamBooth (Stable Diffusion) | 0.668 | 0.803 | 0.305 |
| Textual Inversion (Stable Diffusion) | 0.569 | 0.780 | 0.255 |
The paper argues for DINO over CLIP-I as the subject metric. DINO is trained self-supervised to tell individual images apart, so it rewards the subject's identity. CLIP-I scores two different yellow backpacks as similar because their captions would match. CLIP score explains CLIP-T.
The toy shows why the second term works. The two prompts share the class direction , so every gradient step on "a [V] dog" also moves "a dog". The prior term cancels that component, and the subject ends up stored in the identifier's direction alone. Note that with λ = 1 the drift first rises, peaking at 0.829 at step 9, and only then decays: the prior loss corrects drift, it does not prevent it.
Worked example: the toy run step by step
| Step | Fit (λ = 0) | Drift (λ = 0) | Fit (λ = 1) | Drift (λ = 1) |
|---|---|---|---|---|
| 0 | 2.614 | 0.000 | 2.614 | 0.000 |
| 10 | 0.281 | 1.167 | 0.604 | 0.828 |
| 25 | 0.010 | 1.303 | 0.353 | 0.621 |
| 100 | 0.000 | 1.308 | 0.060 | 0.106 |
| 300 | 0.000 | 1.308 | 0.001 | 0.001 |
At step 0 the reconstruction loss is and the prior loss is 0. Without the prior the fit is fast and the damage is permanent. With it, getting within 0.05 of the subject takes 108 steps instead of 18, and both terms end near zero.
1import torch2import torch.nn.functional as F34# batch = 2 subject photos ("a photo of sks dog") + 2 class images ("a photo of dog"),5# concatenated so one U-Net forward pass serves both loss terms6latents = vae.encode(pixels).latent_dist.sample() * vae.config.scaling_factor7noise = torch.randn_like(latents)8t = torch.randint(0, scheduler.config.num_train_timesteps, (latents.shape[0],), device=latents.device)9noisy = scheduler.add_noise(latents, noise, t)10cond = text_encoder(input_ids)[0] # embeddings of both prompts1112pred = unet(noisy, t, encoder_hidden_states=cond).sample # epsilon prediction13pred_subj, pred_prior = pred.chunk(2)14noise_subj, noise_prior = noise.chunk(2)1516loss = F.mse_loss(pred_subj.float(), noise_subj.float()) # reconstruction17prior_loss = F.mse_loss(pred_prior.float(), noise_prior.float()) # prior preservation18(loss + 1.0 * prior_loss).backward() # lambda = 119optimizer.step(); optimizer.zero_grad() # lr 5e-6 for SDA practical recipe
- 1Collect 3–5 (up to about 20) varied photos: different backgrounds, angles and lighting; crop to the model's resolution.
- 2Pick a rare identifier and the right class noun: "a photo of sks backpack", not "a photo of sks object".
- 3Generate the class images once with the frozen model and the class prompt (diffusers' script defaults to 100 of them; the paper used about 1,000).
- 4Train with λ = 1 and a low learning rate (5 × 10⁻⁶ for full fine-tuning of SD), checking samples every few hundred steps.
- 5Stop when new-context prompts keep the identity; more steps start reproducing the training backgrounds and poses.
Use when
- +Highest subject fidelity of the early personalization methods (DINO 0.696 vs 0.569 for Textual Inversion).
- +Needs only 3–5 photos and minutes of GPU time.
- +Works with any class the base model knows; the class noun supplies the prior.
Avoid when
- −Full fine-tuning produces a whole new checkpoint per subject (gigabytes) unless combined with LoRA.
- −Must generate and train on class images to avoid language drift.
- −Several subjects in one model tend to blend; multi-subject composition needs extra methods.
Check yourself
You fine-tune on 4 photos of your dog captioned "a sks dog" with no prior-preservation term. Afterwards "a dog" mostly draws your dog. What happened?
Check yourself
Why does DreamBooth avoid an identifier like "xxy5syt00"?
Remember
- ✓DreamBooth fine-tunes all weights of a text-to-image diffusion model on 3–5 photos captioned "a [V] class".
- ✓[V] is a rare token with a weak prior ("sks" in Stable Diffusion); the class noun reuses the model's knowledge of the class.
- ✓Prior preservation: λ · reconstruction loss on frozen-model images of "a class" (λ = 1) prevents language drift and keeps outputs diverse.
- ✓Paper settings: about 1,000 steps, lr 10⁻⁵ (Imagen) or 5 × 10⁻⁶ (SD), about 5 minutes on one A100 or TPUv4.
- ✓DINO 0.696 vs 0.569 for Textual Inversion on the paper's 30-subject benchmark; overfitting shows up as copied backgrounds and poses.