# DreamBooth

> DreamBooth (Ruiz et al., CVPR 2023) fine-tunes all weights of a text-to-image diffusion model on 3–5 photos captioned "a [V] dog", with a rare-token identifier [V] and a prior-preservation loss that keeps "a dog" generic.

Source: https://www.mlfoundry.in/learn/dreambooth  
Section: Generative AI › Customization  
Level: advanced · 8 min read · updated 2026-10-21

**DreamBooth:** is a method for **subject-driven generation**: given 3–5 photos of one specific subject (a dog, a backpack, a person), it fine-tunes a pretrained text-to-image diffusion model so that a short identifier token renders that subject in any new prompt. It was introduced by Ruiz, Li, Jampani, Pritch, Rubinstein and Aberman (Google Research and Boston University) in "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" (arXiv August 2022, CVPR 2023), on Imagen and on **Stable Diffusion**.

A base model can draw "a corgi on the beach" but not *your* corgi. Prompting cannot fix this: no text description pins down a particular face, coat pattern or logo. DreamBooth solves the problem by changing the weights. Every training photo gets the same caption, "a [V] dog": a unique identifier [V] plus the subject's coarse class noun. After training, "a [V] dog in the snow" keeps the identity and follows the rest of the prompt.

In a product it sits in the customization step: a user uploads a handful of photos, a short fine-tuning job runs, and the result is either a full checkpoint or, often, a small adapter (DreamBooth combined with **LoRA**). Its cheaper sibling, **Textual inversion**, learns only a new word embedding and leaves the model frozen. **Fine-tuning vs RAG vs prompting** covers when any weight change is worth it.

## Identifier and class noun

The identifier should have a **weak prior**: the model should not already associate it with anything. A common word such as "red" or "unique" must first be unlearned, and its usual meaning gets damaged along the way. A random string such as "xxy5syt00" is no better, because the tokenizer splits it into ordinary subwords that do have priors. The paper picks rare tokens from the vocabulary instead: for Imagen's T5-XXL tokenizer, a sequence of 1–3 tokens drawn from IDs in the range 5,000–10,000, decoded back to text. Stable Diffusion users settled on "sks". The class noun ("dog", "backpack") lets the model reuse what it already knows about the class, which speeds up training and improves fidelity.

## The loss

Plain fine-tuning on 3–5 images has two failure modes. **Language drift**: the class word itself is overwritten, so "a dog" starts producing the subject. **Reduced output diversity**: poses and backgrounds collapse onto the few training views. DreamBooth adds a class-specific **prior-preservation loss**. Before training, the frozen model generates images $x_\text{pr}$ from the prompt "a dog". Fine-tuning then trains on both sets at once:

$$
\mathcal L=\mathbb E_{x,c,\epsilon,t}\Big[w_t\big\|\hat x_\theta(\alpha_t x+\sigma_t\epsilon,\,c)-x\big\|_2^2+\lambda\,w_{t'}\big\|\hat x_\theta(\alpha_{t'}x_\text{pr}+\sigma_{t'}\epsilon',\,c_\text{pr})-x_\text{pr}\big\|_2^2\Big]
$$

- $x,\;c$: a subject photo and the embedding of "a [V] dog"
- $x_\text{pr},\;c_\text{pr}$: an image the frozen model generated for "a dog", and that prompt's embedding
- $\hat x_\theta$: the denoiser being fine-tuned (all weights $\theta$), predicting the clean image from the noised one
- $\alpha_t,\sigma_t,w_t$: noise schedule and per-timestep weight, as in **Diffusion models**
- $\lambda$: prior weight; the paper uses $\lambda=1$

The second term is self-distillation: it pins the model's output for the class prompt to what the original model produced. The subject can then live in the identifier alone. Stable Diffusion code predicts the noise $\epsilon$ rather than $x$. The two forms are equivalent up to the weight $w_t$, so implementations use the mean squared error on $\epsilon$ for both terms.

> Figure: Training: the four subject photos, all captioned "a [V] dog", and about 1,000 "a dog" images sampled once from the frozen original model feed one diffusion model whose weights are all trainable. The reconstruction loss binds the subject to [V]; the prior-preservation loss (λ = 1) keeps "a dog" generic. Inference: the fine-tuned model renders the subject in contexts it never saw.

**Settings and results reported in the paper**

- **Training data:** 3–5 subject images; about 1,000 class images generated by the frozen model (fewer also work)
- **Schedule:** about 1,000 iterations, λ = 1; learning rate 10⁻⁵ (Imagen) or 5 × 10⁻⁶ (Stable Diffusion)
- **Cost:** about 5 minutes on one TPUv4 (Imagen) or one NVIDIA A100 (Stable Diffusion)
- **Imagen detail:** the 64 × 64 base model and the 64 → 256 super-resolution model are both fine-tuned, so fine details of the subject survive upsampling
- **Benchmark:** 30 subjects (21 objects, 9 live subjects such as pets) × 25 prompts; subject fidelity by DINO and CLIP-I similarity to the real photos, prompt fidelity by CLIP-T

Subject and prompt fidelity on the paper's benchmark (higher is better). "Real images" compares held-out real photos of the same subject.

| Method | DINO | CLIP-I | CLIP-T |
| --- | --- | --- | --- |
| Real images | 0.774 | 0.885 | — |
| DreamBooth (Imagen) | 0.696 | 0.812 | 0.306 |
| DreamBooth (Stable Diffusion) | 0.668 | 0.803 | 0.305 |
| Textual Inversion (Stable Diffusion) | 0.569 | 0.780 | 0.255 |

The paper argues for DINO over CLIP-I as the subject metric. DINO is trained self-supervised to tell individual images apart, so it rewards the subject's identity. CLIP-I scores two different yellow backpacks as similar because their captions would match. **CLIP score** explains CLIP-T.

> Figure: A toy linear generator maps prompt embeddings to a 2-D image-feature plane and is fine-tuned by gradient descent. Defaults (rare token "sks", λ = 1, 100 steps): fit to the subject 0.060 (from 2.614), drift of "a dog" 0.106, and the word "red" moved 0.291. Set λ = 0: the fit reaches 0.000 by step 50, but "a dog" moves 1.308 toward the subject, 52% of its 2.530 distance. Switch to the common word "red" (λ = 1): the drift still ends near 0, but "red" itself moves 3.444 by step 100.

The toy shows why the second term works. The two prompts share the class direction $c_\text{dog}$, so every gradient step on "a [V] dog" also moves "a dog". The prior term cancels that component, and the subject ends up stored in the identifier's direction alone. Note that with λ = 1 the drift first rises, peaking at 0.829 at step 9, and only then decays: the prior loss corrects drift, it does not prevent it.

## Worked example: the toy run step by step

Distances in the toy at selected steps (rare token, lr 0.08). Fit is to the centroid of the four subject photos; drift is how far the "a dog" output moved from the frozen model's.

| Step | Fit (λ = 0) | Drift (λ = 0) | Fit (λ = 1) | Drift (λ = 1) |
| --- | --- | --- | --- | --- |
| 0 | 2.614 | 0.000 | 2.614 | 0.000 |
| 10 | 0.281 | 1.167 | 0.604 | 0.828 |
| 25 | 0.010 | 1.303 | 0.353 | 0.621 |
| 100 | 0.000 | 1.308 | 0.060 | 0.106 |
| 300 | 0.000 | 1.308 | 0.001 | 0.001 |

At step 0 the reconstruction loss is $2.614^2=6.833$ and the prior loss is 0. Without the prior the fit is fast and the damage is permanent. With it, getting within 0.05 of the subject takes 108 steps instead of 18, and both terms end near zero.

```python
import torch
import torch.nn.functional as F

# batch = 2 subject photos ("a photo of sks dog") + 2 class images ("a photo of dog"),
# concatenated so one U-Net forward pass serves both loss terms
latents = vae.encode(pixels).latent_dist.sample() * vae.config.scaling_factor
noise = torch.randn_like(latents)
t = torch.randint(0, scheduler.config.num_train_timesteps, (latents.shape[0],), device=latents.device)
noisy = scheduler.add_noise(latents, noise, t)
cond = text_encoder(input_ids)[0]                      # embeddings of both prompts

pred = unet(noisy, t, encoder_hidden_states=cond).sample   # epsilon prediction
pred_subj, pred_prior = pred.chunk(2)
noise_subj, noise_prior = noise.chunk(2)

loss = F.mse_loss(pred_subj.float(), noise_subj.float())          # reconstruction
prior_loss = F.mse_loss(pred_prior.float(), noise_prior.float())  # prior preservation
(loss + 1.0 * prior_loss).backward()                              # lambda = 1
optimizer.step(); optimizer.zero_grad()                           # lr 5e-6 for SD
```

One DreamBooth training step with prior preservation, following the structure of diffusers' train_dreambooth.py (not run here: needs torch, diffusers and a GPU)

**A practical recipe**

1. Collect 3–5 (up to about 20) varied photos: different backgrounds, angles and lighting; crop to the model's resolution.
2. Pick a rare identifier and the right class noun: "a photo of sks backpack", not "a photo of sks object".
3. Generate the class images once with the frozen model and the class prompt (diffusers' script defaults to 100 of them; the paper used about 1,000).
4. Train with λ = 1 and a low learning rate (5 × 10⁻⁶ for full fine-tuning of SD), checking samples every few hundred steps.
5. Stop when new-context prompts keep the identity; more steps start reproducing the training backgrounds and poses.

> **Overfitting shows up as copied context:** Too many steps or too high a learning rate and "a [V] dog on the moon" returns the living-room carpet from a training photo, or the same pose every time. Fewer steps, more varied photos, the prior term and fine-tuning only a **LoRA** adapter all help. Faces are the hardest case: small identity errors are obvious to people.

> **Consent and misuse:** A few public photos are enough to put a real person into any scene, which is why DreamBooth-style tools come up in **Deepfake detection** and **Copyright and licensing**. Hosted products typically require that users own the photos or have the subject's consent, and they filter prompts.

**Strengths**

- Highest subject fidelity of the early personalization methods (DINO 0.696 vs 0.569 for Textual Inversion).
- Needs only 3–5 photos and minutes of GPU time.
- Works with any class the base model knows; the class noun supplies the prior.

**Limitations**

- Full fine-tuning produces a whole new checkpoint per subject (gigabytes) unless combined with LoRA.
- Must generate and train on class images to avoid language drift.
- Several subjects in one model tend to blend; multi-subject composition needs extra methods.

**Question:** You fine-tune on 4 photos of your dog captioned "a sks dog" with no prior-preservation term. Afterwards "a dog" mostly draws your dog. What happened?

- A. The identifier "sks" was not rare enough
- B. Language drift: gradients for "a sks dog" also moved the shared class word "dog" toward the subject
- C. The learning rate was too low
- D. The text encoder was frozen

**Answer:** B. Language drift: gradients for "a sks dog" also moved the shared class word "dog" toward the subject — Both prompts share the class token, so fitting "a sks dog" moves the model's output for "dog" as well. The prior-preservation loss trains on the frozen model's own "a dog" images and pulls that output back. In the toy figure, λ = 0 leaves a drift of 1.308.

**Question:** Why does DreamBooth avoid an identifier like "xxy5syt00"?

- A. It is too long for the 77-token context
- B. The tokenizer splits it into common subword pieces that already carry strong priors
- C. Random strings cannot be embedded
- D. It makes the prior-preservation loss undefined

**Answer:** B. The tokenizer splits it into common subword pieces that already carry strong priors — The model never sees the string as a whole: it sees subword tokens such as "xx", "y", "5", each with existing meanings. A rare single token in the vocabulary has a weak prior, so little must be unlearned.

**Key points**

- DreamBooth fine-tunes all weights of a text-to-image diffusion model on 3–5 photos captioned "a [V] class".
- [V] is a rare token with a weak prior ("sks" in Stable Diffusion); the class noun reuses the model's knowledge of the class.
- Prior preservation: λ · reconstruction loss on frozen-model images of "a class" (λ = 1) prevents language drift and keeps outputs diverse.
- Paper settings: about 1,000 steps, lr 10⁻⁵ (Imagen) or 5 × 10⁻⁶ (SD), about 5 minutes on one A100 or TPUv4.
- DINO 0.696 vs 0.569 for Textual Inversion on the paper's 30-subject benchmark; overfitting shows up as copied backgrounds and poses.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
