DreamBooth

DreamBooth (Ruiz et al., CVPR 2023) fine-tunes all weights of a text-to-image diffusion model on 3–5 photos captioned "a [V] dog", with a rare-token identifier [V] and a prior-preservation loss that keeps "a dog" generic.

8 min read932 wordsadvanced0 viewsupdated 2026-10-21

Definition

DreamBooth is a method for subject-driven generation: given 3–5 photos of one specific subject (a dog, a backpack, a person), it fine-tunes a pretrained text-to-image diffusion model so that a short identifier token renders that subject in any new prompt. It was introduced by Ruiz, Li, Jampani, Pritch, Rubinstein and Aberman (Google Research and Boston University) in "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" (arXiv August 2022, CVPR 2023), on Imagen and on Stable Diffusion.

A base model can draw "a corgi on the beach" but not your corgi. Prompting cannot fix this: no text description pins down a particular face, coat pattern or logo. DreamBooth solves the problem by changing the weights. Every training photo gets the same caption, "a [V] dog": a unique identifier [V] plus the subject's coarse class noun. After training, "a [V] dog in the snow" keeps the identity and follows the rest of the prompt.

In a product it sits in the customization step: a user uploads a handful of photos, a short fine-tuning job runs, and the result is either a full checkpoint or, often, a small adapter (DreamBooth combined with LoRA). Its cheaper sibling, Textual inversion, learns only a new word embedding and leaves the model frozen. Fine-tuning vs RAG vs prompting covers when any weight change is worth it.

Identifier and class noun

The identifier should have a weak prior: the model should not already associate it with anything. A common word such as "red" or "unique" must first be unlearned, and its usual meaning gets damaged along the way. A random string such as "xxy5syt00" is no better, because the tokenizer splits it into ordinary subwords that do have priors. The paper picks rare tokens from the vocabulary instead: for Imagen's T5-XXL tokenizer, a sequence of 1–3 tokens drawn from IDs in the range 5,000–10,000, decoded back to text. Stable Diffusion users settled on "sks". The class noun ("dog", "backpack") lets the model reuse what it already knows about the class, which speeds up training and improves fidelity.

The loss

Plain fine-tuning on 3–5 images has two failure modes. Language drift: the class word itself is overwritten, so "a dog" starts producing the subject. Reduced output diversity: poses and backgrounds collapse onto the few training views. DreamBooth adds a class-specific prior-preservation loss. Before training, the frozen model generates images xprx_\text{pr} from the prompt "a dog". Fine-tuning then trains on both sets at once:

L=Ex,c,ϵ,t[wt∥x^θ(αtx+σtϵ, c)−x∥22+λ wt′∥x^θ(αt′xpr+σt′ϵ′, cpr)−xpr∥22]\mathcal L=\mathbb E_{x,c,\epsilon,t}\Big[w_t\big\|\hat x_\theta(\alpha_t x+\sigma_t\epsilon,\,c)-x\big\|_2^2+\lambda\,w_{t'}\big\|\hat x_\theta(\alpha_{t'}x_\text{pr}+\sigma_{t'}\epsilon',\,c_\text{pr})-x_\text{pr}\big\|_2^2\Big]
x,  cx,\;c
a subject photo and the embedding of "a [V] dog"
xpr,  cprx_\text{pr},\;c_\text{pr}
an image the frozen model generated for "a dog", and that prompt's embedding
x^θ\hat x_\theta
the denoiser being fine-tuned (all weights θ\theta), predicting the clean image from the noised one
αt,σt,wt\alpha_t,\sigma_t,w_t
noise schedule and per-timestep weight, as in Diffusion models
λ\lambda
prior weight; the paper uses λ=1\lambda=1

The second term is self-distillation: it pins the model's output for the class prompt to what the original model produced. The subject can then live in the identifier alone. Stable Diffusion code predicts the noise ϵ\epsilon rather than xx. The two forms are equivalent up to the weight wtw_t, so implementations use the mean squared error on ϵ\epsilon for both terms.

DreamBooth (Ruiz et al., CVPR 2023): fine-tune the whole model on 3–5 photossubject: 4 photoscaption for all:"a [V] dog"frozen original modelprompt "a dog"sample once, beforetraining: ~1,000 x_prdiffusion model θ(Imagen or SD)ALL weights trained(U-Net; text encoder optional in SD)lr 1e-5 Imagen, 5e-6 SDx, cx_prreconstructionw_t‖x̂_θ(z_t, c) − x‖²learn the subjectand bind it to [V]prior preservationλ w_t′‖x̂_θ(z_t′,c_pr)−x_pr‖²λ = 1: keeps "a dog"generic and diverse+ (sum, ~1,000 steps)inference with the fine-tuned θ"a [V] dog on the beach""a [V] dog, oil painting""a dog" → still genericθ → imagesubject innew context[V] = rare token, e.g. "sks"not "xxy5syt00": it splits intocommon subwords with strong priors
fig.Training: the four subject photos, all captioned "a [V] dog", and about 1,000 "a dog" images sampled once from the frozen original model feed one diffusion model whose weights are all trainable. The reconstruction loss binds the subject to [V]; the prior-preservation loss (λ = 1) keeps "a dog" generic. Inference: the fine-tuned model renders the subject in contexts it never saw.

Settings and results reported in the paper

Training data
3–5 subject images; about 1,000 class images generated by the frozen model (fewer also work)
Schedule
about 1,000 iterations, λ = 1; learning rate 10⁻⁵ (Imagen) or 5 × 10⁻⁶ (Stable Diffusion)
Cost
about 5 minutes on one TPUv4 (Imagen) or one NVIDIA A100 (Stable Diffusion)
Imagen detail
the 64 × 64 base model and the 64 → 256 super-resolution model are both fine-tuned, so fine details of the subject survive upsampling
Benchmark
30 subjects (21 objects, 9 live subjects such as pets) × 25 prompts; subject fidelity by DINO and CLIP-I similarity to the real photos, prompt fidelity by CLIP-T
MethodDINOCLIP-ICLIP-T
Real images0.7740.885—
DreamBooth (Imagen)0.6960.8120.306
DreamBooth (Stable Diffusion)0.6680.8030.305
Textual Inversion (Stable Diffusion)0.5690.7800.255
Subject and prompt fidelity on the paper's benchmark (higher is better). "Real images" compares held-out real photos of the same subject.

The paper argues for DINO over CLIP-I as the subject metric. DINO is trained self-supervised to tell individual images apart, so it rewards the subject's identity. CLIP-I scores two different yellow backpacks as similar because their captions would match. CLIP score explains CLIP-T.

L = |W(c_sks + c_dog) − x_subj|² + λ·|W c_dog − W₀ c_dog|² λ = 1, step 1004 subject photos"a dog" before"red" before"a sks dog" output"a dog" samples now"red" output now (dashed ring = before)W: 2 × 4 weights, lr 0.08, full-batch gradient descent"a dog" to subject centroid before training: 2.530distances vs fine-tuning step0240100200300fit: "a sks dog" to subject0.060language drift: "a dog" moved0.106meaning of "red" moved0.291
identifier [V]
fig.A toy linear generator maps prompt embeddings to a 2-D image-feature plane and is fine-tuned by gradient descent. Defaults (rare token "sks", λ = 1, 100 steps): fit to the subject 0.060 (from 2.614), drift of "a dog" 0.106, and the word "red" moved 0.291. Set λ = 0: the fit reaches 0.000 by step 50, but "a dog" moves 1.308 toward the subject, 52% of its 2.530 distance. Switch to the common word "red" (λ = 1): the drift still ends near 0, but "red" itself moves 3.444 by step 100.

The toy shows why the second term works. The two prompts share the class direction cdogc_\text{dog}, so every gradient step on "a [V] dog" also moves "a dog". The prior term cancels that component, and the subject ends up stored in the identifier's direction alone. Note that with λ = 1 the drift first rises, peaking at 0.829 at step 9, and only then decays: the prior loss corrects drift, it does not prevent it.

Worked example: the toy run step by step

StepFit (λ = 0)Drift (λ = 0)Fit (λ = 1)Drift (λ = 1)
02.6140.0002.6140.000
100.2811.1670.6040.828
250.0101.3030.3530.621
1000.0001.3080.0600.106
3000.0001.3080.0010.001
Distances in the toy at selected steps (rare token, lr 0.08). Fit is to the centroid of the four subject photos; drift is how far the "a dog" output moved from the frozen model's.

At step 0 the reconstruction loss is 2.6142=6.8332.614^2=6.833 and the prior loss is 0. Without the prior the fit is fast and the damage is permanent. With it, getting within 0.05 of the subject takes 108 steps instead of 18, and both terms end near zero.

python
1import torch
2import torch.nn.functional as F
3
4# batch = 2 subject photos ("a photo of sks dog") + 2 class images ("a photo of dog"),
5# concatenated so one U-Net forward pass serves both loss terms
6latents = vae.encode(pixels).latent_dist.sample() * vae.config.scaling_factor
7noise = torch.randn_like(latents)
8t = torch.randint(0, scheduler.config.num_train_timesteps, (latents.shape[0],), device=latents.device)
9noisy = scheduler.add_noise(latents, noise, t)
10cond = text_encoder(input_ids)[0] # embeddings of both prompts
11
12pred = unet(noisy, t, encoder_hidden_states=cond).sample # epsilon prediction
13pred_subj, pred_prior = pred.chunk(2)
14noise_subj, noise_prior = noise.chunk(2)
15
16loss = F.mse_loss(pred_subj.float(), noise_subj.float()) # reconstruction
17prior_loss = F.mse_loss(pred_prior.float(), noise_prior.float()) # prior preservation
18(loss + 1.0 * prior_loss).backward() # lambda = 1
19optimizer.step(); optimizer.zero_grad() # lr 5e-6 for SD
One DreamBooth training step with prior preservation, following the structure of diffusers' train_dreambooth.py (not run here: needs torch, diffusers and a GPU)

A practical recipe

  1. 1Collect 3–5 (up to about 20) varied photos: different backgrounds, angles and lighting; crop to the model's resolution.
  2. 2Pick a rare identifier and the right class noun: "a photo of sks backpack", not "a photo of sks object".
  3. 3Generate the class images once with the frozen model and the class prompt (diffusers' script defaults to 100 of them; the paper used about 1,000).
  4. 4Train with λ = 1 and a low learning rate (5 × 10⁻⁶ for full fine-tuning of SD), checking samples every few hundred steps.
  5. 5Stop when new-context prompts keep the identity; more steps start reproducing the training backgrounds and poses.

Use when

  • +Highest subject fidelity of the early personalization methods (DINO 0.696 vs 0.569 for Textual Inversion).
  • +Needs only 3–5 photos and minutes of GPU time.
  • +Works with any class the base model knows; the class noun supplies the prior.

Avoid when

  • −Full fine-tuning produces a whole new checkpoint per subject (gigabytes) unless combined with LoRA.
  • −Must generate and train on class images to avoid language drift.
  • −Several subjects in one model tend to blend; multi-subject composition needs extra methods.

Check yourself

You fine-tune on 4 photos of your dog captioned "a sks dog" with no prior-preservation term. Afterwards "a dog" mostly draws your dog. What happened?

Check yourself

Why does DreamBooth avoid an identifier like "xxy5syt00"?

Remember

  • ✓DreamBooth fine-tunes all weights of a text-to-image diffusion model on 3–5 photos captioned "a [V] class".
  • ✓[V] is a rare token with a weak prior ("sks" in Stable Diffusion); the class noun reuses the model's knowledge of the class.
  • ✓Prior preservation: λ · reconstruction loss on frozen-model images of "a class" (λ = 1) prevents language drift and keeps outputs diverse.
  • ✓Paper settings: about 1,000 steps, lr 10⁻⁵ (Imagen) or 5 × 10⁻⁶ (SD), about 5 minutes on one A100 or TPUv4.
  • ✓DINO 0.696 vs 0.569 for Textual Inversion on the paper's 30-subject benchmark; overfitting shows up as copied backgrounds and poses.
← back to Generative AI roadmap