Textual inversion

Textual inversion (Gal et al., ICLR 2023) personalizes a frozen text-to-image model by learning one new word embedding v* from 3–5 images with the usual denoising loss: 768 numbers (3 KB) in Stable Diffusion v1, usable in any prompt.

7 min read669 wordsadvanced0 viewsupdated 2026-10-21

Definition

Textual inversion is a personalization method that adds one new pseudo-word S∗S_* to a frozen text-to-image model and learns only its embedding vector, so that prompts containing S∗S_* render a concept shown in 3–5 example images. It was introduced by Gal, Alaluf, Atzmon, Patashnik, Bermano, Chechik and Cohen-Or (Tel Aviv University and NVIDIA) in "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion" (arXiv August 2022, ICLR 2023).

The name says what it does. Instead of inverting an image into a GAN's latent space, it inverts a set of images into the model's text embedding space: it finds the input vector that makes the frozen model reproduce them. No weight of the text encoder, U-Net or VAE changes. That makes the result tiny, safe to share and stackable: several learned words can sit in one prompt next to ordinary words.

It is the lightest of the three classic customization tools for diffusion models. DreamBooth fine-tunes all weights for higher fidelity. LoRA adds low-rank weight updates. ControlNet adds spatial conditioning rather than new concepts. Textual inversion is the right tool when the base model can already draw something close and you only need a handle for it: a style, a texture, a product's look.

The optimisation

The placeholder gets a new token ID and a new row vv in the embedding table. Training reuses the latent-diffusion loss of Latent diffusion models unchanged, with captions built from neutral templates such as "a photo of S∗S_*" or "a rendition of S∗S_*" (taken from CLIP's ImageNet prompt templates):

v∗=arg⁡min⁡v  Ez∼E(x), y, ϵ∼N(0,I), t[∥ϵ−ϵθ(zt, t, cθ(y;v))∥22]v_*=\arg\min_{v}\;\mathbb E_{z\sim\mathcal E(x),\,y,\,\epsilon\sim\mathcal N(0,I),\,t}\Big[\big\|\epsilon-\epsilon_\theta\big(z_t,\,t,\,c_\theta(y;v)\big)\big\|_2^2\Big]
vv
embedding of S∗S_*: the only trainable parameters (768 numbers for Stable Diffusion v1's CLIP ViT-L/14 text encoder)
E(x)\mathcal E(x)
frozen VAE encoder applied to one of the 3–5 concept images
yy
a sampled template caption containing S∗S_*
cθ,  ϵθc_\theta,\;\epsilon_\theta
frozen text encoder and frozen denoising U-Net
ztz_t
the latent noised to timestep tt

vv is initialised with the embedding of a single-word coarse descriptor of the concept ("cat", "sculpture"). The optimisation is non-convex in vv: the frozen model only renders a few regions of embedding space usefully, and a distant start can sit on a flat plateau. The learned vector usually ends up away from every real word. It is a point the model understands but the tokenizer never produces.

Textual inversion (Gal et al., ICLR 2023): learn one word, freeze the modeltemplate"a photo of S*""a rendition of S*"tokenizera photo of+ new ID S*embedding tablea [ .01 −.02 …]photo [−.03 .02 …]of [ .00 .01 …]S* v* (init: toy)text encoderfrozenSD v1: CLIP, 768-d3–5 photos xof the subjectVAE encoder (frozen)z = E(x), t ~ U, ε ~ N(0, I)z_t = αₜ z + σₜ εcU-Net ε_θ(z_t, t, c)frozen: no weight changespredicts the added noiseL = ‖ε − ε_θ(z_t, t, c)‖²same loss as LDM training∇ flows to v* onlyresult: one vector, 768 floats = 3,072 bytes in fp32; the checkpoint is untoucheduse it in any prompt: "an oil painting of S*", "S* on the beach"
fig.The prompt "a photo of S*" is tokenized; in the embedding table only the new S* row (initialised from "toy") is trainable. The frozen text encoder and frozen U-Net predict the noise added to a subject photo's latent, and the gradient of the standard noise-prediction loss reaches v* alone. The output is 768 floats, 3,072 bytes in fp32.

Paper setup and findings

Base model
the 1.4B-parameter latent diffusion model trained on LAION-400M, which has its own transformer text encoder; later ported to Stable Diffusion
Optimisation
5,000 steps, 2 × V100, batch size 4, base learning rate 0.005 scaled by GPUs × batch to 0.04
Evaluation
reconstruction: CLIP image similarity of generations to the concept images; editability: CLIP similarity of generations to the prompt text
More vectors
learning several vectors per concept raised reconstruction but lowered editability; one vector was the chosen balance
Uses shown
objects in new scenes, styles ("a painting in the style of S*"), composing two learned words, and steering away from dataset bias

On the DreamBooth benchmark (30 subjects, Stable Diffusion), Textual Inversion scored DINO 0.569, CLIP-I 0.780 and CLIP-T 0.255, against 0.668, 0.803 and 0.305 for DreamBooth. One vector fed through a frozen model cannot store every detail of a specific object. It does best on styles and textures, and on concepts close to what the model already draws.

only v moves: v ← v − η ∇ᵥ mean‖g(v) − xᵢ‖², generator g frozen, η = 0.15toycatdogteapotcarsculptureword-embedding plane · brighter = lower loss● initializer word ● learned v for S*toycatdogteapotcarsculptureimage features: what g draws□ 4 subject photos ● g(v)loss (log) vs step, 0–400step 100: loss 0.0089 (start 0.1978)nearest word to v: "toy" at distance 0.557loss < 0.0095 after 21 steps · trainable: 2 numbers
initializer
fig.A frozen toy generator renders an embedding as an attention blend of six known concepts; gradient descent moves only v toward four photos of a cat-shaped toy. Default (initialise from "toy", step 100): loss 0.0089 from 0.1978 (−95.5%), below 0.0095 after 21 steps, and v ends 0.557 from the nearest real word. From "teapot" it takes 62 steps; from "car" v sits on a plateau (loss 2.6195 at step 100) and needs 119.

The loss never reaches the 0.0066 that a perfect match to the photos' mean would give. The best rendering, (0.532, 0.614), is a blend of concepts the model already knows; the photos' mean, (0.5625, 0.65), is not quite reachable. A real model has the same limit: textual inversion can only find what the frozen generator can already express.

Worked example: what gets trained and stored

QuantityComputationValue
Trainable parameters1 token × 768 dims768
Stored size (fp32)768 × 4 bytes3,072 bytes
Full U-Net, as DreamBooth trains it860M × 2 bytes (fp16)1.72 GB
Ratio U-Net : embedding860,000,000 / 768≈ 1.12 million
Effective learning rate0.005 × 2 GPUs × batch 40.04
Stable Diffusion v1 sizes (U-Net about 860M parameters) and the paper's learning-rate scaling
python
1import torch
2from diffusers import StableDiffusionPipeline
3
4pipe = StableDiffusionPipeline.from_pretrained(
5 "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16
6).to("cuda")
7pipe.load_textual_inversion("sd-concepts-library/cat-toy") # adds the token <cat-toy>
8
9tok_id = pipe.tokenizer.convert_tokens_to_ids("<cat-toy>")
10v = pipe.text_encoder.get_input_embeddings().weight[tok_id]
11print(v.shape) # torch.Size([768])
12
13image = pipe("an oil painting of a <cat-toy> on a beach",
14 num_inference_steps=50).images[0]
Loading a published embedding with diffusers and inspecting it (not run here: needs torch, diffusers and a GPU). The sd-concepts-library hosts community embeddings such as <cat-toy>.

Training scripts usually make the whole embedding matrix trainable, for simplicity, and copy every original row back after each optimiser step. Only the new row really changes. The diffusers example does it this way.

Use when

  • +Model weights untouched: no forgetting, no language drift, nothing to merge.
  • +A few kilobytes per concept; many concepts can be combined in one prompt.
  • +Works with any checkpoint that shares the same text encoder and vocabulary.

Avoid when

  • −Lower subject fidelity than weight-changing methods (DINO 0.569 vs 0.668 for DreamBooth on SD).
  • −Slow for what it learns: thousands of steps to fit 768 numbers.
  • −Limited to what the frozen model can already render; fine shapes, text and faces are hard.

Later methods sit between the extremes. Custom Diffusion (Kumari et al., CVPR 2023) learns a new token plus the key and value projections of the U-Net's cross-attention layers. That raises fidelity while training a small fraction of the weights.

Check yourself

During textual-inversion training, which parameters receive gradient updates?

Check yourself

You learned v* for a mug on Stable Diffusion v1.5 and load it into Stable Diffusion 2.1. What happens?

Remember

  • ✓Textual inversion learns one new token embedding v* for a frozen text-to-image model from 3–5 images.
  • ✓Loss: the standard noise-prediction loss ∥ϵ−ϵθ(zt,t,cθ(y;v))∥2\|\epsilon-\epsilon_\theta(z_t,t,c_\theta(y;v))\|^2, with template captions like "a photo of S∗S_*".
  • ✓Initialise from a coarse descriptor word; the optimum usually lies away from every real word.
  • ✓SD v1: 768 floats, 3,072 bytes in fp32; tied to its text encoder.
  • ✓More vectors per concept: better reconstruction, worse editability; fidelity trails DreamBooth (DINO 0.569 vs 0.668).
← back to Generative AI roadmap