Definition
Textual inversion is a personalization method that adds one new pseudo-word to a frozen text-to-image model and learns only its embedding vector, so that prompts containing render a concept shown in 3–5 example images. It was introduced by Gal, Alaluf, Atzmon, Patashnik, Bermano, Chechik and Cohen-Or (Tel Aviv University and NVIDIA) in "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion" (arXiv August 2022, ICLR 2023).
The name says what it does. Instead of inverting an image into a GAN's latent space, it inverts a set of images into the model's text embedding space: it finds the input vector that makes the frozen model reproduce them. No weight of the text encoder, U-Net or VAE changes. That makes the result tiny, safe to share and stackable: several learned words can sit in one prompt next to ordinary words.
It is the lightest of the three classic customization tools for diffusion models. DreamBooth fine-tunes all weights for higher fidelity. LoRA adds low-rank weight updates. ControlNet adds spatial conditioning rather than new concepts. Textual inversion is the right tool when the base model can already draw something close and you only need a handle for it: a style, a texture, a product's look.
The optimisation
The placeholder gets a new token ID and a new row in the embedding table. Training reuses the latent-diffusion loss of Latent diffusion models unchanged, with captions built from neutral templates such as "a photo of " or "a rendition of " (taken from CLIP's ImageNet prompt templates):
- embedding of : the only trainable parameters (768 numbers for Stable Diffusion v1's CLIP ViT-L/14 text encoder)
- frozen VAE encoder applied to one of the 3–5 concept images
- a sampled template caption containing
- frozen text encoder and frozen denoising U-Net
- the latent noised to timestep
is initialised with the embedding of a single-word coarse descriptor of the concept ("cat", "sculpture"). The optimisation is non-convex in : the frozen model only renders a few regions of embedding space usefully, and a distant start can sit on a flat plateau. The learned vector usually ends up away from every real word. It is a point the model understands but the tokenizer never produces.
Paper setup and findings
- Base model
- the 1.4B-parameter latent diffusion model trained on LAION-400M, which has its own transformer text encoder; later ported to Stable Diffusion
- Optimisation
- 5,000 steps, 2 × V100, batch size 4, base learning rate 0.005 scaled by GPUs × batch to 0.04
- Evaluation
- reconstruction: CLIP image similarity of generations to the concept images; editability: CLIP similarity of generations to the prompt text
- More vectors
- learning several vectors per concept raised reconstruction but lowered editability; one vector was the chosen balance
- Uses shown
- objects in new scenes, styles ("a painting in the style of S*"), composing two learned words, and steering away from dataset bias
On the DreamBooth benchmark (30 subjects, Stable Diffusion), Textual Inversion scored DINO 0.569, CLIP-I 0.780 and CLIP-T 0.255, against 0.668, 0.803 and 0.305 for DreamBooth. One vector fed through a frozen model cannot store every detail of a specific object. It does best on styles and textures, and on concepts close to what the model already draws.
The loss never reaches the 0.0066 that a perfect match to the photos' mean would give. The best rendering, (0.532, 0.614), is a blend of concepts the model already knows; the photos' mean, (0.5625, 0.65), is not quite reachable. A real model has the same limit: textual inversion can only find what the frozen generator can already express.
Worked example: what gets trained and stored
| Quantity | Computation | Value |
|---|---|---|
| Trainable parameters | 1 token × 768 dims | 768 |
| Stored size (fp32) | 768 × 4 bytes | 3,072 bytes |
| Full U-Net, as DreamBooth trains it | 860M × 2 bytes (fp16) | 1.72 GB |
| Ratio U-Net : embedding | 860,000,000 / 768 | ≈ 1.12 million |
| Effective learning rate | 0.005 × 2 GPUs × batch 4 | 0.04 |
1import torch2from diffusers import StableDiffusionPipeline34pipe = StableDiffusionPipeline.from_pretrained(5 "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float166).to("cuda")7pipe.load_textual_inversion("sd-concepts-library/cat-toy") # adds the token <cat-toy>89tok_id = pipe.tokenizer.convert_tokens_to_ids("<cat-toy>")10v = pipe.text_encoder.get_input_embeddings().weight[tok_id]11print(v.shape) # torch.Size([768])1213image = pipe("an oil painting of a <cat-toy> on a beach",14 num_inference_steps=50).images[0]Training scripts usually make the whole embedding matrix trainable, for simplicity, and copy every original row back after each optimiser step. Only the new row really changes. The diffusers example does it this way.
Use when
- +Model weights untouched: no forgetting, no language drift, nothing to merge.
- +A few kilobytes per concept; many concepts can be combined in one prompt.
- +Works with any checkpoint that shares the same text encoder and vocabulary.
Avoid when
- −Lower subject fidelity than weight-changing methods (DINO 0.569 vs 0.668 for DreamBooth on SD).
- −Slow for what it learns: thousands of steps to fit 768 numbers.
- −Limited to what the frozen model can already render; fine shapes, text and faces are hard.
Later methods sit between the extremes. Custom Diffusion (Kumari et al., CVPR 2023) learns a new token plus the key and value projections of the U-Net's cross-attention layers. That raises fidelity while training a small fraction of the weights.
Check yourself
During textual-inversion training, which parameters receive gradient updates?
Check yourself
You learned v* for a mug on Stable Diffusion v1.5 and load it into Stable Diffusion 2.1. What happens?
Remember
- ✓Textual inversion learns one new token embedding v* for a frozen text-to-image model from 3–5 images.
- ✓Loss: the standard noise-prediction loss , with template captions like "a photo of ".
- ✓Initialise from a coarse descriptor word; the optimum usually lies away from every real word.
- ✓SD v1: 768 floats, 3,072 bytes in fp32; tied to its text encoder.
- ✓More vectors per concept: better reconstruction, worse editability; fidelity trails DreamBooth (DINO 0.569 vs 0.668).