# Textual inversion

> Textual inversion (Gal et al., ICLR 2023) personalizes a frozen text-to-image model by learning one new word embedding v* from 3–5 images with the usual denoising loss: 768 numbers (3 KB) in Stable Diffusion v1, usable in any prompt.

Source: https://www.mlfoundry.in/learn/textual-inversion  
Section: Generative AI › Customization  
Level: advanced · 7 min read · updated 2026-10-21

**Textual inversion:** is a personalization method that adds one new pseudo-word $S_*$ to a frozen text-to-image model and learns only its embedding vector, so that prompts containing $S_*$ render a concept shown in 3–5 example images. It was introduced by Gal, Alaluf, Atzmon, Patashnik, Bermano, Chechik and Cohen-Or (Tel Aviv University and NVIDIA) in "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion" (arXiv August 2022, ICLR 2023).

The name says what it does. Instead of inverting an image into a GAN's latent space, it inverts a set of images into the model's *text* embedding space: it finds the input vector that makes the frozen model reproduce them. No weight of the text encoder, U-Net or VAE changes. That makes the result tiny, safe to share and stackable: several learned words can sit in one prompt next to ordinary words.

It is the lightest of the three classic customization tools for diffusion models. **DreamBooth** fine-tunes all weights for higher fidelity. **LoRA** adds low-rank weight updates. **ControlNet** adds spatial conditioning rather than new concepts. Textual inversion is the right tool when the base model can already draw something close and you only need a handle for it: a style, a texture, a product's look.

## The optimisation

The placeholder gets a new token ID and a new row $v$ in the embedding table. Training reuses the latent-diffusion loss of **Latent diffusion models** unchanged, with captions built from neutral templates such as "a photo of $S_*$" or "a rendition of $S_*$" (taken from CLIP's ImageNet prompt templates):

$$
v_*=\arg\min_{v}\;\mathbb E_{z\sim\mathcal E(x),\,y,\,\epsilon\sim\mathcal N(0,I),\,t}\Big[\big\|\epsilon-\epsilon_\theta\big(z_t,\,t,\,c_\theta(y;v)\big)\big\|_2^2\Big]
$$

- $v$: embedding of $S_*$: the only trainable parameters (768 numbers for Stable Diffusion v1's CLIP ViT-L/14 text encoder)
- $\mathcal E(x)$: frozen VAE encoder applied to one of the 3–5 concept images
- $y$: a sampled template caption containing $S_*$
- $c_\theta,\;\epsilon_\theta$: frozen text encoder and frozen denoising U-Net
- $z_t$: the latent noised to timestep $t$

$v$ is initialised with the embedding of a single-word coarse descriptor of the concept ("cat", "sculpture"). The optimisation is non-convex in $v$: the frozen model only renders a few regions of embedding space usefully, and a distant start can sit on a flat plateau. The learned vector usually ends up away from every real word. It is a point the model understands but the tokenizer never produces.

> Figure: The prompt "a photo of S*" is tokenized; in the embedding table only the new S* row (initialised from "toy") is trainable. The frozen text encoder and frozen U-Net predict the noise added to a subject photo's latent, and the gradient of the standard noise-prediction loss reaches v* alone. The output is 768 floats, 3,072 bytes in fp32.

**Paper setup and findings**

- **Base model:** the 1.4B-parameter latent diffusion model trained on LAION-400M, which has its own transformer text encoder; later ported to Stable Diffusion
- **Optimisation:** 5,000 steps, 2 × V100, batch size 4, base learning rate 0.005 scaled by GPUs × batch to 0.04
- **Evaluation:** reconstruction: CLIP image similarity of generations to the concept images; editability: CLIP similarity of generations to the prompt text
- **More vectors:** learning several vectors per concept raised reconstruction but lowered editability; one vector was the chosen balance
- **Uses shown:** objects in new scenes, styles ("a painting in the style of S*"), composing two learned words, and steering away from dataset bias

On the DreamBooth benchmark (30 subjects, Stable Diffusion), Textual Inversion scored DINO 0.569, CLIP-I 0.780 and CLIP-T 0.255, against 0.668, 0.803 and 0.305 for DreamBooth. One vector fed through a frozen model cannot store every detail of a specific object. It does best on styles and textures, and on concepts close to what the model already draws.

> Figure: A frozen toy generator renders an embedding as an attention blend of six known concepts; gradient descent moves only v toward four photos of a cat-shaped toy. Default (initialise from "toy", step 100): loss 0.0089 from 0.1978 (−95.5%), below 0.0095 after 21 steps, and v ends 0.557 from the nearest real word. From "teapot" it takes 62 steps; from "car" v sits on a plateau (loss 2.6195 at step 100) and needs 119.

The loss never reaches the 0.0066 that a perfect match to the photos' mean would give. The best rendering, (0.532, 0.614), is a blend of concepts the model already knows; the photos' mean, (0.5625, 0.65), is not quite reachable. A real model has the same limit: textual inversion can only find what the frozen generator can already express.

## Worked example: what gets trained and stored

Stable Diffusion v1 sizes (U-Net about 860M parameters) and the paper's learning-rate scaling

| Quantity | Computation | Value |
| --- | --- | --- |
| Trainable parameters | 1 token × 768 dims | 768 |
| Stored size (fp32) | 768 × 4 bytes | 3,072 bytes |
| Full U-Net, as DreamBooth trains it | 860M × 2 bytes (fp16) | 1.72 GB |
| Ratio U-Net : embedding | 860,000,000 / 768 | ≈ 1.12 million |
| Effective learning rate | 0.005 × 2 GPUs × batch 4 | 0.04 |

```python
import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")
pipe.load_textual_inversion("sd-concepts-library/cat-toy")    # adds the token <cat-toy>

tok_id = pipe.tokenizer.convert_tokens_to_ids("<cat-toy>")
v = pipe.text_encoder.get_input_embeddings().weight[tok_id]
print(v.shape)                                                # torch.Size([768])

image = pipe("an oil painting of a <cat-toy> on a beach",
             num_inference_steps=50).images[0]
```

Loading a published embedding with diffusers and inspecting it (not run here: needs torch, diffusers and a GPU). The sd-concepts-library hosts community embeddings such as <cat-toy>.

Training scripts usually make the whole embedding matrix trainable, for simplicity, and copy every original row back after each optimiser step. Only the new row really changes. The diffusers example does it this way.

> **An embedding belongs to one text encoder:** A vector learned for Stable Diffusion v1 lives in CLIP ViT-L/14's 768-d space. Stable Diffusion 2 uses an OpenCLIP encoder with 1,024-d embeddings, so the file does not load there at all. Even between fine-tunes that share an encoder, results shift, because the U-Net that interprets the vector has changed.

> **Negative embeddings:** Because the result is just a word, it can go in a negative prompt. Communities train embeddings on images with a failure mode (bad hands, blur) and use them to push generations away from it with classifier-free guidance.

**Strengths**

- Model weights untouched: no forgetting, no language drift, nothing to merge.
- A few kilobytes per concept; many concepts can be combined in one prompt.
- Works with any checkpoint that shares the same text encoder and vocabulary.

**Limitations**

- Lower subject fidelity than weight-changing methods (DINO 0.569 vs 0.668 for DreamBooth on SD).
- Slow for what it learns: thousands of steps to fit 768 numbers.
- Limited to what the frozen model can already render; fine shapes, text and faces are hard.

Later methods sit between the extremes. Custom Diffusion (Kumari et al., CVPR 2023) learns a new token plus the key and value projections of the U-Net's cross-attention layers. That raises fidelity while training a small fraction of the weights.

**Question:** During textual-inversion training, which parameters receive gradient updates?

- A. The U-Net's cross-attention layers
- B. All of the text encoder
- C. Only the embedding vector of the new placeholder token
- D. The VAE decoder

**Answer:** C. Only the embedding vector of the new placeholder token — Everything else is frozen. The gradient of the noise-prediction loss flows back through the U-Net and text encoder, but only the embedding row for $S_*$ is updated: 768 numbers in SD v1.

**Question:** You learned v* for a mug on Stable Diffusion v1.5 and load it into Stable Diffusion 2.1. What happens?

- A. It works the same, since the U-Nets are similar
- B. It fails: SD 2 uses an OpenCLIP text encoder with 1,024-d embeddings, so the 768-d vector has no meaning there
- C. It works but at lower resolution
- D. It loads but only with a negative prompt

**Answer:** B. It fails: SD 2 uses an OpenCLIP text encoder with 1,024-d embeddings, so the 768-d vector has no meaning there — The vector is a point in one specific embedding space. A different text encoder has a different space, and here even a different dimension. Retrain the embedding for each text encoder.

**Key points**

- Textual inversion learns one new token embedding v* for a frozen text-to-image model from 3–5 images.
- Loss: the standard noise-prediction loss $\|\epsilon-\epsilon_\theta(z_t,t,c_\theta(y;v))\|^2$, with template captions like "a photo of $S_*$".
- Initialise from a coarse descriptor word; the optimum usually lies away from every real word.
- SD v1: 768 floats, 3,072 bytes in fp32; tied to its text encoder.
- More vectors per concept: better reconstruction, worse editability; fidelity trails DreamBooth (DINO 0.569 vs 0.668).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
