Definition
CLIP score measures how well a text matches an image as the clipped, rescaled cosine similarity of their CLIP embeddings. It was introduced as CLIPScore by Hessel, Holtzman, Forbes, Le Bras and Choi, "CLIPScore: A Reference-free Evaluation Metric for Image Captioning" (EMNLP 2021), and is now also the standard prompt-alignment metric for text-to-image models.
Caption metrics such as BLEU and CIDEr compare a candidate caption with human reference captions (see BLEU, ROUGE and METEOR). They need references, and they punish a correct caption that uses different words. CLIP was trained on hundreds of millions of image–text pairs to put matching pairs close together, so it can compare the caption with the image directly.
In text-to-image work the roles flip: the prompt is the text, the generated image is scored, and the mean over a prompt set measures prompt adherence. It complements FID and Inception Score, which measure realism but ignore the prompt. Papers sweep the classifier-free guidance scale and plot FID against CLIP score, because more guidance usually raises alignment and, past a point, worsens FID.
The score
- frozen CLIP text and image encoders (ViT-B/32 in the paper); the caption is prefixed with "A photo depicts"
- cosine of the two embeddings, in [−1, 1]; for real pairs it rarely exceeds about 0.4
- a rescaling constant chosen so scores spread over roughly [0, 1]; it does not change rankings
- negative cosines are clipped to 0, the floor for an unrelated text
A corpus-level score is the mean over all (caption, image) pairs. When reference captions exist, RefCLIPScore adds them through a harmonic mean, so a caption must match both the image and the references:
- human reference captions, embedded with the same text encoder
- harmonic mean: low if either part is low
Setting and conventions
- Paper setup
- CLIP ViT-B/32, prompt prefix "A photo depicts", w = 2.5; no references and no training on caption data
- Paper finding
- on literal image-description benchmarks such as Flickr8K-Expert, CLIP-S correlated with human judgements better than reference-based metrics like CIDEr and SPICE; it did worse on news captions that need context beyond the image
- ×100 convention
- text-to-image papers and
torchmetricsreport max(100 · cos, 0), so 0.31 is 31.0; the backbone (often ViT-L/14) must be stated too - Text length
- CLIP's text encoder has a 77-token context; longer prompts are truncated, and their tail is never scored
- Data filtering
- LAION-400M kept web pairs with ViT-B/32 cosine ≥ 0.3; the same score is used to filter synthetic training pairs
- Subject fidelity
- DreamBooth (Ruiz et al., 2023) reports CLIP-T (prompt–image, this score) and CLIP-I (image–image) with DINO similarity for the subject
Worked example
| Candidate | cos(c, v) | CLIP-S | max cos(c, r) | RefCLIP-S |
|---|---|---|---|---|
| "a dog leaps for a frisbee" | 0.31 | 2.5 × 0.31 = 0.775 | 0.80 | 2·0.775·0.80 / 1.575 = 0.787 |
| "a dog on a beach" | 0.24 | 0.600 | 0.62 | 2·0.600·0.62 / 1.220 = 0.610 |
| "a plate of pasta" | −0.03 | max(−0.075, 0) = 0 | 0.05 | 0 (H with a zero is 0) |
The ranking is set by the cosines alone, and only scales them. A difference of 0.07 in cosine is 0.175 in CLIP-S and 7 points on the ×100 scale, so the same gap looks large or small depending on the convention. Always state which one you report.
1import torch2from PIL import Image3from transformers import CLIPModel, CLIPProcessor45name = "openai/clip-vit-base-patch32" # the paper's backbone6model = CLIPModel.from_pretrained(name).eval()7proc = CLIPProcessor.from_pretrained(name)89@torch.no_grad()10def clip_score(images, captions, w=2.5, prefix="A photo depicts "):11 """images[i] is scored against captions[i]; returns one score per pair."""12 inp = proc(text=[prefix + c for c in captions], images=images,13 return_tensors="pt", padding=True, truncation=True) # truncates at 77 tokens14 v = model.get_image_features(pixel_values=inp["pixel_values"])15 t = model.get_text_features(input_ids=inp["input_ids"], attention_mask=inp["attention_mask"])16 v, t = v / v.norm(dim=-1, keepdim=True), t / t.norm(dim=-1, keepdim=True)17 return w * (v * t).sum(-1).clamp(min=0)1819imgs = [Image.open(p).convert("RGB") for p in ["gen_000.png", "gen_001.png"]]20s = clip_score(imgs, ["a red cube on a blue sphere", "an astronaut riding a horse"])21print(s.tolist(), s.mean().item()) # report the mean over the prompt setRunning a text-to-image evaluation
- 1Fix a prompt set (COCO captions, or a suite with counts, colours and relations) and generate a fixed number of images per prompt with recorded seeds.
- 2Pick one CLIP checkpoint and preprocessing, and state them; do not use the model that guided or filtered the generator.
- 3Embed prompt and image, L2-normalise, take the cosine, clip at 0 and scale (, or 100 for the common convention).
- 4Average over images and prompts; bootstrap over prompts for a confidence interval.
- 5Report it next to FID for the same samples, and check a sample of low- and high-scoring pairs by eye.
Where it's used
Caption evaluation
reference-free scoring of captioning models, with RefCLIP-S when references exist (see Image captioning)
Text-to-image benchmarks
prompt adherence on COCO captions or prompt suites, plotted against FID across guidance scales (see Text-to-image generation)
Data filtering
dropping mismatched web or synthetic image–text pairs below a cosine threshold (see Synthetic data generation)
Best-of-n reranking
picking the generated image that best matches the prompt, with the risk of optimising the metric
Check yourself
Model A scores 0.78 CLIP-S (w = 2.5, ViT-B/32) and model B scores 31.5 (torchmetrics, ViT-L/14) on different prompt sets. Which is better aligned?
Check yourself
For the prompt "a blue cube on a red sphere", the generated image shows a red cube on a blue sphere. What will CLIP score most likely do?
Remember
- ✓CLIP-S , reference-free; RefCLIP-S takes the harmonic mean with the best caption–reference cosine.
- ✓Paper setup: CLIP ViT-B/32, prefix "A photo depicts". Text-to-image work usually reports max(100·cos, 0) with a stated backbone.
- ✓Measures alignment, not realism: report it alongside FID, typically as a curve over guidance scale.
- ✓Blind spots: attribute binding, counting, spatial relations, text past 77 tokens; use VQA-based metrics or humans for those.
- ✓Compare only within one checkpoint, prompt set and scale, and never on a model guided or filtered by the same CLIP.