CLIP score

CLIPScore (Hessel et al., 2021) = 2.5 · max(cos(text, image), 0) from a frozen CLIP: a reference-free text–image alignment score used for captions and text-to-image prompts, blind to binding, counts and relations.

7 min read661 wordscore0 viewsupdated 2026-10-21

Definition

CLIP score measures how well a text matches an image as the clipped, rescaled cosine similarity of their CLIP embeddings. It was introduced as CLIPScore by Hessel, Holtzman, Forbes, Le Bras and Choi, "CLIPScore: A Reference-free Evaluation Metric for Image Captioning" (EMNLP 2021), and is now also the standard prompt-alignment metric for text-to-image models.

Caption metrics such as BLEU and CIDEr compare a candidate caption with human reference captions (see BLEU, ROUGE and METEOR). They need references, and they punish a correct caption that uses different words. CLIP was trained on hundreds of millions of image–text pairs to put matching pairs close together, so it can compare the caption with the image directly.

In text-to-image work the roles flip: the prompt is the text, the generated image is scored, and the mean over a prompt set measures prompt adherence. It complements FID and Inception Score, which measure realism but ignore the prompt. Papers sweep the classifier-free guidance scale and plot FID against CLIP score, because more guidance usually raises alignment and, past a point, worsens FID.

The score

CLIP-S(c,v)=w⋅max⁡(cos⁡(ET(c), EI(v)), 0),w=2.5\text{CLIP-S}(c,v)=w\cdot\max\big(\cos(E_T(c),\,E_I(v)),\,0\big),\qquad w=2.5
ET, EIE_T,\,E_I
frozen CLIP text and image encoders (ViT-B/32 in the paper); the caption is prefixed with "A photo depicts"
cos⁡\cos
cosine of the two embeddings, in [−1, 1]; for real pairs it rarely exceeds about 0.4
ww
a rescaling constant chosen so scores spread over roughly [0, 1]; it does not change rankings
max⁡(⋅,0)\max(\cdot,0)
negative cosines are clipped to 0, the floor for an unrelated text

A corpus-level score is the mean over all (caption, image) pairs. When reference captions RR exist, RefCLIPScore adds them through a harmonic mean, so a caption must match both the image and the references:

RefCLIP-S(c,R,v)=H(CLIP-S(c,v),  max⁡(max⁡r∈Rcos⁡(ET(c),ET(r)), 0)),H(a,b)=2aba+b\text{RefCLIP-S}(c,R,v)=\mathrm{H}\Big(\text{CLIP-S}(c,v),\;\max\big(\max_{r\in R}\cos(E_T(c),E_T(r)),\,0\big)\Big),\qquad \mathrm{H}(a,b)=\frac{2ab}{a+b}
r∈Rr\in R
human reference captions, embedded with the same text encoder
H\mathrm H
harmonic mean: low if either part is low
CLIPScore: no reference caption needed, one frozen CLIP (ViT-B/32 in the paper)image vdog leaps for frisbeecandidate c"A photo depicts" + cimage encoderfrozentext encoder≤ 77 tokensv ∈ ℝ⁵¹²‖v‖ = 1c ∈ ℝ⁵¹²‖c‖ = 1cos(c, v) = 0.31max(·, 0) × 2.5CLIP-S = 0.775higher = betterRefCLIPScore: when references existreferences R"a dog catches a frisbee", …maxᵣ cos(c, r)= 0.80, clip at 0harmonic meanH(0.775, 0.80) = 0.787text-to-image use: score each (prompt, generated image) pair, average over a prompt set"a red cube on a blue sphere"cos 0.31 → 0.775"an astronaut riding a horse"cos 0.27 → 0.675"three cats on a sofa"cos 0.22 → 0.550mean CLIP-S 0.667= 26.7 on the ×100 scaleblind spots: attribute binding, counting, spatial relations, text over 77 tokensscores depend on the CLIP checkpoint: compare only within one backbone and one scale
fig.The caption, prefixed with "A photo depicts", and the image go through the frozen CLIP towers and are compared by cosine. A cosine of 0.31 gives CLIP-S = 2.5 × 0.31 = 0.775. With references, the best caption-to-reference cosine 0.80 is combined by a harmonic mean: 2 · 0.775 · 0.80 / 1.575 = 0.787. For text-to-image evaluation three prompts with cosines 0.31, 0.27 and 0.22 give a mean CLIP-S of 0.667, which the ×100 convention reports as 26.7.

Setting and conventions

Paper setup
CLIP ViT-B/32, prompt prefix "A photo depicts", w = 2.5; no references and no training on caption data
Paper finding
on literal image-description benchmarks such as Flickr8K-Expert, CLIP-S correlated with human judgements better than reference-based metrics like CIDEr and SPICE; it did worse on news captions that need context beyond the image
×100 convention
text-to-image papers and torchmetrics report max(100 · cos, 0), so 0.31 is 31.0; the backbone (often ViT-L/14) must be stated too
Text length
CLIP's text encoder has a 77-token context; longer prompts are truncated, and their tail is never scored
Data filtering
LAION-400M kept web pairs with ViT-B/32 cosine ≥ 0.3; the same score is used to filter synthetic training pairs
Subject fidelity
DreamBooth (Ruiz et al., 2023) reports CLIP-T (prompt–image, this score) and CLIP-I (image–image) with DINO similarity for the subject
CLIP-S(c, v) = w · max(cos(c, v), 0), w = 2.5 (Hessel et al.)image contentdog 1.0ball 0.8red 0.6grass 0.7+ noise 0.5+ image offset 2.5text tower: sum of wordvectors + text offset,then L2-normalisedrank · candidate captionCLIP-S (0 … 2.5)cos → score1. a dog with a red ball on grass0.297 → 0.7432. a red dog with a ball on grass0.297 → 0.7433. a dog on grass0.241 → 0.6015. a dog with a blue ball0.117 → 0.2936. a cat on a sofa-0.027 → 0.0004. a blue ball on grass0.160 → 0.400top 2 captions tie at cos 0.297: same words, different binding, same score1 negative cosine clipped to 0 · offset 2.5 caps every cosine: compare rankings, not levels
image
scale w
fig.A toy CLIP space: an image is a weighted sum of concept vectors plus noise, a caption is the normalised sum of its word vectors, and each tower adds its own offset (the modality gap). At the defaults (dog-and-red-ball image, gap 2.5, noise 0.5) the correct caption and its attribute-swapped twin "a red dog with a ball on grass" tie at cos 0.297, CLIP-S 0.743; "a dog on grass" scores 0.241 → 0.601 and "a cat on a sofa" has cos −0.027, clipped to 0. Gap 0 lifts the top cosine to 0.901; gap 4 cuts it to 0.145, with the same ranking. For the cat image only the right caption scores clearly: 0.202 → 0.505. Real CLIP does read some word order, but benchmarks such as ARO show it close to this bag-of-words behaviour on swaps.

Worked example

Candidatecos(c, v)CLIP-Smax cos(c, r)RefCLIP-S
"a dog leaps for a frisbee"0.312.5 × 0.31 = 0.7750.802·0.775·0.80 / 1.575 = 0.787
"a dog on a beach"0.240.6000.622·0.600·0.62 / 1.220 = 0.610
"a plate of pasta"−0.03max(−0.075, 0) = 00.050 (H with a zero is 0)
Three candidate captions for one image; one reference cosine per caption (computed in Python)

The ranking is set by the cosines alone, and ww only scales them. A difference of 0.07 in cosine is 0.175 in CLIP-S and 7 points on the ×100 scale, so the same gap looks large or small depending on the convention. Always state which one you report.

python
1import torch
2from PIL import Image
3from transformers import CLIPModel, CLIPProcessor
4
5name = "openai/clip-vit-base-patch32" # the paper's backbone
6model = CLIPModel.from_pretrained(name).eval()
7proc = CLIPProcessor.from_pretrained(name)
8
9@torch.no_grad()
10def clip_score(images, captions, w=2.5, prefix="A photo depicts "):
11 """images[i] is scored against captions[i]; returns one score per pair."""
12 inp = proc(text=[prefix + c for c in captions], images=images,
13 return_tensors="pt", padding=True, truncation=True) # truncates at 77 tokens
14 v = model.get_image_features(pixel_values=inp["pixel_values"])
15 t = model.get_text_features(input_ids=inp["input_ids"], attention_mask=inp["attention_mask"])
16 v, t = v / v.norm(dim=-1, keepdim=True), t / t.norm(dim=-1, keepdim=True)
17 return w * (v * t).sum(-1).clamp(min=0)
18
19imgs = [Image.open(p).convert("RGB") for p in ["gen_000.png", "gen_001.png"]]
20s = clip_score(imgs, ["a red cube on a blue sphere", "an astronaut riding a horse"])
21print(s.tolist(), s.mean().item()) # report the mean over the prompt set
CLIPScore for paired images and captions with Hugging Face transformers (not run here: needs torch, transformers and the model download)

Running a text-to-image evaluation

  1. 1Fix a prompt set (COCO captions, or a suite with counts, colours and relations) and generate a fixed number of images per prompt with recorded seeds.
  2. 2Pick one CLIP checkpoint and preprocessing, and state them; do not use the model that guided or filtered the generator.
  3. 3Embed prompt and image, L2-normalise, take the cosine, clip at 0 and scale (w=2.5w=2.5, or 100 for the common convention).
  4. 4Average over images and prompts; bootstrap over prompts for a confidence interval.
  5. 5Report it next to FID for the same samples, and check a sample of low- and high-scoring pairs by eye.

Where it's used

Caption evaluation

reference-free scoring of captioning models, with RefCLIP-S when references exist (see Image captioning)

Text-to-image benchmarks

prompt adherence on COCO captions or prompt suites, plotted against FID across guidance scales (see Text-to-image generation)

Data filtering

dropping mismatched web or synthetic image–text pairs below a cosine threshold (see Synthetic data generation)

Best-of-n reranking

picking the generated image that best matches the prompt, with the risk of optimising the metric

Check yourself

Model A scores 0.78 CLIP-S (w = 2.5, ViT-B/32) and model B scores 31.5 (torchmetrics, ViT-L/14) on different prompt sets. Which is better aligned?

Check yourself

For the prompt "a blue cube on a red sphere", the generated image shows a red cube on a blue sphere. What will CLIP score most likely do?

Remember

  • ✓CLIP-S =2.5⋅max⁡(cos⁡(ET(c),EI(v)),0)=2.5\cdot\max(\cos(E_T(c),E_I(v)),0), reference-free; RefCLIP-S takes the harmonic mean with the best caption–reference cosine.
  • ✓Paper setup: CLIP ViT-B/32, prefix "A photo depicts". Text-to-image work usually reports max(100·cos, 0) with a stated backbone.
  • ✓Measures alignment, not realism: report it alongside FID, typically as a curve over guidance scale.
  • ✓Blind spots: attribute binding, counting, spatial relations, text past 77 tokens; use VQA-based metrics or humans for those.
  • ✓Compare only within one checkpoint, prompt set and scale, and never on a model guided or filtered by the same CLIP.
← back to Generative AI roadmap