# CLIP score

> CLIPScore (Hessel et al., 2021) = 2.5 · max(cos(text, image), 0) from a frozen CLIP: a reference-free text–image alignment score used for captions and text-to-image prompts, blind to binding, counts and relations.

Source: https://www.mlfoundry.in/learn/clip-score  
Section: Generative AI › Evaluation of Generative Models  
Level: core · 7 min read · updated 2026-10-21

**CLIP score:** measures how well a text matches an image as the clipped, rescaled cosine similarity of their **CLIP** embeddings. It was introduced as **CLIPScore** by Hessel, Holtzman, Forbes, Le Bras and Choi, "CLIPScore: A Reference-free Evaluation Metric for Image Captioning" (EMNLP 2021), and is now also the standard prompt-alignment metric for text-to-image models.

Caption metrics such as BLEU and CIDEr compare a candidate caption with human reference captions (see **BLEU, ROUGE and METEOR**). They need references, and they punish a correct caption that uses different words. CLIP was trained on hundreds of millions of image–text pairs to put matching pairs close together, so it can compare the caption with the **image** directly.

In text-to-image work the roles flip: the prompt is the text, the generated image is scored, and the mean over a prompt set measures prompt adherence. It complements **FID and Inception Score**, which measure realism but ignore the prompt. Papers sweep the classifier-free guidance scale and plot FID against CLIP score, because more guidance usually raises alignment and, past a point, worsens FID.

## The score

$$
\text{CLIP-S}(c,v)=w\cdot\max\big(\cos(E_T(c),\,E_I(v)),\,0\big),\qquad w=2.5
$$

- $E_T,\,E_I$: frozen CLIP text and image encoders (ViT-B/32 in the paper); the caption is prefixed with "A photo depicts"
- $\cos$: cosine of the two embeddings, in [−1, 1]; for real pairs it rarely exceeds about 0.4
- $w$: a rescaling constant chosen so scores spread over roughly [0, 1]; it does not change rankings
- $\max(\cdot,0)$: negative cosines are clipped to 0, the floor for an unrelated text

A corpus-level score is the mean over all (caption, image) pairs. When reference captions $R$ exist, **RefCLIPScore** adds them through a harmonic mean, so a caption must match both the image and the references:

$$
\text{RefCLIP-S}(c,R,v)=\mathrm{H}\Big(\text{CLIP-S}(c,v),\;\max\big(\max_{r\in R}\cos(E_T(c),E_T(r)),\,0\big)\Big),\qquad \mathrm{H}(a,b)=\frac{2ab}{a+b}
$$

- $r\in R$: human reference captions, embedded with the same text encoder
- $\mathrm H$: harmonic mean: low if either part is low

> Figure: The caption, prefixed with "A photo depicts", and the image go through the frozen CLIP towers and are compared by cosine. A cosine of 0.31 gives CLIP-S = 2.5 × 0.31 = 0.775. With references, the best caption-to-reference cosine 0.80 is combined by a harmonic mean: 2 · 0.775 · 0.80 / 1.575 = 0.787. For text-to-image evaluation three prompts with cosines 0.31, 0.27 and 0.22 give a mean CLIP-S of 0.667, which the ×100 convention reports as 26.7.

**Setting and conventions**

- **Paper setup:** CLIP ViT-B/32, prompt prefix "A photo depicts", w = 2.5; no references and no training on caption data
- **Paper finding:** on literal image-description benchmarks such as Flickr8K-Expert, CLIP-S correlated with human judgements better than reference-based metrics like CIDEr and SPICE; it did worse on news captions that need context beyond the image
- **×100 convention:** text-to-image papers and `torchmetrics` report max(100 · cos, 0), so 0.31 is 31.0; the backbone (often ViT-L/14) must be stated too
- **Text length:** CLIP's text encoder has a 77-token context; longer prompts are truncated, and their tail is never scored
- **Data filtering:** LAION-400M kept web pairs with ViT-B/32 cosine ≥ 0.3; the same score is used to filter synthetic training pairs
- **Subject fidelity:** DreamBooth (Ruiz et al., 2023) reports CLIP-T (prompt–image, this score) and CLIP-I (image–image) with DINO similarity for the subject

> Figure: A toy CLIP space: an image is a weighted sum of concept vectors plus noise, a caption is the normalised sum of its word vectors, and each tower adds its own offset (the modality gap). At the defaults (dog-and-red-ball image, gap 2.5, noise 0.5) the correct caption and its attribute-swapped twin "a red dog with a ball on grass" tie at cos 0.297, CLIP-S 0.743; "a dog on grass" scores 0.241 → 0.601 and "a cat on a sofa" has cos −0.027, clipped to 0. Gap 0 lifts the top cosine to 0.901; gap 4 cuts it to 0.145, with the same ranking. For the cat image only the right caption scores clearly: 0.202 → 0.505. Real CLIP does read some word order, but benchmarks such as ARO show it close to this bag-of-words behaviour on swaps.

## Worked example

Three candidate captions for one image; one reference cosine per caption (computed in Python)

| Candidate | cos(c, v) | CLIP-S | max cos(c, r) | RefCLIP-S |
| --- | --- | --- | --- | --- |
| "a dog leaps for a frisbee" | 0.31 | 2.5 × 0.31 = 0.775 | 0.80 | 2·0.775·0.80 / 1.575 = 0.787 |
| "a dog on a beach" | 0.24 | 0.600 | 0.62 | 2·0.600·0.62 / 1.220 = 0.610 |
| "a plate of pasta" | −0.03 | max(−0.075, 0) = 0 | 0.05 | 0 (H with a zero is 0) |

The ranking is set by the cosines alone, and $w$ only scales them. A difference of 0.07 in cosine is 0.175 in CLIP-S and 7 points on the ×100 scale, so the same gap looks large or small depending on the convention. Always state which one you report.

```python
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

name = "openai/clip-vit-base-patch32"           # the paper's backbone
model = CLIPModel.from_pretrained(name).eval()
proc = CLIPProcessor.from_pretrained(name)

@torch.no_grad()
def clip_score(images, captions, w=2.5, prefix="A photo depicts "):
    """images[i] is scored against captions[i]; returns one score per pair."""
    inp = proc(text=[prefix + c for c in captions], images=images,
               return_tensors="pt", padding=True, truncation=True)   # truncates at 77 tokens
    v = model.get_image_features(pixel_values=inp["pixel_values"])
    t = model.get_text_features(input_ids=inp["input_ids"], attention_mask=inp["attention_mask"])
    v, t = v / v.norm(dim=-1, keepdim=True), t / t.norm(dim=-1, keepdim=True)
    return w * (v * t).sum(-1).clamp(min=0)

imgs = [Image.open(p).convert("RGB") for p in ["gen_000.png", "gen_001.png"]]
s = clip_score(imgs, ["a red cube on a blue sphere", "an astronaut riding a horse"])
print(s.tolist(), s.mean().item())               # report the mean over the prompt set
```

CLIPScore for paired images and captions with Hugging Face transformers (not run here: needs torch, transformers and the model download)

## Running a text-to-image evaluation

1. Fix a prompt set (COCO captions, or a suite with counts, colours and relations) and generate a fixed number of images per prompt with recorded seeds.
2. Pick one CLIP checkpoint and preprocessing, and state them; do not use the model that guided or filtered the generator.
3. Embed prompt and image, L2-normalise, take the cosine, clip at 0 and scale ($w=2.5$, or 100 for the common convention).
4. Average over images and prompts; bootstrap over prompts for a confidence interval.
5. Report it next to FID for the same samples, and check a sample of low- and high-scoring pairs by eye.

> **Bag-of-words blind spots:** CLIP's contrastive training rarely needs word order, so it often scores "a red cube on a blue sphere" and "a blue cube on a red sphere" alike. Winoground (Thrush et al., CVPR 2022) and ARO (Yuksekgonul et al., ICLR 2023) show CLIP-style models near chance on such swaps, and on counting and spatial relations. A high CLIP score does not certify that a generated image has three cats or the right colours on the right objects. For compositional prompts, use question-answering metrics (TIFA, VQAScore) or human checks.

> **Do not optimise the metric you report:** If a generator was guided by CLIP, fine-tuned with a CLIP reward, or trained on data filtered by CLIP score, it can learn what CLIP likes, such as rendering the prompt's words as text in the image. Its CLIP score is then inflated. Evaluate with a different backbone or a different metric family than the one used in training or filtering.

> **Levels mean little; differences within one setup do:** Image and text embeddings occupy separate regions of CLIP space (the modality gap), so even perfect matches give cosines around 0.3–0.4. Absolute scores shift with the backbone, the prefix and image preprocessing. Compare systems on the same prompts with the same checkpoint, and report confidence intervals: on a few hundred prompts, differences of a few tenths of a point on the ×100 scale are often noise.

- **Caption evaluation:** reference-free scoring of captioning models, with RefCLIP-S when references exist (see **Image captioning**)
- **Text-to-image benchmarks:** prompt adherence on COCO captions or prompt suites, plotted against FID across guidance scales (see **Text-to-image generation**)
- **Data filtering:** dropping mismatched web or synthetic image–text pairs below a cosine threshold (see **Synthetic data generation**)
- **Best-of-n reranking:** picking the generated image that best matches the prompt, with the risk of optimising the metric

**Question:** Model A scores 0.78 CLIP-S (w = 2.5, ViT-B/32) and model B scores 31.5 (torchmetrics, ViT-L/14) on different prompt sets. Which is better aligned?

- A. A, because 0.78 × 100 > 31.5
- B. B, because 31.5 / 100 × 2.5 = 0.79 > 0.78
- C. Cannot tell: the backbone and the prompts differ, so the scores are not comparable
- D. They are equal

**Answer:** C. Cannot tell: the backbone and the prompts differ, so the scores are not comparable — The scale $w$ can be converted ($0.78/2.5 = 0.312$ cosine against $0.315$), but the backbone cannot: different CLIP checkpoints give different cosine levels for the same pair, and different prompt sets differ in difficulty. Rescore both models with one checkpoint on one prompt set.

**Question:** For the prompt "a blue cube on a red sphere", the generated image shows a red cube on a blue sphere. What will CLIP score most likely do?

- A. Give it about the same score as a correct image
- B. Give it 0, because the colours are wrong
- C. Give it a negative score
- D. Refuse to score it

**Answer:** A. Give it about the same score as a correct image — The image contains all the right words: blue, red, cube, sphere. CLIP's embeddings behave much like a bag of words on attribute-binding swaps, so the cosine is close to that of a correct image. Clipping at 0 only matters for unrelated texts, and CLIP-S is never negative.

**Key points**

- CLIP-S $=2.5\cdot\max(\cos(E_T(c),E_I(v)),0)$, reference-free; RefCLIP-S takes the harmonic mean with the best caption–reference cosine.
- Paper setup: CLIP ViT-B/32, prefix "A photo depicts". Text-to-image work usually reports max(100·cos, 0) with a stated backbone.
- Measures alignment, not realism: report it alongside FID, typically as a curve over guidance scale.
- Blind spots: attribute binding, counting, spatial relations, text past 77 tokens; use VQA-based metrics or humans for those.
- Compare only within one checkpoint, prompt set and scale, and never on a model guided or filtered by the same CLIP.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
