CLIP

CLIP trains an image and a text encoder on 400M web pairs to score matches by cosine similarity. Prompts like 'a photo of a {label}.' turn it into a zero-shot classifier: 76.2% on ImageNet with no ImageNet labels.

8 min read877 wordscore0 viewsupdated 2026-10-15

Definition

CLIP (Contrastive Language–Image Pre-training) is a pair of encoders, one for images and one for text, trained so that an image and its caption map to nearby unit vectors in a shared space. Introduced by Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger and Sutskever in "Learning Transferable Visual Models From Natural Language Supervision" (OpenAI, ICML 2021). Once trained, any set of class names written as sentences becomes a classifier, with no task-specific training.

Standard Image classification learns a fixed set of labels, such as the 1,000 ImageNet classes. Adding a class means collecting labelled images and retraining the head. CLIP learns from free-form captions instead, so its "label space" is any text. The input is an image and a list of candidate texts. The output is a similarity score for each text.

Learning from captions had been tried before. VirTex and ICMLM predicted caption words, and Visual N-Grams (Li et al., 2017) reached only 11.5% zero-shot on ImageNet. CLIP's changes were scale (400M pairs) and the objective. Predicting the exact caption with a Transformer language model learned 3× slower than predicting a bag of its words. Swapping prediction for a contrastive objective gave a further 4× gain in zero-shot ImageNet efficiency.

Training: match the diagonal

The data is WIT (WebImageText): 400M image–text pairs collected with 500,000 text queries, at most 20,000 pairs per query. Each step takes a batch of N=32,768N=32{,}768 pairs. Both encoders embed their side, the embeddings are L2-normalised, and every image is scored against every caption. This gives an N×NN\times N matrix with the true pairs on the diagonal. The loss is the symmetric InfoNCE loss (see also Contrastive language learning):

L=12[1N∑i=1N−log⁡es Ii⊤Ti∑jes Ii⊤Tj  +  1N∑j=1N−log⁡es Ij⊤Tj∑ies Ii⊤Tj],s=et′\mathcal L=\frac12\Big[\frac1N\sum_{i=1}^{N}-\log\frac{e^{s\,I_i^\top T_i}}{\sum_{j}e^{s\,I_i^\top T_j}}\;+\;\frac1N\sum_{j=1}^{N}-\log\frac{e^{s\,I_j^\top T_j}}{\sum_{i}e^{s\,I_i^\top T_j}}\Big],\qquad s=e^{t'}
Ii, TjI_i,\,T_j
unit-norm image and text embeddings (768-d for ViT-L/14), each a linear projection of its encoder's output
first/secondsumfirst / second sum
cross-entropy over each row (pick the right caption) and each column (pick the right image)
s=1/τs=1/\tau
learned logit scale, initialised to 1/0.07≈14.31/0.07\approx14.3 and clipped at 100 to keep training stable
1 · contrastive pre-training (400M pairs)"a dog naps on a sofa", …text encoder (Transformer)T1T2T3T4imagesimage encoderResNet or ViTI1I2I3I4I1·T1I1·T2I1·T3I1·T4I2·T1I2·T2I2·T3I2·T4I3·T1I3·T2I3·T3I3·T4I4·T1I4·T2I4·T3I4·T4logits s·IᵢᵀTⱼ, N = 32,768 pairs/batchCE over rows (image→text) + columns(text→image), halved; targets = diagonal2 · zero-shot classificationdogwolfcatfoxprompt: "a photo of a {label}."same text encoder, run once per classT_dogT_wolfT_catT_fox= weights of a linear classifier, cachedimage encoderIcosp0.3120.8160.2970.1820.231<.0010.2480.001p = softmax(100 · cos) → predict dogimage, ViT-L/14@336px: 24 × 24 = 576 patches + [CLS] = 577 tokens → 1024-d → linear → 768-dtext: BPE vocab 49,152, at most 76 tokens; [EOS] state of the last layer → linear → 768-ds = exp(t′) is learned: initialised to 1/0.07 ≈ 14.3, clipped at 100 (where it ends up)no projection MLP, no cross-attention: the two towers meet only in the dot product
fig.Left: a batch of N images and N captions goes through the two towers, and cross-entropy along rows and columns makes each diagonal entry win. Right: at test time each class name is written into the prompt "a photo of a {label}.", encoded once, and used as a linear classifier. With cosines 0.312, 0.297, 0.231 and 0.248 for dog, wolf, cat and fox, softmax(100·cos) gives dog 0.816 and wolf 0.182. A cosine gap of 0.015 becomes a factor of e^1.5 ≈ 4.5 in probability.

The towers never attend to each other. The image encoder is a modified ResNet (attention pooling instead of global average pooling) or a Vision Transformers (ViT). The text encoder is a 12-layer causal Transformer read out at the [EOS] token. Data augmentation is only a random square crop. Projections are linear, not MLPs.

Configuration and results (Radford et al., 2021)

Models
ResNet-50, RN101, RN50x4, RN50x16, RN50x64; ViT-B/32, ViT-B/16, ViT-L/14. The best, ViT-L/14@336px, is ViT-L/14 trained one extra epoch at 336 px.
Training
32 epochs over WIT, batch 32,768, Adam with cosine schedule. ViT-L/14 took 12 days on 256 V100 GPUs.
Text side
lower-cased BPE, vocabulary 49,152, at most 76 tokens; 63M-parameter text Transformer for the base models
Zero-shot ImageNet
76.2% top-1 for ViT-L/14@336px, matching the original supervised ResNet-50 without using any of the 1.28M training labels
Breadth
zero-shot CLIP beats a linear probe on ResNet-50 features on 16 of 27 datasets, and matches a 4-shot linear probe on its own features

Zero-shot classification

To classify into KK classes, write each class name into one or more prompt templates, encode them with the text tower, and normalise. The result is a K×dK\times d weight matrix computed once. An image is then classified by one matrix–vector product:

p(y=k∣x)=exp⁡ ⁣(s ⟨f(x),wk⟩)∑j=1Kexp⁡ ⁣(s ⟨f(x),wj⟩),wk=wˉk∥wˉk∥,wˉk=1M∑m=1Mg(πm(ck))∥g(πm(ck))∥p(y=k\mid x)=\frac{\exp\!\big(s\,\langle f(x),w_k\rangle\big)}{\sum_{j=1}^{K}\exp\!\big(s\,\langle f(x),w_j\rangle\big)},\qquad w_k=\frac{\bar w_k}{\lVert\bar w_k\rVert},\quad \bar w_k=\frac1M\sum_{m=1}^{M}\frac{g(\pi_m(c_k))}{\lVert g(\pi_m(c_k))\rVert}
f, gf,\,g
image and text encoders; f(x)f(x) is normalised
πm(ck)\pi_m(c_k)
template mm filled with class name ckc_k, e.g. "a sketch of a fox."
MM
number of ensembled templates; averaging happens in embedding space, so inference costs the same as with one prompt

The bare word is a poor query. Captions on the web are rarely a single word, and many words are ambiguous: "crane" is a bird and a machine, and "boxer" a dog breed and an athlete. The template "a photo of a {label}." raised ImageNet accuracy by 1.3 points over the bare label. Adding context helps further, for example "a photo of a {label}, a type of pet." on Oxford Pets. Ensembling 80 templates ("a bad photo of a {}.", "a sketch of a {}.", …) added another 3.5 points on ImageNet. Across 36 datasets, prompt engineering plus ensembling gave almost 5 points on average.

image #5 of class dog · p = softmax(100 · cos(image, class weight))cosprobabilitydog (true)"a photo of a dog."0.2900.427wolf"a photo of a wolf."0.2910.445 ✗cat"a photo of a cat."0.2360.002fox"a photo of a fox."0.2710.063rabbit"a photo of a rabbit."0.2710.063test accuracy (300 images) vs prompts averaged50%70%90%12345678bare label 57.0%confusion: row true, col predicteddogd41127wolfw93867catc1237164foxf11049rabbitr60test set: accuracy 75.0% · mean confidence 70.4% · mean NLL 0.610the logit scale changes confidence and NLL, never the argmax, so accuracy stays put
image
text side
fig.A toy CLIP space with five classes and 300 test images, where dog–wolf and cat–fox are close concepts. Default (one template, logit scale 100): dog #5 has cosines 0.290 (dog) and 0.291 (wolf), so it is misread as wolf, 0.445 vs 0.427. Test accuracy is 75.0%. Averaging all 8 templates gives dog 0.643 for that image and 86.7% accuracy. Bare labels give 57.0%. Drag the logit scale down to 10: accuracy stays 75.0%, but mean confidence falls from 70.4% to 26.2% and NLL rises from 0.610 to 1.366. The scale sets calibration, not the ranking.

Worked example: the logit scale

One image, classes dog / wolf / cat, cosines 0.30 / 0.28 / 0.22

  1. 1Trained scale s=100s=100: logits 30, 28, 2230,\ 28,\ 22. Softmax gives 0.8805, 0.1192, 0.00030.8805,\ 0.1192,\ 0.0003, so the prediction is dog and the loss for the true class is −ln⁡0.8805=0.127-\ln0.8805=0.127.
  2. 2Initial scale s=1/0.07≈14.29s=1/0.07\approx14.29: logits 4.286, 4.000, 3.1434.286,\ 4.000,\ 3.143. Softmax gives 0.483, 0.363, 0.1540.483,\ 0.363,\ 0.154. The prediction is still dog, but the loss is 0.7280.728.
  3. 3The ranking is set by the cosines and never changes with ss. CLIP raises ss during training because on a batch of 32,768 the right caption must beat 32,767 others, which needs sharp logits. Unit vectors keep cosines within [−1,1][-1,1], so ss alone sets how peaked the softmax can be.
python
1import torch
2from PIL import Image
3from transformers import CLIPModel, CLIPProcessor
4
5name = "openai/clip-vit-large-patch14-336"
6model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)
7classes = ["dog", "wolf", "cat", "fox"]
8templates = ["a photo of a {}.", "a bad photo of a {}.", "a sketch of a {}.", "a close-up photo of a {}."]
9
10with torch.no_grad():
11 W = []
12 for c in classes: # one weight vector per class
13 t = proc(text=[tp.format(c) for tp in templates], return_tensors="pt", padding=True)
14 e = model.get_text_features(**t)
15 e = (e / e.norm(dim=-1, keepdim=True)).mean(0) # average normalised prompt embeddings
16 W.append(e / e.norm())
17 W = torch.stack(W) # (4, 768), cache and reuse
18
19 x = proc(images=Image.open("pet.jpg"), return_tensors="pt")
20 f = model.get_image_features(**x)
21 f = f / f.norm(dim=-1, keepdim=True)
22 probs = (model.logit_scale.exp() * f @ W.T).softmax(-1) # logit_scale.exp() = 100
23print(dict(zip(classes, probs[0].tolist())))
Zero-shot classification with prompt ensembling, using Hugging Face transformers (not run here)

Robustness to distribution shift

An ImageNet-trained ResNet-101 and zero-shot CLIP ViT-L/14@336 both score 76.2% on ImageNet. On shifted test sets they part ways: ImageNet-R (renditions) 37.7% vs 88.9%, ObjectNet 32.6% vs 72.3%, ImageNet Sketch 25.2% vs 60.2%, ImageNet-A (adversarially filtered) 2.7% vs 77.1%. Radford et al. report that zero-shot CLIP closes up to 75% of this robustness gap. Fitting a logistic-regression head on ImageNet features adds 9.2 points on ImageNet but slightly lowers the average on the shifted sets. Part of the robustness comes from never fitting ImageNet's own biases.

Where it's used

Open-vocabulary classification

new label sets by editing a list of strings; no retraining

Retrieval

text→image and image→text search with precomputed embeddings and a nearest-neighbour index

Data filtering

CLIP score (image–caption cosine) filters web data, as in LAION-400M's threshold of 0.3 with ViT-B/32

Backbone

image features for vision-language models, conditioning for text-to-image generation, open-vocabulary detection and segmentation

Check yourself

You lower CLIP's logit scale from 100 to 14.3 at inference. What changes on a test set?

Check yourself

Why does the prompt 'a photo of a {label}.' beat the bare label as a class query?

Remember

  • ✓CLIP: image and text towers trained with symmetric InfoNCE on 400M pairs, batch 32,768; they meet only in a dot product of unit vectors.
  • ✓Learned logit scale s=1/τs=1/\tau: initialised to 14.3 (τ = 0.07), clipped at 100. It sets confidence, not the ranking.
  • ✓Zero-shot: class weights wkw_k are normalised text embeddings of prompts; p(y=k∣x)=softmaxk(s⟨f(x),wk⟩)p(y=k\mid x)=\mathrm{softmax}_k(s\langle f(x),w_k\rangle).
  • ✓Prompt templates (+1.3 points on ImageNet) and 80-template ensembling (+3.5 more) are averaged in embedding space at no extra inference cost.
  • ✓ViT-L/14@336: 76.2% zero-shot ImageNet, and far more robust on ImageNet-R, ObjectNet, Sketch and ImageNet-A than an ImageNet-trained ResNet-101 with the same ImageNet accuracy.
← back to Deep Learning roadmap