Definition
CLIP (Contrastive Language–Image Pre-training) is a pair of encoders, one for images and one for text, trained so that an image and its caption map to nearby unit vectors in a shared space. Introduced by Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger and Sutskever in "Learning Transferable Visual Models From Natural Language Supervision" (OpenAI, ICML 2021). Once trained, any set of class names written as sentences becomes a classifier, with no task-specific training.
Standard Image classification learns a fixed set of labels, such as the 1,000 ImageNet classes. Adding a class means collecting labelled images and retraining the head. CLIP learns from free-form captions instead, so its "label space" is any text. The input is an image and a list of candidate texts. The output is a similarity score for each text.
Learning from captions had been tried before. VirTex and ICMLM predicted caption words, and Visual N-Grams (Li et al., 2017) reached only 11.5% zero-shot on ImageNet. CLIP's changes were scale (400M pairs) and the objective. Predicting the exact caption with a Transformer language model learned 3× slower than predicting a bag of its words. Swapping prediction for a contrastive objective gave a further 4× gain in zero-shot ImageNet efficiency.
Training: match the diagonal
The data is WIT (WebImageText): 400M image–text pairs collected with 500,000 text queries, at most 20,000 pairs per query. Each step takes a batch of pairs. Both encoders embed their side, the embeddings are L2-normalised, and every image is scored against every caption. This gives an matrix with the true pairs on the diagonal. The loss is the symmetric InfoNCE loss (see also Contrastive language learning):
- unit-norm image and text embeddings (768-d for ViT-L/14), each a linear projection of its encoder's output
- cross-entropy over each row (pick the right caption) and each column (pick the right image)
- learned logit scale, initialised to and clipped at 100 to keep training stable
The towers never attend to each other. The image encoder is a modified ResNet (attention pooling instead of global average pooling) or a Vision Transformers (ViT). The text encoder is a 12-layer causal Transformer read out at the [EOS] token. Data augmentation is only a random square crop. Projections are linear, not MLPs.
Configuration and results (Radford et al., 2021)
- Models
- ResNet-50, RN101, RN50x4, RN50x16, RN50x64; ViT-B/32, ViT-B/16, ViT-L/14. The best, ViT-L/14@336px, is ViT-L/14 trained one extra epoch at 336 px.
- Training
- 32 epochs over WIT, batch 32,768, Adam with cosine schedule. ViT-L/14 took 12 days on 256 V100 GPUs.
- Text side
- lower-cased BPE, vocabulary 49,152, at most 76 tokens; 63M-parameter text Transformer for the base models
- Zero-shot ImageNet
- 76.2% top-1 for ViT-L/14@336px, matching the original supervised ResNet-50 without using any of the 1.28M training labels
- Breadth
- zero-shot CLIP beats a linear probe on ResNet-50 features on 16 of 27 datasets, and matches a 4-shot linear probe on its own features
Zero-shot classification
To classify into classes, write each class name into one or more prompt templates, encode them with the text tower, and normalise. The result is a weight matrix computed once. An image is then classified by one matrix–vector product:
- image and text encoders; is normalised
- template filled with class name , e.g. "a sketch of a fox."
- number of ensembled templates; averaging happens in embedding space, so inference costs the same as with one prompt
The bare word is a poor query. Captions on the web are rarely a single word, and many words are ambiguous: "crane" is a bird and a machine, and "boxer" a dog breed and an athlete. The template "a photo of a {label}." raised ImageNet accuracy by 1.3 points over the bare label. Adding context helps further, for example "a photo of a {label}, a type of pet." on Oxford Pets. Ensembling 80 templates ("a bad photo of a {}.", "a sketch of a {}.", …) added another 3.5 points on ImageNet. Across 36 datasets, prompt engineering plus ensembling gave almost 5 points on average.
Worked example: the logit scale
One image, classes dog / wolf / cat, cosines 0.30 / 0.28 / 0.22
- 1Trained scale : logits . Softmax gives , so the prediction is dog and the loss for the true class is .
- 2Initial scale : logits . Softmax gives . The prediction is still dog, but the loss is .
- 3The ranking is set by the cosines and never changes with . CLIP raises during training because on a batch of 32,768 the right caption must beat 32,767 others, which needs sharp logits. Unit vectors keep cosines within , so alone sets how peaked the softmax can be.
1import torch2from PIL import Image3from transformers import CLIPModel, CLIPProcessor45name = "openai/clip-vit-large-patch14-336"6model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)7classes = ["dog", "wolf", "cat", "fox"]8templates = ["a photo of a {}.", "a bad photo of a {}.", "a sketch of a {}.", "a close-up photo of a {}."]910with torch.no_grad():11 W = []12 for c in classes: # one weight vector per class13 t = proc(text=[tp.format(c) for tp in templates], return_tensors="pt", padding=True)14 e = model.get_text_features(**t)15 e = (e / e.norm(dim=-1, keepdim=True)).mean(0) # average normalised prompt embeddings16 W.append(e / e.norm())17 W = torch.stack(W) # (4, 768), cache and reuse1819 x = proc(images=Image.open("pet.jpg"), return_tensors="pt")20 f = model.get_image_features(**x)21 f = f / f.norm(dim=-1, keepdim=True)22 probs = (model.logit_scale.exp() * f @ W.T).softmax(-1) # logit_scale.exp() = 10023print(dict(zip(classes, probs[0].tolist())))Robustness to distribution shift
An ImageNet-trained ResNet-101 and zero-shot CLIP ViT-L/14@336 both score 76.2% on ImageNet. On shifted test sets they part ways: ImageNet-R (renditions) 37.7% vs 88.9%, ObjectNet 32.6% vs 72.3%, ImageNet Sketch 25.2% vs 60.2%, ImageNet-A (adversarially filtered) 2.7% vs 77.1%. Radford et al. report that zero-shot CLIP closes up to 75% of this robustness gap. Fitting a logistic-regression head on ImageNet features adds 9.2 points on ImageNet but slightly lowers the average on the shifted sets. Part of the robustness comes from never fitting ImageNet's own biases.
Where it's used
Open-vocabulary classification
new label sets by editing a list of strings; no retraining
Retrieval
text→image and image→text search with precomputed embeddings and a nearest-neighbour index
Data filtering
CLIP score (image–caption cosine) filters web data, as in LAION-400M's threshold of 0.3 with ViT-B/32
Backbone
image features for vision-language models, conditioning for text-to-image generation, open-vocabulary detection and segmentation
Check yourself
You lower CLIP's logit scale from 100 to 14.3 at inference. What changes on a test set?
Check yourself
Why does the prompt 'a photo of a {label}.' beat the bare label as a class query?
Remember
- ✓CLIP: image and text towers trained with symmetric InfoNCE on 400M pairs, batch 32,768; they meet only in a dot product of unit vectors.
- ✓Learned logit scale : initialised to 14.3 (τ = 0.07), clipped at 100. It sets confidence, not the ranking.
- ✓Zero-shot: class weights are normalised text embeddings of prompts; .
- ✓Prompt templates (+1.3 points on ImageNet) and 80-template ensembling (+3.5 more) are averaged in embedding space at no extra inference cost.
- ✓ViT-L/14@336: 76.2% zero-shot ImageNet, and far more robust on ImageNet-R, ObjectNet, Sketch and ImageNet-A than an ImageNet-trained ResNet-101 with the same ImageNet accuracy.