# CLIP

> CLIP trains an image and a text encoder on 400M web pairs to score matches by cosine similarity. Prompts like 'a photo of a {label}.' turn it into a zero-shot classifier: 76.2% on ImageNet with no ImageNet labels.

Source: https://www.mlfoundry.in/learn/clip  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**CLIP:** (Contrastive Language–Image Pre-training) is a pair of encoders, one for images and one for text, trained so that an image and its caption map to nearby unit vectors in a shared space. Introduced by Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger and Sutskever in "Learning Transferable Visual Models From Natural Language Supervision" (OpenAI, ICML 2021). Once trained, any set of class names written as sentences becomes a classifier, with no task-specific training.

Standard **Image classification** learns a fixed set of labels, such as the 1,000 ImageNet classes. Adding a class means collecting labelled images and retraining the head. CLIP learns from free-form captions instead, so its "label space" is any text. The input is an image and a list of candidate texts. The output is a similarity score for each text.

Learning from captions had been tried before. VirTex and ICMLM predicted caption words, and Visual N-Grams (Li et al., 2017) reached only 11.5% zero-shot on ImageNet. CLIP's changes were scale (400M pairs) and the objective. Predicting the exact caption with a Transformer language model learned 3× slower than predicting a bag of its words. Swapping prediction for a contrastive objective gave a further 4× gain in zero-shot ImageNet efficiency.

## Training: match the diagonal

The data is WIT (WebImageText): 400M image–text pairs collected with 500,000 text queries, at most 20,000 pairs per query. Each step takes a batch of $N=32{,}768$ pairs. Both encoders embed their side, the embeddings are L2-normalised, and every image is scored against every caption. This gives an $N\times N$ matrix with the true pairs on the diagonal. The loss is the symmetric **InfoNCE loss** (see also **Contrastive language learning**):

$$
\mathcal L=\frac12\Big[\frac1N\sum_{i=1}^{N}-\log\frac{e^{s\,I_i^\top T_i}}{\sum_{j}e^{s\,I_i^\top T_j}}\;+\;\frac1N\sum_{j=1}^{N}-\log\frac{e^{s\,I_j^\top T_j}}{\sum_{i}e^{s\,I_i^\top T_j}}\Big],\qquad s=e^{t'}
$$

- $I_i,\,T_j$: unit-norm image and text embeddings (768-d for ViT-L/14), each a linear projection of its encoder's output
- $first / second sum$: cross-entropy over each row (pick the right caption) and each column (pick the right image)
- $s=1/\tau$: learned logit scale, initialised to $1/0.07\approx14.3$ and clipped at 100 to keep training stable

> Figure: Left: a batch of N images and N captions goes through the two towers, and cross-entropy along rows and columns makes each diagonal entry win. Right: at test time each class name is written into the prompt "a photo of a {label}.", encoded once, and used as a linear classifier. With cosines 0.312, 0.297, 0.231 and 0.248 for dog, wolf, cat and fox, softmax(100·cos) gives dog 0.816 and wolf 0.182. A cosine gap of 0.015 becomes a factor of e^1.5 ≈ 4.5 in probability.

The towers never attend to each other. The image encoder is a modified ResNet (attention pooling instead of global average pooling) or a **Vision Transformers (ViT)**. The text encoder is a 12-layer causal Transformer read out at the [EOS] token. Data augmentation is only a random square crop. Projections are linear, not MLPs.

**Configuration and results (Radford et al., 2021)**

- **Models:** ResNet-50, RN101, RN50x4, RN50x16, RN50x64; ViT-B/32, ViT-B/16, ViT-L/14. The best, ViT-L/14@336px, is ViT-L/14 trained one extra epoch at 336 px.
- **Training:** 32 epochs over WIT, batch 32,768, Adam with cosine schedule. ViT-L/14 took 12 days on 256 V100 GPUs.
- **Text side:** lower-cased BPE, vocabulary 49,152, at most 76 tokens; 63M-parameter text Transformer for the base models
- **Zero-shot ImageNet:** 76.2% top-1 for ViT-L/14@336px, matching the original supervised ResNet-50 without using any of the 1.28M training labels
- **Breadth:** zero-shot CLIP beats a linear probe on ResNet-50 features on 16 of 27 datasets, and matches a 4-shot linear probe on its own features

## Zero-shot classification

To classify into $K$ classes, write each class name into one or more prompt templates, encode them with the text tower, and normalise. The result is a $K\times d$ weight matrix computed once. An image is then classified by one matrix–vector product:

$$
p(y=k\mid x)=\frac{\exp\!\big(s\,\langle f(x),w_k\rangle\big)}{\sum_{j=1}^{K}\exp\!\big(s\,\langle f(x),w_j\rangle\big)},\qquad w_k=\frac{\bar w_k}{\lVert\bar w_k\rVert},\quad \bar w_k=\frac1M\sum_{m=1}^{M}\frac{g(\pi_m(c_k))}{\lVert g(\pi_m(c_k))\rVert}
$$

- $f,\,g$: image and text encoders; $f(x)$ is normalised
- $\pi_m(c_k)$: template $m$ filled with class name $c_k$, e.g. "a sketch of a fox."
- $M$: number of ensembled templates; averaging happens in embedding space, so inference costs the same as with one prompt

The bare word is a poor query. Captions on the web are rarely a single word, and many words are ambiguous: "crane" is a bird and a machine, and "boxer" a dog breed and an athlete. The template "a photo of a {label}." raised ImageNet accuracy by 1.3 points over the bare label. Adding context helps further, for example "a photo of a {label}, a type of pet." on Oxford Pets. Ensembling 80 templates ("a bad photo of a {}.", "a sketch of a {}.", …) added another 3.5 points on ImageNet. Across 36 datasets, prompt engineering plus ensembling gave almost 5 points on average.

> Figure: A toy CLIP space with five classes and 300 test images, where dog–wolf and cat–fox are close concepts. Default (one template, logit scale 100): dog #5 has cosines 0.290 (dog) and 0.291 (wolf), so it is misread as wolf, 0.445 vs 0.427. Test accuracy is 75.0%. Averaging all 8 templates gives dog 0.643 for that image and 86.7% accuracy. Bare labels give 57.0%. Drag the logit scale down to 10: accuracy stays 75.0%, but mean confidence falls from 70.4% to 26.2% and NLL rises from 0.610 to 1.366. The scale sets calibration, not the ranking.

## Worked example: the logit scale

**One image, classes dog / wolf / cat, cosines 0.30 / 0.28 / 0.22**

1. Trained scale $s=100$: logits $30,\ 28,\ 22$. Softmax gives $0.8805,\ 0.1192,\ 0.0003$, so the prediction is dog and the loss for the true class is $-\ln0.8805=0.127$.
2. Initial scale $s=1/0.07\approx14.29$: logits $4.286,\ 4.000,\ 3.143$. Softmax gives $0.483,\ 0.363,\ 0.154$. The prediction is still dog, but the loss is $0.728$.
3. The ranking is set by the cosines and never changes with $s$. CLIP raises $s$ during training because on a batch of 32,768 the right caption must beat 32,767 others, which needs sharp logits. Unit vectors keep cosines within $[-1,1]$, so $s$ alone sets how peaked the softmax can be.

```python
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

name = "openai/clip-vit-large-patch14-336"
model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)
classes = ["dog", "wolf", "cat", "fox"]
templates = ["a photo of a {}.", "a bad photo of a {}.", "a sketch of a {}.", "a close-up photo of a {}."]

with torch.no_grad():
    W = []
    for c in classes:                                    # one weight vector per class
        t = proc(text=[tp.format(c) for tp in templates], return_tensors="pt", padding=True)
        e = model.get_text_features(**t)
        e = (e / e.norm(dim=-1, keepdim=True)).mean(0)   # average normalised prompt embeddings
        W.append(e / e.norm())
    W = torch.stack(W)                                   # (4, 768), cache and reuse

    x = proc(images=Image.open("pet.jpg"), return_tensors="pt")
    f = model.get_image_features(**x)
    f = f / f.norm(dim=-1, keepdim=True)
    probs = (model.logit_scale.exp() * f @ W.T).softmax(-1)   # logit_scale.exp() = 100
print(dict(zip(classes, probs[0].tolist())))
```

Zero-shot classification with prompt ensembling, using Hugging Face transformers (not run here)

## Robustness to distribution shift

An ImageNet-trained ResNet-101 and zero-shot CLIP ViT-L/14@336 both score 76.2% on ImageNet. On shifted test sets they part ways: ImageNet-R (renditions) 37.7% vs 88.9%, ObjectNet 32.6% vs 72.3%, ImageNet Sketch 25.2% vs 60.2%, ImageNet-A (adversarially filtered) 2.7% vs 77.1%. Radford et al. report that zero-shot CLIP closes up to 75% of this robustness gap. Fitting a logistic-regression head on ImageNet features adds 9.2 points on ImageNet but slightly lowers the average on the shifted sets. Part of the robustness comes from never fitting ImageNet's own biases.

> **Where zero-shot CLIP is weak:** It does poorly on fine-grained, abstract or out-of-distribution tasks: counting objects, satellite imagery, tumour patches, distance estimation. On MNIST digits it reaches only 88%, below logistic regression on raw pixels. It is also sensitive to text in the image: a hand-written label "iPod" stuck on an apple can flip the prediction (Goh et al., "Multimodal Neurons", 2021). Check the target domain before trusting zero-shot numbers.

> **The image tower outlived the classifier:** CLIP's image encoder became a standard backbone. **Stable Diffusion** conditions on CLIP text embeddings, LLaVA uses CLIP ViT-L/14 as its vision tower (see **Multimodal transformers**), and **SigLIP** replaces the softmax loss with a sigmoid one. Open re-trainings (OpenCLIP on LAION) reproduce the scaling behaviour with public data.

- **Open-vocabulary classification:** new label sets by editing a list of strings; no retraining
- **Retrieval:** text→image and image→text search with precomputed embeddings and a nearest-neighbour index
- **Data filtering:** CLIP score (image–caption cosine) filters web data, as in LAION-400M's threshold of 0.3 with ViT-B/32
- **Backbone:** image features for vision-language models, conditioning for text-to-image generation, open-vocabulary detection and segmentation

**Question:** You lower CLIP's logit scale from 100 to 14.3 at inference. What changes on a test set?

- A. Accuracy drops, because the classes become harder to separate
- B. Accuracy is unchanged; the softmax becomes flatter, so confidence and NLL change
- C. Nothing, because the embeddings are normalised
- D. Accuracy rises, because the model becomes less overconfident

**Answer:** B. Accuracy is unchanged; the softmax becomes flatter, so confidence and NLL change — Multiplying all logits by a positive constant keeps their order, so the argmax and accuracy are unchanged. The probabilities change: with cosines 0.30 / 0.28 / 0.22, the top probability falls from 0.880 at $s=100$ to 0.483 at $s=14.3$.

**Question:** Why does the prompt 'a photo of a {label}.' beat the bare label as a class query?

- A. The text encoder cannot process single words
- B. Longer prompts produce longer embedding vectors
- C. Web captions are sentences, and context resolves ambiguous words such as 'crane' or 'boxer'
- D. It doubles the number of classes the model can score

**Answer:** C. Web captions are sentences, and context resolves ambiguous words such as 'crane' or 'boxer' — The text encoder was trained on captions, and single-word captions are rare. The template moves the query toward the training distribution and tells the model it is describing an image of an object. Ensembling several such templates averages out the noise of any single phrasing.

**Key points**

- CLIP: image and text towers trained with symmetric InfoNCE on 400M pairs, batch 32,768; they meet only in a dot product of unit vectors.
- Learned logit scale $s=1/\tau$: initialised to 14.3 (τ = 0.07), clipped at 100. It sets confidence, not the ranking.
- Zero-shot: class weights $w_k$ are normalised text embeddings of prompts; $p(y=k\mid x)=\mathrm{softmax}_k(s\langle f(x),w_k\rangle)$.
- Prompt templates (+1.3 points on ImageNet) and 80-template ensembling (+3.5 more) are averaged in embedding space at no extra inference cost.
- ViT-L/14@336: 76.2% zero-shot ImageNet, and far more robust on ImageNet-R, ObjectNet, Sketch and ImageNet-A than an ImageNet-trained ResNet-101 with the same ImageNet accuracy.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
