# Image captioning

> Image captioning generates a sentence for an image with a vision encoder and a text decoder, from Show and Tell (2015) to BLIP. It is scored on COCO's 5 references per image with CIDEr: tf-idf n-gram cosine, ×10, with a length penalty.

Source: https://www.mlfoundry.in/learn/image-captioning  
Section: Generative AI › Multimodal Generative AI  
Level: core · 7 min read · updated 2026-10-21

**Image captioning:** generating a natural-language description of an image, modelled as $p(\text{caption}\mid\text{image})$. The neural template is **Show and Tell** (Vinyals, Toshev, Bengio and Erhan, CVPR 2015): a CNN encodes the image and an LSTM writes the words. Show, Attend and Tell (Xu et al., ICML 2015) added attention over spatial features. **BLIP** (Li et al., ICML 2022) and BLIP-2 (2023) pre-train on web image–text pairs. The standard benchmark is COCO Captions (Chen et al., 2015), scored with CIDEr (Vedantam, Zitnick and Parikh, CVPR 2015).

Captions do real work in products. They become alt text for screen readers, search-index text for photo libraries, and frame descriptions for video search. They are also used to relabel training data. DALL-E 3 (Betker et al., 2023) was trained on synthetic, detailed captions in place of noisy alt text, because better captions gave better prompt following.

The applied questions are which model to use, how to decode, and how to measure quality. The architectures are covered in **BLIP**, **Vision-language models** and **Multimodal LLMs**. Here we focus on the decoding objective, the CIDEr metric and how captioners are tuned against it.

## Pipeline: encode, decode, score

$$
\log p(S\mid I)=\sum_{t=1}^{T}\log p_\theta\big(w_t\mid I,\,w_1,\dots,w_{t-1}\big)
$$

- $I$: the image, seen through the vision encoder's features (a CNN vector in Show and Tell, patch features in ViT-based models)
- $S=(w_1,\dots,w_T)$: the caption, ending in an end-of-sentence token
- $\theta$: decoder weights, trained with teacher forcing on human captions (cross-entropy)

> Figure: (a) The decoder predicts each word from the image features and the words so far; beam search keeps the 3 best partial captions (example log-probabilities). (b) The output is scored against 5 human references by tf-idf weighted n-grams. On a toy 6-image corpus the word "a" gets idf 0 and "giraffe" 1.79, and the top beam scores CIDEr-D 2.989. (c) After cross-entropy training, SCST samples captions and rewards each by its CIDEr minus the greedy caption's: +0.426 for w¹, −2.225 for w².

Decoding is usually beam search with a small beam and a length limit. Greedy decoding gives short, safe captions; sampling gives more variety and more errors (see **Greedy and beam search**). BLIP-style models also accept a text prefix, such as "a picture of", that the decoder continues.

## CIDEr: consensus with the references

BLEU and ROUGE count n-gram overlap (see **BLEU, ROUGE and METEOR**). CIDEr changes two things. It weights each n-gram by how rare it is across the test set's images, and it averages agreement over all references for the image. A caption scores well when it says what most annotators said and uses the words that set this image apart from the others.

$$
g_k(s)=h_k(s)\cdot\log\frac{|I|}{\max\!\big(1,\ \mathrm{df}_k\big)},\qquad \mathrm{CIDEr\text{-}D}_n(c,S)=\frac{10}{m}\sum_{j=1}^{m} e^{-\frac{(l_c-l_{s_j})^2}{2\sigma^2}}\,\frac{\min\!\big(g^n(c),g^n(s_j)\big)\cdot g^n(s_j)}{\lVert g^n(c)\rVert\,\lVert g^n(s_j)\rVert}
$$

- $h_k(s)$: count of n-gram $k$ in sentence $s$
- $|I|,\ \mathrm{df}_k$: number of images in the test set, and how many of their reference sets contain $k$
- $m$: number of references for the image (5 on COCO)
- $\min(\cdot,\cdot)$: clips candidate n-gram weights at the reference's, so repeating a word earns nothing extra
- $\sigma$: length-penalty width, 6 in the standard implementation; $l$ is sentence length

The final score averages $n=1$ to 4 with equal weights. The original CIDEr has no clipping, no length penalty and no factor 10. CIDEr-D adds them to make the metric harder to game, and it is what the COCO evaluation server and `pycocoevalcap` report. Papers usually print it ×100: a CIDEr of 1.2 appears as 120.

**Datasets, metrics and training tricks**

- **COCO Captions:** 5 human captions per image; 82,783 train2014 and 40,504 val2014 images
- **Karpathy split:** the usual COCO protocol: 113,287 train, 5,000 val and 5,000 test images
- **nocaps:** Agrawal et al., ICCV 2019: Open Images photos with objects absent from COCO captions, to test novel-object captioning
- **SPICE:** Anderson et al., ECCV 2016: F-score over scene-graph tuples (objects, attributes, relations) parsed from the captions
- **CHAIR:** Rohrbach et al., EMNLP 2018: share of mentioned objects that are not in the image, a hallucination rate for captions
- **SCST:** Rennie et al., CVPR 2017: REINFORCE on the CIDEr reward with the greedy caption as baseline

> Figure: CIDEr-D computed live, as in pycocoevalcap, with idf from 6 toy images × 5 references. Giraffe, specific caption: 2.989 (CIDEr₁…₄ = 5.920, 3.271, 1.440, 1.323). The generic "an animal standing in a field" scores 0.338, below the wrong-object "a horse eating grass in a field" at 0.453: CIDEr checks consensus wording, not truth. The 19-word long caption scores 0.443; turn off the length penalty and it jumps to 1.704. Set max n-gram to 1 and the specific caption scores 5.920. The word "a" occurs in all 6 reference sets, so its idf is ln 6 − ln 6 = 0.

## Worked example: four captions for the giraffe

CIDEr-D against the 5 giraffe references, idf from the 6-image toy corpus, σ = 6 (computed in Python)

| Candidate | CIDEr₁ | CIDEr₂ | CIDEr₃ | CIDEr₄ | CIDEr-D |
| --- | --- | --- | --- | --- | --- |
| a giraffe eating leaves from a tree | 5.920 | 3.271 | 1.440 | 1.323 | 2.989 |
| a giraffe standing next to a tree | 4.729 | 2.793 | 1.395 | 1.334 | 2.563 |
| a horse eating grass in a field | 1.229 | 0.303 | 0.279 | 0.000 | 0.453 |
| an animal standing in a field | 0.763 | 0.307 | 0.283 | 0.000 | 0.338 |

The idf weights explain the ranking. With $|I|=6$, "a" appears in all 6 reference sets: weight $\ln 6-\ln 6=0$. "in" appears in 4: $\ln(6/4)=0.405$. "giraffe", "leaves" and "tree" appear only in the giraffe set: $\ln 6=1.792$. A caption earns its score with the specific words. On a real 5,000-image test set the spread is wider: an n-gram in 1,000 reference sets weighs $\ln 5=1.61$, and one in 50 weighs $\ln 100=4.61$.

$$
\nabla_\theta L(\theta)\approx-\big(r(w^s)-r(\hat w)\big)\,\nabla_\theta\log p_\theta(w^s\mid I)
$$

- $w^s$: a caption sampled from the model
- $\hat w$: the greedy caption, whose reward is the baseline (the self-critic)
- $r$: CIDEr-D of the caption against the references

SCST fine-tunes a cross-entropy model directly on the metric. In the diagram the greedy caption scores 2.563. A sample scoring 2.989 gets advantage $+0.426$ and becomes more likely; one scoring 0.338 gets $-2.225$ and becomes less likely. No value network is needed, and test-time decoding (greedy or beam) is exactly what the baseline measures.

```python
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
from pycocoevalcap.cider.cider import Cider

name = "Salesforce/blip-image-captioning-large"
processor = BlipProcessor.from_pretrained(name)
model = BlipForConditionalGeneration.from_pretrained(name)

def caption(path, prefix="a picture of"):
    inputs = processor(images=Image.open(path).convert("RGB"), text=prefix, return_tensors="pt")
    ids = model.generate(**inputs, num_beams=3, max_new_tokens=30)
    return processor.decode(ids[0], skip_special_tokens=True).removeprefix(prefix).strip()

# idf comes from the references of every image in gts, so score the whole test set at once
gts = {img_id: refs[img_id] for img_id in test_ids}          # 5 lower-cased references each
res = {img_id: [caption(f"val/{img_id}.jpg")] for img_id in test_ids}
corpus_cider, per_image = Cider().compute_score(gts, res)     # ~1.2 is reported as "120"
```

Caption with BLIP and score with pycocoevalcap (not run here: needs transformers, torch and pycocoevalcap; the official pipeline also tokenises with the PTB tokenizer, which needs Java)

> **CIDEr rewards consensus wording, not correctness:** In the figure a caption with the wrong animal outscores a vague but true one, because it shares "eating" and "in a field" with the references. SCST-tuned models learn the dataset's phrasing ("a ... sitting on top of a ..."), and they can mention objects that are not there. Report a hallucination measure (CHAIR, or **CLIP score** as a reference-free check) next to CIDEr, and look at samples. Per-image CIDEr is also not comparable across test sets, because the idf depends on the whole set.

> **Short COCO captions are a style, not the goal:** COCO references are one plain sentence. Alt text, retrieval indexes and training-data recaptioning often need more: text in the image, counts, positions. Prompting a multimodal LLM (see **Multimodal LLMs**) for a detailed description gives richer text, but it scores badly on CIDEr because it no longer matches the reference style. Choose the metric for the use: CIDEr and SPICE for COCO-style captions, human or LLM-judged ratings for long descriptions (see **LLM-as-a-judge**).

- **Accessibility:** alt text for images that have none; a person should review it where accuracy matters
- **Search and indexing:** captions make photo and video libraries searchable by text with an ordinary keyword or embedding index
- **Training data:** recaptioning web images with a captioner (BLIP's CapFilt, DALL-E 3's descriptive captions) to train better text-to-image models

**Question:** On a 5,000-image test set, a bigram appears in the reference sets of 1,000 images and another in 50. What are their CIDEr idf weights?

- A. 0.2 and 0.01
- B. ln 5 = 1.61 and ln 100 = 4.61
- C. 1,000 and 50
- D. Both 1, because idf is normalised away by the cosine

**Answer:** B. ln 5 = 1.61 and ln 100 = 4.61 — The weight is $\log(|I|/\mathrm{df})$: $\ln(5000/1000)=1.61$ and $\ln(5000/50)=4.61$. The cosine normalises vector length, not the relative weights, so the rarer bigram still counts about 2.9× as much.

**Question:** Why does SCST use the reward of the model's own greedy caption as the baseline?

- A. It makes the reward always positive
- B. It reduces gradient variance without a learned critic, and rewards samples only when they beat what test-time decoding already produces
- C. Greedy captions are the human references
- D. It removes the need for cross-entropy pre-training

**Answer:** B. It reduces gradient variance without a learned critic, and rewards samples only when they beat what test-time decoding already produces — Subtracting a baseline leaves the policy gradient unbiased but lowers its variance. Using the greedy caption means a sample is pushed up only if it beats the model's own test-time output, here +0.426 versus −2.225. SCST still starts from a cross-entropy model.

**Key points**

- Captioning maximises $\sum_t\log p(w_t\mid I,w_{<t})$: a vision encoder plus a text decoder, from Show and Tell (CNN + LSTM, 2015) to BLIP.
- COCO Captions has 5 references per image; the Karpathy split is 113,287 / 5,000 / 5,000 images.
- CIDEr-D: tf-idf n-grams ($n$ = 1–4), clipped cosine with each reference, length penalty $\sigma=6$, ×10; papers print it ×100.
- Idf comes from the whole test set: "a" weighs 0, image-specific words carry the score.
- SCST optimises CIDEr directly with REINFORCE and a greedy baseline; check hallucination (CHAIR) separately, because CIDEr does not.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
