Image captioning

Image captioning generates a sentence for an image with a vision encoder and a text decoder, from Show and Tell (2015) to BLIP. It is scored on COCO's 5 references per image with CIDEr: tf-idf n-gram cosine, ×10, with a length penalty.

7 min read726 wordscore0 viewsupdated 2026-10-21

Definition

Image captioning generating a natural-language description of an image, modelled as p(caption∣image)p(\text{caption}\mid\text{image}). The neural template is Show and Tell (Vinyals, Toshev, Bengio and Erhan, CVPR 2015): a CNN encodes the image and an LSTM writes the words. Show, Attend and Tell (Xu et al., ICML 2015) added attention over spatial features. BLIP (Li et al., ICML 2022) and BLIP-2 (2023) pre-train on web image–text pairs. The standard benchmark is COCO Captions (Chen et al., 2015), scored with CIDEr (Vedantam, Zitnick and Parikh, CVPR 2015).

Captions do real work in products. They become alt text for screen readers, search-index text for photo libraries, and frame descriptions for video search. They are also used to relabel training data. DALL-E 3 (Betker et al., 2023) was trained on synthetic, detailed captions in place of noisy alt text, because better captions gave better prompt following.

The applied questions are which model to use, how to decode, and how to measure quality. The architectures are covered in BLIP, Vision-language models and Multimodal LLMs. Here we focus on the decoding objective, the CIDEr metric and how captioners are tuned against it.

Pipeline: encode, decode, score

log⁡p(S∣I)=∑t=1Tlog⁡pθ(wt∣I, w1,…,wt−1)\log p(S\mid I)=\sum_{t=1}^{T}\log p_\theta\big(w_t\mid I,\,w_1,\dots,w_{t-1}\big)
II
the image, seen through the vision encoder's features (a CNN vector in Show and Tell, patch features in ViT-based models)
S=(w1,…,wT)S=(w_1,\dots,w_T)
the caption, ending in an end-of-sentence token
θ\theta
decoder weights, trained with teacher forcing on human captions (cross-entropy)
(a) generate: p(caption | image) = Πₜ p(wₜ | image, w₁…wₜ₋₁)imagevision encoderCNN or ViTpatch featurestext decoderLSTM / transformerattends to patchescaptiona giraffe eating leavesfrom a treebeam search, width 3 (example)Σ log pbeam 1 a giraffe eating leaves from a tree−3.9beam 2 a giraffe standing next to a tree−4.4beam 3 a tall giraffe eating from a tree−5.0(b) score against 5 human references (COCO-style)r1 a giraffe eating leaves from a tall treer2 a giraffe standing next to a tree in a fieldr3 a tall giraffe reaching up to eat leavesr4 a giraffe feeding on the leaves of a treer5 one giraffe eating from the top of a treetf-idf n-grams, n = 1…4idf: a 0.00 · on 0.18tree 1.79 · giraffe 1.79clipped cosine × 10CIDEr-D 2.989candidate (a)toy 6-imagecorpus(c) train: cross-entropy, then self-critical RL on CIDEr (SCST)stage 1: MLEteacher forcing−Σ log p(w*ₜ | …)on human captionsgreedy ŵ a giraffe standing next to a treer = 2.563baselinesample w¹ a giraffe eating leaves from a treer = 2.989A = +0.426sample w² an animal standing in a fieldr = 0.338A = −2.225stage 2: ∇L ≈ −(r(wˢ) − r(ŵ)) ∇ log p(wˢ): w¹ pushed up, w² down
fig.(a) The decoder predicts each word from the image features and the words so far; beam search keeps the 3 best partial captions (example log-probabilities). (b) The output is scored against 5 human references by tf-idf weighted n-grams. On a toy 6-image corpus the word "a" gets idf 0 and "giraffe" 1.79, and the top beam scores CIDEr-D 2.989. (c) After cross-entropy training, SCST samples captions and rewards each by its CIDEr minus the greedy caption's: +0.426 for w¹, −2.225 for w².

Decoding is usually beam search with a small beam and a length limit. Greedy decoding gives short, safe captions; sampling gives more variety and more errors (see Greedy and beam search). BLIP-style models also accept a text prefix, such as "a picture of", that the decoder continues.

CIDEr: consensus with the references

BLEU and ROUGE count n-gram overlap (see BLEU, ROUGE and METEOR). CIDEr changes two things. It weights each n-gram by how rare it is across the test set's images, and it averages agreement over all references for the image. A caption scores well when it says what most annotators said and uses the words that set this image apart from the others.

gk(s)=hk(s)⋅log⁡∣I∣max⁡ ⁣(1, dfk),CIDEr-Dn(c,S)=10m∑j=1me−(lc−lsj)22σ2 min⁡ ⁣(gn(c),gn(sj))⋅gn(sj)∥gn(c)∥ ∥gn(sj)∥g_k(s)=h_k(s)\cdot\log\frac{|I|}{\max\!\big(1,\ \mathrm{df}_k\big)},\qquad \mathrm{CIDEr\text{-}D}_n(c,S)=\frac{10}{m}\sum_{j=1}^{m} e^{-\frac{(l_c-l_{s_j})^2}{2\sigma^2}}\,\frac{\min\!\big(g^n(c),g^n(s_j)\big)\cdot g^n(s_j)}{\lVert g^n(c)\rVert\,\lVert g^n(s_j)\rVert}
hk(s)h_k(s)
count of n-gram kk in sentence ss
∣I∣, dfk|I|,\ \mathrm{df}_k
number of images in the test set, and how many of their reference sets contain kk
mm
number of references for the image (5 on COCO)
min⁡(⋅,⋅)\min(\cdot,\cdot)
clips candidate n-gram weights at the reference's, so repeating a word earns nothing extra
σ\sigma
length-penalty width, 6 in the standard implementation; ll is sentence length

The final score averages n=1n=1 to 4 with equal weights. The original CIDEr has no clipping, no length penalty and no factor 10. CIDEr-D adds them to make the metric harder to game, and it is what the COCO evaluation server and pycocoevalcap report. Papers usually print it ×100: a CIDEr of 1.2 appears as 120.

Datasets, metrics and training tricks

COCO Captions
5 human captions per image; 82,783 train2014 and 40,504 val2014 images
Karpathy split
the usual COCO protocol: 113,287 train, 5,000 val and 5,000 test images
nocaps
Agrawal et al., ICCV 2019: Open Images photos with objects absent from COCO captions, to test novel-object captioning
SPICE
Anderson et al., ECCV 2016: F-score over scene-graph tuples (objects, attributes, relations) parsed from the captions
CHAIR
Rohrbach et al., EMNLP 2018: share of mentioned objects that are not in the image, a hallucination rate for captions
SCST
Rennie et al., CVPR 2017: REINFORCE on the CIDEr reward with the greedy caption as baseline
image: giraffe · 5 human references · idf from 6 imagesr1 a giraffe eating leaves from a tall tree r2 a giraffe standing next to a tree in a field r3 a tall giraffe reaching up to eat leaves r4 a giraffe feeding on the leaves of a tree r5 one giraffe eating from the top of a tree candidate (specific, 7 words) · number under each word = idf = ln 6 − ln dfa0.00giraffe1.79eating1.79leaves1.79from1.79a0.00tree1.79CIDEr_15.920CIDEr_23.271CIDEr_31.440CIDEr_41.323010 (identical to every reference)CIDEr-D = mean over n = 1…4 = 2.989length penalty exp(−Δ²/2σ²), σ = 6: mean factor over the 5 references = 0.949every word appears in the references; rarest match "giraffe" has idf 1.792
image
candidate
fig.CIDEr-D computed live, as in pycocoevalcap, with idf from 6 toy images × 5 references. Giraffe, specific caption: 2.989 (CIDEr₁…₄ = 5.920, 3.271, 1.440, 1.323). The generic "an animal standing in a field" scores 0.338, below the wrong-object "a horse eating grass in a field" at 0.453: CIDEr checks consensus wording, not truth. The 19-word long caption scores 0.443; turn off the length penalty and it jumps to 1.704. Set max n-gram to 1 and the specific caption scores 5.920. The word "a" occurs in all 6 reference sets, so its idf is ln 6 − ln 6 = 0.

Worked example: four captions for the giraffe

CandidateCIDEr₁CIDEr₂CIDEr₃CIDEr₄CIDEr-D
a giraffe eating leaves from a tree5.9203.2711.4401.3232.989
a giraffe standing next to a tree4.7292.7931.3951.3342.563
a horse eating grass in a field1.2290.3030.2790.0000.453
an animal standing in a field0.7630.3070.2830.0000.338
CIDEr-D against the 5 giraffe references, idf from the 6-image toy corpus, σ = 6 (computed in Python)

The idf weights explain the ranking. With ∣I∣=6|I|=6, "a" appears in all 6 reference sets: weight ln⁡6−ln⁡6=0\ln 6-\ln 6=0. "in" appears in 4: ln⁡(6/4)=0.405\ln(6/4)=0.405. "giraffe", "leaves" and "tree" appear only in the giraffe set: ln⁡6=1.792\ln 6=1.792. A caption earns its score with the specific words. On a real 5,000-image test set the spread is wider: an n-gram in 1,000 reference sets weighs ln⁡5=1.61\ln 5=1.61, and one in 50 weighs ln⁡100=4.61\ln 100=4.61.

∇θL(θ)≈−(r(ws)−r(w^)) ∇θlog⁡pθ(ws∣I)\nabla_\theta L(\theta)\approx-\big(r(w^s)-r(\hat w)\big)\,\nabla_\theta\log p_\theta(w^s\mid I)
wsw^s
a caption sampled from the model
w^\hat w
the greedy caption, whose reward is the baseline (the self-critic)
rr
CIDEr-D of the caption against the references

SCST fine-tunes a cross-entropy model directly on the metric. In the diagram the greedy caption scores 2.563. A sample scoring 2.989 gets advantage +0.426+0.426 and becomes more likely; one scoring 0.338 gets −2.225-2.225 and becomes less likely. No value network is needed, and test-time decoding (greedy or beam) is exactly what the baseline measures.

python
1from PIL import Image
2from transformers import BlipProcessor, BlipForConditionalGeneration
3from pycocoevalcap.cider.cider import Cider
4
5name = "Salesforce/blip-image-captioning-large"
6processor = BlipProcessor.from_pretrained(name)
7model = BlipForConditionalGeneration.from_pretrained(name)
8
9def caption(path, prefix="a picture of"):
10 inputs = processor(images=Image.open(path).convert("RGB"), text=prefix, return_tensors="pt")
11 ids = model.generate(**inputs, num_beams=3, max_new_tokens=30)
12 return processor.decode(ids[0], skip_special_tokens=True).removeprefix(prefix).strip()
13
14# idf comes from the references of every image in gts, so score the whole test set at once
15gts = {img_id: refs[img_id] for img_id in test_ids} # 5 lower-cased references each
16res = {img_id: [caption(f"val/{img_id}.jpg")] for img_id in test_ids}
17corpus_cider, per_image = Cider().compute_score(gts, res) # ~1.2 is reported as "120"
Caption with BLIP and score with pycocoevalcap (not run here: needs transformers, torch and pycocoevalcap; the official pipeline also tokenises with the PTB tokenizer, which needs Java)

Where it's used

Accessibility

alt text for images that have none; a person should review it where accuracy matters

Search and indexing

captions make photo and video libraries searchable by text with an ordinary keyword or embedding index

Training data

recaptioning web images with a captioner (BLIP's CapFilt, DALL-E 3's descriptive captions) to train better text-to-image models

Check yourself

On a 5,000-image test set, a bigram appears in the reference sets of 1,000 images and another in 50. What are their CIDEr idf weights?

Check yourself

Why does SCST use the reward of the model's own greedy caption as the baseline?

Remember

  • ✓Captioning maximises ∑tlog⁡p(wt∣I,w<t)\sum_t\log p(w_t\mid I,w_{<t}): a vision encoder plus a text decoder, from Show and Tell (CNN + LSTM, 2015) to BLIP.
  • ✓COCO Captions has 5 references per image; the Karpathy split is 113,287 / 5,000 / 5,000 images.
  • ✓CIDEr-D: tf-idf n-grams (nn = 1–4), clipped cosine with each reference, length penalty σ=6\sigma=6, ×10; papers print it ×100.
  • ✓Idf comes from the whole test set: "a" weighs 0, image-specific words carry the score.
  • ✓SCST optimises CIDEr directly with REINFORCE and a greedy baseline; check hallucination (CHAIR) separately, because CIDEr does not.
← back to Generative AI roadmap