Definition
Image captioning generating a natural-language description of an image, modelled as . The neural template is Show and Tell (Vinyals, Toshev, Bengio and Erhan, CVPR 2015): a CNN encodes the image and an LSTM writes the words. Show, Attend and Tell (Xu et al., ICML 2015) added attention over spatial features. BLIP (Li et al., ICML 2022) and BLIP-2 (2023) pre-train on web image–text pairs. The standard benchmark is COCO Captions (Chen et al., 2015), scored with CIDEr (Vedantam, Zitnick and Parikh, CVPR 2015).
Captions do real work in products. They become alt text for screen readers, search-index text for photo libraries, and frame descriptions for video search. They are also used to relabel training data. DALL-E 3 (Betker et al., 2023) was trained on synthetic, detailed captions in place of noisy alt text, because better captions gave better prompt following.
The applied questions are which model to use, how to decode, and how to measure quality. The architectures are covered in BLIP, Vision-language models and Multimodal LLMs. Here we focus on the decoding objective, the CIDEr metric and how captioners are tuned against it.
Pipeline: encode, decode, score
- the image, seen through the vision encoder's features (a CNN vector in Show and Tell, patch features in ViT-based models)
- the caption, ending in an end-of-sentence token
- decoder weights, trained with teacher forcing on human captions (cross-entropy)
Decoding is usually beam search with a small beam and a length limit. Greedy decoding gives short, safe captions; sampling gives more variety and more errors (see Greedy and beam search). BLIP-style models also accept a text prefix, such as "a picture of", that the decoder continues.
CIDEr: consensus with the references
BLEU and ROUGE count n-gram overlap (see BLEU, ROUGE and METEOR). CIDEr changes two things. It weights each n-gram by how rare it is across the test set's images, and it averages agreement over all references for the image. A caption scores well when it says what most annotators said and uses the words that set this image apart from the others.
- count of n-gram in sentence
- number of images in the test set, and how many of their reference sets contain
- number of references for the image (5 on COCO)
- clips candidate n-gram weights at the reference's, so repeating a word earns nothing extra
- length-penalty width, 6 in the standard implementation; is sentence length
The final score averages to 4 with equal weights. The original CIDEr has no clipping, no length penalty and no factor 10. CIDEr-D adds them to make the metric harder to game, and it is what the COCO evaluation server and pycocoevalcap report. Papers usually print it ×100: a CIDEr of 1.2 appears as 120.
Datasets, metrics and training tricks
- COCO Captions
- 5 human captions per image; 82,783 train2014 and 40,504 val2014 images
- Karpathy split
- the usual COCO protocol: 113,287 train, 5,000 val and 5,000 test images
- nocaps
- Agrawal et al., ICCV 2019: Open Images photos with objects absent from COCO captions, to test novel-object captioning
- SPICE
- Anderson et al., ECCV 2016: F-score over scene-graph tuples (objects, attributes, relations) parsed from the captions
- CHAIR
- Rohrbach et al., EMNLP 2018: share of mentioned objects that are not in the image, a hallucination rate for captions
- SCST
- Rennie et al., CVPR 2017: REINFORCE on the CIDEr reward with the greedy caption as baseline
Worked example: four captions for the giraffe
| Candidate | CIDEr₁ | CIDEr₂ | CIDEr₃ | CIDEr₄ | CIDEr-D |
|---|---|---|---|---|---|
| a giraffe eating leaves from a tree | 5.920 | 3.271 | 1.440 | 1.323 | 2.989 |
| a giraffe standing next to a tree | 4.729 | 2.793 | 1.395 | 1.334 | 2.563 |
| a horse eating grass in a field | 1.229 | 0.303 | 0.279 | 0.000 | 0.453 |
| an animal standing in a field | 0.763 | 0.307 | 0.283 | 0.000 | 0.338 |
The idf weights explain the ranking. With , "a" appears in all 6 reference sets: weight . "in" appears in 4: . "giraffe", "leaves" and "tree" appear only in the giraffe set: . A caption earns its score with the specific words. On a real 5,000-image test set the spread is wider: an n-gram in 1,000 reference sets weighs , and one in 50 weighs .
- a caption sampled from the model
- the greedy caption, whose reward is the baseline (the self-critic)
- CIDEr-D of the caption against the references
SCST fine-tunes a cross-entropy model directly on the metric. In the diagram the greedy caption scores 2.563. A sample scoring 2.989 gets advantage and becomes more likely; one scoring 0.338 gets and becomes less likely. No value network is needed, and test-time decoding (greedy or beam) is exactly what the baseline measures.
1from PIL import Image2from transformers import BlipProcessor, BlipForConditionalGeneration3from pycocoevalcap.cider.cider import Cider45name = "Salesforce/blip-image-captioning-large"6processor = BlipProcessor.from_pretrained(name)7model = BlipForConditionalGeneration.from_pretrained(name)89def caption(path, prefix="a picture of"):10 inputs = processor(images=Image.open(path).convert("RGB"), text=prefix, return_tensors="pt")11 ids = model.generate(**inputs, num_beams=3, max_new_tokens=30)12 return processor.decode(ids[0], skip_special_tokens=True).removeprefix(prefix).strip()1314# idf comes from the references of every image in gts, so score the whole test set at once15gts = {img_id: refs[img_id] for img_id in test_ids} # 5 lower-cased references each16res = {img_id: [caption(f"val/{img_id}.jpg")] for img_id in test_ids}17corpus_cider, per_image = Cider().compute_score(gts, res) # ~1.2 is reported as "120"Where it's used
Accessibility
alt text for images that have none; a person should review it where accuracy matters
Search and indexing
captions make photo and video libraries searchable by text with an ordinary keyword or embedding index
Training data
recaptioning web images with a captioner (BLIP's CapFilt, DALL-E 3's descriptive captions) to train better text-to-image models
Check yourself
On a 5,000-image test set, a bigram appears in the reference sets of 1,000 images and another in 50. What are their CIDEr idf weights?
Check yourself
Why does SCST use the reward of the model's own greedy caption as the baseline?
Remember
- ✓Captioning maximises : a vision encoder plus a text decoder, from Show and Tell (CNN + LSTM, 2015) to BLIP.
- ✓COCO Captions has 5 references per image; the Karpathy split is 113,287 / 5,000 / 5,000 images.
- ✓CIDEr-D: tf-idf n-grams ( = 1–4), clipped cosine with each reference, length penalty , ×10; papers print it ×100.
- ✓Idf comes from the whole test set: "a" weighs 0, image-specific words carry the score.
- ✓SCST optimises CIDEr directly with REINFORCE and a greedy baseline; check hallucination (CHAIR) separately, because CIDEr does not.