Definition
Visual question answering (VQA) given an image and a natural-language question about it, produce the answer: "yes", "2", "red", "tennis". The task and the first large benchmark come from Antol et al. (ICCV 2015). VQA v2 (Goyal, Khot, Summers-Stay, Batra and Parikh, "Making the V in VQA Matter", CVPR 2017) rebalanced it so that every question comes with a similar image whose answer differs. Each question has 10 human answers, and accuracy is .
In products, VQA is the "ask about this picture" feature: reading a label for a blind user, checking a photo of a meter or receipt, inspecting a part, answering questions about a chart. Today it is usually served by a multimodal LLM prompted with the image and the question (see Multimodal LLMs). The model design is covered in Vision-language models. This article covers the benchmark, the metric and the failure that shaped both: answering from the question instead of the image.
Language priors and the balanced pairs
Questions carry strong hints. "Is there a ...?" is usually answered "yes", because people ask about what they see. "What sport is ...?" is usually tennis or baseball in COCO photos, and "how many" is usually a small number. On VQA v1 a question-only model with no image input scored well above chance. High accuracy did not show that a model looked at the image.
Goyal et al. fixed the dataset, not the model. For each (image, question, answer), annotators picked a visually similar COCO image on which the same question has a different answer. A model that answers from the question alone can now get at most one half of each pair right. Models trained on v1 lost accuracy on v2, which measured how much they had leaned on priors.
The VQA accuracy metric
- the model's answer after normalisation
- the 10 human answers, normalised the same way
- the annotator left out; the score averages over all 10 subsets of 9 humans
The short form, , gives full credit when at least 3 people gave the answer: answers are open-ended, and people disagree on "maroon" versus "red". The official script averages it over leave-one-out subsets so that a human's own accuracy can be computed the same way. Before matching, answers are normalised: lower-cased, punctuation removed, number words mapped to digits ("two" → "2"), and articles dropped.
Benchmarks (dataset papers)
- VQA v2
- about 1.1M questions on about 200K COCO images, 10 answers each; answer types yes/no, number and other
- Answer vocabulary
- classifier-style models predict over the 3,129 most frequent training answers, the standard set since the bottom-up top-down model (Anderson et al., 2018)
- VQA-CP
- Agrawal et al., CVPR 2018: re-splits VQA so that each question type's answer distribution differs between train and test
- GQA
- Hudson and Manning, CVPR 2019: compositional questions generated from scene graphs
- TextVQA / DocVQA
- Singh et al., CVPR 2019 and Mathew et al., WACV 2021: questions that require reading text in photos or documents
- VizWiz
- Gurari et al., CVPR 2018: photos taken by blind users, often blurred or off-frame, with unanswerable questions
Worked example: one counting question
Ten humans answered "how many dogs?": "2" four times, "two" once, "3" three times and "4" twice. After normalisation that is 2 ×5, 3 ×3, 4 ×2.
| Model answer | Humans matching | The 10 subsets of 9 | Acc | Short form min(n/3, 1) |
|---|---|---|---|---|
| Two. → 2 | 5 | 4 or 5 matches each: all 1 | 1.0 | 1.0 |
| 3 | 3 | 3 subsets drop a match: 2/3; 7 keep all three: 1 | 0.9 | 1.0 |
| 4 | 2 | 2 subsets: 1/3; 8 subsets: 2/3 | 0.6 | 0.67 |
| two, with no number mapping | 1 | 1 subset: 0; 9 subsets: 1/3 | 0.3 | 0.33 |
| There are two dogs. | 0 | 0 everywhere | 0.0 | 0.0 |
The score rewards answers that match the annotators' format. A correct sentence earns 0. That is why open-ended models are evaluated with an instruction such as "Answer the question using a single word or phrase.", which LLaVA-1.5 used for its short-answer training data.
1import re23NUM = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",4 "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}5ARTICLES = {"a", "an", "the"}67def normalize(ans: str) -> str:8 ans = re.sub(r"(?<=\d),(?=\d)", "", ans.lower().strip()) # "1,000" -> "1000"9 ans = re.sub(r"[^\w\s]", " ", ans)10 return " ".join(NUM.get(w, w) for w in ans.split() if w not in ARTICLES)1112def vqa_accuracy(pred: str, humans: list[str]) -> float:13 p, h = normalize(pred), [normalize(a) for a in humans]14 subsets = [min(sum(a == p for j, a in enumerate(h) if j != k) / 3, 1) for k in range(len(h))]15 return sum(subsets) / len(subsets)1617humans = ["2", "2", "2", "2", "two", "3", "3", "3", "4", "4"]18for pred in ["Two.", "3", "4", "There are two dogs."]:19 print(pred, vqa_accuracy(pred, humans))Use when
- +Soft accuracy tolerates real disagreement among annotators (red vs maroon)
- +Balanced pairs make a question-only model visibly fail
- +Per-type scores (yes/no, number, other) show where a model is weak
Avoid when
- −Exact string matching punishes correct but differently worded answers
- −Priors still leak through: balancing is pairwise, not across all question types
- −Short answers say nothing about whether the reasoning used the right region of the image
Check yourself
Ten humans answered: "red" ×3, "dark red" ×5, "maroon" ×2. What is the official VQA accuracy of the answer "red"?
Check yourself
On a balanced yes/no set, where every question appears once with answer yes and once with no, what is the best possible score of a model that never sees the image?
Remember
- ✓VQA: image + question → short answer. VQA v2 (Goyal et al., 2017) has about 1.1M questions on COCO images with 10 human answers each.
- ✓Accuracy , averaged over the 10 leave-one-out subsets: 3 matches give 0.9, 2 give 0.6.
- ✓Normalise first (case, punctuation, "two" → "2", articles), and prompt generative models for a single word or phrase.
- ✓Language priors let blind models score well; complementary image pairs cap them at one answer per pair.
- ✓Always report a blind baseline and per-type accuracy; use VQA-CP, POPE-style absent-object tests and your own pairs for robustness.