Visual question answering

VQA answers a free-form question about an image. VQA v2 (Goyal et al., 2017) pairs each question with a similar image that has a different answer, which punishes language priors, and scores answers by min(#matching humans / 3, 1) over 10 annotators.

7 min read649 wordscore0 viewsupdated 2026-10-21

Definition

Visual question answering (VQA) given an image and a natural-language question about it, produce the answer: "yes", "2", "red", "tennis". The task and the first large benchmark come from Antol et al. (ICCV 2015). VQA v2 (Goyal, Khot, Summers-Stay, Batra and Parikh, "Making the V in VQA Matter", CVPR 2017) rebalanced it so that every question comes with a similar image whose answer differs. Each question has 10 human answers, and accuracy is min⁡(#humans who gave that answer/3, 1)\min(\#\text{humans who gave that answer}/3,\,1).

In products, VQA is the "ask about this picture" feature: reading a label for a blind user, checking a photo of a meter or receipt, inspecting a part, answering questions about a chart. Today it is usually served by a multimodal LLM prompted with the image and the question (see Multimodal LLMs). The model design is covered in Vision-language models. This article covers the benchmark, the metric and the failure that shaped both: answering from the question instead of the image.

Language priors and the balanced pairs

Questions carry strong hints. "Is there a ...?" is usually answered "yes", because people ask about what they see. "What sport is ...?" is usually tennis or baseball in COCO photos, and "how many" is usually a small number. On VQA v1 a question-only model with no image input scored well above chance. High accuracy did not show that a model looked at the image.

Goyal et al. fixed the dataset, not the model. For each (image, question, answer), annotators picked a visually similar COCO image on which the same question has a different answer. A model that answers from the question alone can now get at most one half of each pair right. Models trained on v1 lost accuracy on v2, which measured how much they had leaned on priors.

(a) VQA v2 balancing: one question, two similar images, different answersimage A · humans: yes ×10image B · humans: no ×10Q: Is the umbrella open?asked about both imagesblind model: "yes" to both → 1.0 and 0.0a language prior can win only one half of a pair(b) answering and scoring an open-ended model<image> How many dogs?Answer the question usinga single word or phrase.MLLM"Two."rawnormalise → "2"lower-case, strip punctuation,drop a/an/the, two → 210 human answers2222two33344normalised: "2" ×5 · "3" ×3 · "4" ×2Acc(a) = mean over the 10 leave-one-outsets of 9 of min(#humans saying a / 3, 1)"2" 1.0"3" 0.9"4" 0.6"5" 0.0no format instruction: "There are two dogs." → "there are 2 dogs" → 0 matches → 0.0no number mapping: "two" matches only the one human who typed "two" → 0.3answer types in VQA v2: yes/no · number · other; accuracy is reported for each and overall
fig.(a) The same question on two similar images with opposite answers: a blind model that always says "yes" scores 1.0 on one and 0.0 on the other. (b) Scoring an open-ended model: the prompt asks for a single word or phrase, the raw output "Two." is normalised to "2", and five of the ten human answers normalise to "2", so accuracy is 1.0. "3" (three humans) scores 0.9 and "4" (two) scores 0.6. A full sentence matches no one.

The VQA accuracy metric

Acc(a)=110∑k=110min⁡ ⁣(#{j≠k: hj=a}3, 1)\mathrm{Acc}(a)=\frac{1}{10}\sum_{k=1}^{10}\min\!\Big(\frac{\#\{j\ne k:\ h_j=a\}}{3},\ 1\Big)
aa
the model's answer after normalisation
h1,…,h10h_1,\dots,h_{10}
the 10 human answers, normalised the same way
kk
the annotator left out; the score averages over all 10 subsets of 9 humans

The short form, min⁡(#humans/3,1)\min(\#\text{humans}/3,1), gives full credit when at least 3 people gave the answer: answers are open-ended, and people disagree on "maroon" versus "red". The official script averages it over leave-one-out subsets so that a human's own accuracy can be computed the same way. Before matching, answers are normalised: lower-cased, punctuation removed, number words mapped to digits ("two" → "2"), and articles dropped.

Benchmarks (dataset papers)

VQA v2
about 1.1M questions on about 200K COCO images, 10 answers each; answer types yes/no, number and other
Answer vocabulary
classifier-style models predict over the 3,129 most frequent training answers, the standard set since the bottom-up top-down model (Anderson et al., 2018)
VQA-CP
Agrawal et al., CVPR 2018: re-splits VQA so that each question type's answer distribution differs between train and test
GQA
Hudson and Manning, CVPR 2019: compositional questions generated from scene graphs
TextVQA / DocVQA
Singh et al., CVPR 2019 and Mathew et al., WACV 2021: questions that require reading text in photos or documents
VizWiz
Gurari et al., CVPR 2018: photos taken by blind users, often blurred or off-frame, with unanswerable questions
answer score = λ·(visual evidence) + (1 − λ)·log prior(answer | question type)150 questions per type · VQA accuracy vs 10 humans · λ 0.50 · β 1.50 · prior: skewed train0.000.250.500.751.00.83.84yes/no.70.64color.76.73count.81.71sport.77.73overallunbalanced test (v1-like, same skew as training)balanced pairs (v2-like)blind model (λ = 0): answers from the question aloneone complementary pair per type: same question, two images (rose: same answer to both)is there a dog?A: truth no → no (1.0)B: truth yes → yes (1.0)what color is the bus?A: truth green → black (0.0)B: truth black → black (1.0)how many people are there?A: truth 2 → 2 (1.0)B: truth 3 → 3 (1.0)what sport is being played?A: truth frisbee → baseball (0.0)B: truth soccer → tennis (0.6)overall: unbalanced 0.775 → balanced 0.727 (−4.8 pts) · blind 0.578 → 0.465 (−11.3)
presets
fig.A toy model scores each answer as λ·(noisy visual evidence) + (1 − λ)·log(prior learned from skewed training data), and is scored with the official metric against 10 synthetic annotators. At λ = 0.5, β = 1.5 it gets 0.775 on an unbalanced test set but 0.727 on balanced pairs (−4.8 points). The blind model (dashed ticks) falls from 0.578 to 0.465, and on yes/no from 0.792 to 0.580. At λ = 1 (vision only) the two test sets agree (0.759 and 0.760), though the unbalanced score is lower than at λ = 0.5: on a biased test set, priors pay. Training on balanced pairs at λ = 0.5 narrows the gap to 0.774 versus 0.766.

Worked example: one counting question

Ten humans answered "how many dogs?": "2" four times, "two" once, "3" three times and "4" twice. After normalisation that is 2 ×5, 3 ×3, 4 ×2.

Model answerHumans matchingThe 10 subsets of 9AccShort form min(n/3, 1)
Two. → 254 or 5 matches each: all 11.01.0
333 subsets drop a match: 2/3; 7 keep all three: 10.91.0
422 subsets: 1/3; 8 subsets: 2/30.60.67
two, with no number mapping11 subset: 0; 9 subsets: 1/30.30.33
There are two dogs.00 everywhere0.00.0
Official leave-one-out accuracy (computed in Python with the code below)

The score rewards answers that match the annotators' format. A correct sentence earns 0. That is why open-ended models are evaluated with an instruction such as "Answer the question using a single word or phrase.", which LLaVA-1.5 used for its short-answer training data.

python
1import re
2
3NUM = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
4 "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}
5ARTICLES = {"a", "an", "the"}
6
7def normalize(ans: str) -> str:
8 ans = re.sub(r"(?<=\d),(?=\d)", "", ans.lower().strip()) # "1,000" -> "1000"
9 ans = re.sub(r"[^\w\s]", " ", ans)
10 return " ".join(NUM.get(w, w) for w in ans.split() if w not in ARTICLES)
11
12def vqa_accuracy(pred: str, humans: list[str]) -> float:
13 p, h = normalize(pred), [normalize(a) for a in humans]
14 subsets = [min(sum(a == p for j, a in enumerate(h) if j != k) / 3, 1) for k in range(len(h))]
15 return sum(subsets) / len(subsets)
16
17humans = ["2", "2", "2", "2", "two", "3", "3", "3", "4", "4"]
18for pred in ["Two.", "3", "4", "There are two dogs."]:
19 print(pred, vqa_accuracy(pred, humans))
VQA accuracy with a simplified normaliser (run here; prints 1.0, 0.9, 0.6 and 0.0). The official script also fixes contractions and handles more punctuation cases.

Use when

  • +Soft accuracy tolerates real disagreement among annotators (red vs maroon)
  • +Balanced pairs make a question-only model visibly fail
  • +Per-type scores (yes/no, number, other) show where a model is weak

Avoid when

  • −Exact string matching punishes correct but differently worded answers
  • −Priors still leak through: balancing is pairwise, not across all question types
  • −Short answers say nothing about whether the reasoning used the right region of the image

Check yourself

Ten humans answered: "red" ×3, "dark red" ×5, "maroon" ×2. What is the official VQA accuracy of the answer "red"?

Check yourself

On a balanced yes/no set, where every question appears once with answer yes and once with no, what is the best possible score of a model that never sees the image?

Remember

  • ✓VQA: image + question → short answer. VQA v2 (Goyal et al., 2017) has about 1.1M questions on COCO images with 10 human answers each.
  • ✓Accuracy =min⁡(#matching humans/3,1)=\min(\#\text{matching humans}/3,1), averaged over the 10 leave-one-out subsets: 3 matches give 0.9, 2 give 0.6.
  • ✓Normalise first (case, punctuation, "two" → "2", articles), and prompt generative models for a single word or phrase.
  • ✓Language priors let blind models score well; complementary image pairs cap them at one answer per pair.
  • ✓Always report a blind baseline and per-type accuracy; use VQA-CP, POPE-style absent-object tests and your own pairs for robustness.
← back to Generative AI roadmap