# Visual question answering

> VQA answers a free-form question about an image. VQA v2 (Goyal et al., 2017) pairs each question with a similar image that has a different answer, which punishes language priors, and scores answers by min(#matching humans / 3, 1) over 10 annotators.

Source: https://www.mlfoundry.in/learn/visual-question-answering  
Section: Generative AI › Multimodal Generative AI  
Level: core · 7 min read · updated 2026-10-21

**Visual question answering (VQA):** given an image and a natural-language question about it, produce the answer: "yes", "2", "red", "tennis". The task and the first large benchmark come from Antol et al. (ICCV 2015). **VQA v2** (Goyal, Khot, Summers-Stay, Batra and Parikh, "Making the V in VQA Matter", CVPR 2017) rebalanced it so that every question comes with a similar image whose answer differs. Each question has 10 human answers, and accuracy is $\min(\#\text{humans who gave that answer}/3,\,1)$.

In products, VQA is the "ask about this picture" feature: reading a label for a blind user, checking a photo of a meter or receipt, inspecting a part, answering questions about a chart. Today it is usually served by a multimodal LLM prompted with the image and the question (see **Multimodal LLMs**). The model design is covered in **Vision-language models**. This article covers the benchmark, the metric and the failure that shaped both: answering from the question instead of the image.

## Language priors and the balanced pairs

Questions carry strong hints. "Is there a ...?" is usually answered "yes", because people ask about what they see. "What sport is ...?" is usually tennis or baseball in COCO photos, and "how many" is usually a small number. On VQA v1 a question-only model with no image input scored well above chance. High accuracy did not show that a model looked at the image.

Goyal et al. fixed the dataset, not the model. For each (image, question, answer), annotators picked a visually similar COCO image on which the same question has a different answer. A model that answers from the question alone can now get at most one half of each pair right. Models trained on v1 lost accuracy on v2, which measured how much they had leaned on priors.

> Figure: (a) The same question on two similar images with opposite answers: a blind model that always says "yes" scores 1.0 on one and 0.0 on the other. (b) Scoring an open-ended model: the prompt asks for a single word or phrase, the raw output "Two." is normalised to "2", and five of the ten human answers normalise to "2", so accuracy is 1.0. "3" (three humans) scores 0.9 and "4" (two) scores 0.6. A full sentence matches no one.

## The VQA accuracy metric

$$
\mathrm{Acc}(a)=\frac{1}{10}\sum_{k=1}^{10}\min\!\Big(\frac{\#\{j\ne k:\ h_j=a\}}{3},\ 1\Big)
$$

- $a$: the model's answer after normalisation
- $h_1,\dots,h_{10}$: the 10 human answers, normalised the same way
- $k$: the annotator left out; the score averages over all 10 subsets of 9 humans

The short form, $\min(\#\text{humans}/3,1)$, gives full credit when at least 3 people gave the answer: answers are open-ended, and people disagree on "maroon" versus "red". The official script averages it over leave-one-out subsets so that a human's own accuracy can be computed the same way. Before matching, answers are normalised: lower-cased, punctuation removed, number words mapped to digits ("two" → "2"), and articles dropped.

**Benchmarks (dataset papers)**

- **VQA v2:** about 1.1M questions on about 200K COCO images, 10 answers each; answer types yes/no, number and other
- **Answer vocabulary:** classifier-style models predict over the 3,129 most frequent training answers, the standard set since the bottom-up top-down model (Anderson et al., 2018)
- **VQA-CP:** Agrawal et al., CVPR 2018: re-splits VQA so that each question type's answer distribution differs between train and test
- **GQA:** Hudson and Manning, CVPR 2019: compositional questions generated from scene graphs
- **TextVQA / DocVQA:** Singh et al., CVPR 2019 and Mathew et al., WACV 2021: questions that require reading text in photos or documents
- **VizWiz:** Gurari et al., CVPR 2018: photos taken by blind users, often blurred or off-frame, with unanswerable questions

> Figure: A toy model scores each answer as λ·(noisy visual evidence) + (1 − λ)·log(prior learned from skewed training data), and is scored with the official metric against 10 synthetic annotators. At λ = 0.5, β = 1.5 it gets 0.775 on an unbalanced test set but 0.727 on balanced pairs (−4.8 points). The blind model (dashed ticks) falls from 0.578 to 0.465, and on yes/no from 0.792 to 0.580. At λ = 1 (vision only) the two test sets agree (0.759 and 0.760), though the unbalanced score is lower than at λ = 0.5: on a biased test set, priors pay. Training on balanced pairs at λ = 0.5 narrows the gap to 0.774 versus 0.766.

## Worked example: one counting question

Ten humans answered "how many dogs?": "2" four times, "two" once, "3" three times and "4" twice. After normalisation that is 2 ×5, 3 ×3, 4 ×2.

Official leave-one-out accuracy (computed in Python with the code below)

| Model answer | Humans matching | The 10 subsets of 9 | Acc | Short form min(n/3, 1) |
| --- | --- | --- | --- | --- |
| Two. → 2 | 5 | 4 or 5 matches each: all 1 | 1.0 | 1.0 |
| 3 | 3 | 3 subsets drop a match: 2/3; 7 keep all three: 1 | 0.9 | 1.0 |
| 4 | 2 | 2 subsets: 1/3; 8 subsets: 2/3 | 0.6 | 0.67 |
| two, with no number mapping | 1 | 1 subset: 0; 9 subsets: 1/3 | 0.3 | 0.33 |
| There are two dogs. | 0 | 0 everywhere | 0.0 | 0.0 |

The score rewards answers that match the annotators' format. A correct sentence earns 0. That is why open-ended models are evaluated with an instruction such as "Answer the question using a single word or phrase.", which LLaVA-1.5 used for its short-answer training data.

```python
import re

NUM = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
       "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}
ARTICLES = {"a", "an", "the"}

def normalize(ans: str) -> str:
    ans = re.sub(r"(?<=\d),(?=\d)", "", ans.lower().strip())   # "1,000" -> "1000"
    ans = re.sub(r"[^\w\s]", " ", ans)
    return " ".join(NUM.get(w, w) for w in ans.split() if w not in ARTICLES)

def vqa_accuracy(pred: str, humans: list[str]) -> float:
    p, h = normalize(pred), [normalize(a) for a in humans]
    subsets = [min(sum(a == p for j, a in enumerate(h) if j != k) / 3, 1) for k in range(len(h))]
    return sum(subsets) / len(subsets)

humans = ["2", "2", "2", "2", "two", "3", "3", "3", "4", "4"]
for pred in ["Two.", "3", "4", "There are two dogs."]:
    print(pred, vqa_accuracy(pred, humans))
```

VQA accuracy with a simplified normaliser (run here; prints 1.0, 0.9, 0.6 and 0.0). The official script also fixes contractions and handles more punctuation cases.

> **A high score can come from the question:** Before trusting a VQA number, run a blind baseline: the same model with a blank or shuffled image. If it keeps most of its accuracy, the benchmark is measuring priors. Report accuracy per answer type, since yes/no inflates the total. For your own product data, build complementary pairs the way VQA v2 did, and test absent objects ("is there a cat?" when there is none), which is where multimodal models say "yes" too often (see **Hallucination detection**).

> **Long answers need a different scorer:** Exact match against 10 short answers only works for short answers. For explanations, chart reasoning or document questions, teams extract a final answer with a fixed format and score it, or use a rubric and **LLM-as-a-judge**. Benchmarks such as DocVQA use ANLS (average normalised Levenshtein similarity), which gives partial credit for near-miss strings read from a document.

**Strengths**

- Soft accuracy tolerates real disagreement among annotators (red vs maroon)
- Balanced pairs make a question-only model visibly fail
- Per-type scores (yes/no, number, other) show where a model is weak

**Limitations**

- Exact string matching punishes correct but differently worded answers
- Priors still leak through: balancing is pairwise, not across all question types
- Short answers say nothing about whether the reasoning used the right region of the image

**Question:** Ten humans answered: "red" ×3, "dark red" ×5, "maroon" ×2. What is the official VQA accuracy of the answer "red"?

- A. 0.3
- B. 0.6
- C. 0.9
- D. 1.0

**Answer:** C. 0.9 — Three humans match. In the 3 subsets that leave out one of them, 2 matches remain: $2/3$. In the other 7 subsets all 3 remain: 1. So $(3\cdot\tfrac23+7\cdot1)/10=0.9$. The short form $\min(3/3,1)$ would give 1.0.

**Question:** On a balanced yes/no set, where every question appears once with answer yes and once with no, what is the best possible score of a model that never sees the image?

- A. 0%
- B. About 50%
- C. About 70%, the share of yes in training
- D. 100%, if it learned the priors well

**Answer:** B. About 50% — It must give the same answer to both images of a pair, so it is right on exactly one of them. In the figure the blind model's yes/no accuracy falls from 0.792 on the unbalanced set to 0.580 on the balanced one. The remainder above 0.5 is partial credit from annotators who disagreed.

**Key points**

- VQA: image + question → short answer. VQA v2 (Goyal et al., 2017) has about 1.1M questions on COCO images with 10 human answers each.
- Accuracy $=\min(\#\text{matching humans}/3,1)$, averaged over the 10 leave-one-out subsets: 3 matches give 0.9, 2 give 0.6.
- Normalise first (case, punctuation, "two" → "2", articles), and prompt generative models for a single word or phrase.
- Language priors let blind models score well; complementary image pairs cap them at one answer per pair.
- Always report a blind baseline and per-type accuracy; use VQA-CP, POPE-style absent-object tests and your own pairs for robustness.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
