Question answering

Answer a question from text: extract a span by scoring start and end tokens (SQuAD), retrieve passages first for open-domain QA, or generate the answer; scored by exact match and token F1.

8 min read711 wordscore0 viewsupdated 2026-10-15

Definition

Question answering (QA) maps a natural-language question, and optionally a passage or a corpus, to an answer. Extractive QA returns a span of a given passage (SQuAD, Rajpurkar et al. 2016). Open-domain QA first retrieves passages from a large corpus such as Wikipedia, then reads them (DrQA, Chen et al. 2017). Generative QA writes the answer with a decoder, with or without retrieved evidence.

The task

Input: a question such as “Which prize did Curie win in 1903?” and the passage “Marie Curie won the Nobel Prize in Physics in 1903 and the Nobel Prize in Chemistry in 1911.” Output: “the Nobel Prize in Physics”.

The hard part is matching the question's meaning to the right place in the text. The passage holds two prizes and two years, so word overlap alone cannot decide. The model has to tie “1903” to the right prize. Some questions have no answer in the passage, and the model must notice that instead of guessing.

Earlier systems were pipelines: question-type classification, named-entity tagging, passage retrieval and hand-written answer-extraction rules. IBM Watson won Jeopardy! in 2011 with a large ensemble of such components. Neural readers replaced most of that with one model trained on question–passage–answer triples.

Extractive QA: start and end logits

BERT (Devlin et al. 2019) frames span extraction as two token classifications. The input is [CLS] question [SEP] passage [SEP]. The only new parameters are a start vector SS and an end vector EE, each in RH\mathbb{R}^H.

open-domain QA: retriever → reader (DrQA 2017, DPR 2020)questionWhich prize didCurie win in 1903?retrieverBM25 (sparse) orDPR: E_Q(q)·E_P(p),MIPS over 21M psgstop-k passages1 Marie Curie won…2 Pierre Curie…3 The Nobel Prize…readerextractive: spangenerative: decode(e.g. FiD, T5)“the Nobel Prize in Physics”closed-book: no retriever,the LM answers from its weightsextractive reader (BERT): new parameters are only S, E ∈ ℝ^Hbest span: 4.2 + 4.6 = 8.8 > null 1.0end E·Tstart S·T0.5····0.4-0.7-1.00.23.2-0.84.6·0.5····0.2-0.63.64.2-0.2-0.50.8·nullT0T1T2T3T4T5T6T7T8T9T10T11T12T_i ∈ ℝ^HBERT encoder: question and passage attend to each otherwindows of 384 tokens, stride 128, for long passages[CLS]whichprize…[SEP]CuriewontheNobelPrizeinPhysics[SEP]inputquestion (segment A)passage (segment B): answers come only from here
fig.Top: open-domain QA retrieves the top-k passages from millions, then a reader extracts or generates the answer. Bottom: the extractive reader. Each passage token gets a start logit S·T_i and an end logit E·T_i; the best span “Nobel Prize in Physics” scores 4.2 + 4.6 = 8.8, far above the no-answer score 1.0 at [CLS].
Pstart(i)=eS⋅Ti∑keS⋅Tk,Pend(j)=eE⋅Tj∑keE⋅Tk,ı^,ȷ^=arg⁡max⁡i≤j<i+Lmax⁡  S⋅Ti+E⋅TjP_{\text{start}}(i)=\frac{e^{S\cdot T_i}}{\sum_k e^{S\cdot T_k}},\qquad P_{\text{end}}(j)=\frac{e^{E\cdot T_j}}{\sum_k e^{E\cdot T_k}},\qquad \hat\imath,\hat\jmath=\arg\max_{i\le j<i+L_{\max}}\;S\cdot T_i+E\cdot T_j
TiT_i
final hidden state of token ii, in RH\mathbb{R}^H
S,  ES,\;E
learned start and end vectors (a linear layer with 2 outputs)
Lmax⁡L_{\max}
maximum answer length in tokens (30 in the BERT and Hugging Face SQuAD scripts)

Training maximises the log-likelihood of the gold start and the gold end, L=−log⁡Pstart(s∗)−log⁡Pend(e∗)\mathcal{L}=-\log P_{\text{start}}(s^*)-\log P_{\text{end}}(e^*) (the Hugging Face head averages the two). Start and end are independent softmaxes, so a span's log-probability is the sum of two logits, and the search is a double loop over i≤ji\le j.

Passages longer than the 512-token limit are cut into overlapping windows: 384 tokens with a stride of 128 in the original SQuAD script. Each window is read separately, and the best span over all windows wins.

Unanswerable questions (SQuAD 2.0) reuse [CLS] as the null span. Its score is snull=S⋅C+E⋅Cs_{\text{null}}=S\cdot C+E\cdot C, and the model answers only if the best span beats it: s^ı^,ȷ^>snull+τ\hat s_{\hat\imath,\hat\jmath}>s_{\text{null}}+\tau. The threshold τ\tau is tuned on the dev set for F1.

Open-domain and generative QA

Without a given passage, a retriever first ranks millions of passages. DrQA used TF-IDF over hashed bigrams. Dense Passage Retrieval (DPR, Karpukhin et al. 2020) uses two BERT encoders and scores EQ(q)⋅EP(p)E_Q(q)\cdot E_P(p) with maximum inner-product search over 21M 100-word Wikipedia passages. On Natural Questions, top-20 retrieval accuracy rose from 59.1% (BM25) to 78.4%.

A reader then extracts or generates the answer from the top kk. Fusion-in-Decoder (Izacard and Grave 2021) encodes each retrieved passage separately with T5 and lets the decoder attend to all of them. Generating from retrieved text is the subject of Retrieval-augmented generation (RAG). Closed-book QA drops the retriever: a model like T5 answers from its weights alone (Roberts, Raffel and Shazeer 2020).

EM=1[norm(a^)=norm(a)],F1=2PRP+R,P=∣a^∩a∣∣a^∣,  R=∣a^∩a∣∣a∣\mathrm{EM}=\mathbb{1}\big[\mathrm{norm}(\hat a)=\mathrm{norm}(a)\big],\qquad \mathrm{F1}=\frac{2PR}{P+R},\quad P=\frac{|\hat a\cap a|}{|\hat a|},\;R=\frac{|\hat a\cap a|}{|a|}
norm\mathrm{norm}
lower-case, strip punctuation, drop a / an / the, collapse whitespace
∣a^∩a∣|\hat a\cap a|
bag-of-tokens overlap between prediction and gold answer; take the max over all gold answers

Benchmarks

SQuAD 1.1
Rajpurkar, Zhang, Lopyrev, Liang 2016: 107,785 crowd-written questions on 536 Wikipedia articles; every answer is a span. Human test 82.3 EM / 91.2 F1; BERT-large single model 84.1 / 90.9 on dev.
SQuAD 2.0
Rajpurkar, Jia, Liang 2018: adds 53,775 unanswerable questions written to resemble answerable ones. Human F1 89.5; BERT-large 80.0 EM / 83.1 F1 on test.
Natural Questions
Kwiatkowski et al. 2019: real anonymised Google search queries answered from Wikipedia pages, 307,373 training examples.
TriviaQA
Joshi et al. 2017: about 95K trivia question–answer pairs with independently gathered evidence documents.
HotpotQA
Yang et al. 2018: multi-hop questions whose answer needs two Wikipedia paragraphs.
Q: Which prize did Curie win in 1903?gold: “the Nobel Prize in Physics”tokenP(start) = softmax(S·T)P(end) = softmax(E·T)[CLS]0.01MarieCuriewonthe0.27Nobel0.49Prize0.15inPhysics0.020.60in19030.01andthe0.05Nobel0.08Prize0.04inChemistry0.12in1911.0.01max over i ≤ j < i + L ofS·T_i + E·T_j190 candidate spans (L = 30)top spansscoreP1. Nobel Prize in P…8.800.2932. the Nobel Prize …8.200.1613. Nobel Prize7.400.072null = S·C + E·C at [CLS]= 1.00, P = 0.000answer iff 8.80 > 1.00 + τ (0.0)prediction:“Nobel Prize in Physics”EM 1 · F1 1.000after lower-casing and droppingpunctuation and a / an / theshaded = predicted span · green = gold-answer token · bars scaled to the largest probabilityspan score = start logit + end logit, i.e. P(span) = P_start(i)·P_end(j)
question
task
fig.A span reader with hand-set logits on the Curie passage. Default (“which prize”, L = 30): 190 candidate spans; the best is “Nobel Prize in Physics” at 8.8 with P = 0.293, EM 1, F1 1. Cap the length at 3 and it returns “Nobel Prize” (EM 0, F1 0.667); at 1, “Physics” (F1 0.4). The “whom married” question has no answer: the best span “Marie Curie” scores 5.1 against a null score of 6.0, so the reader abstains at τ = 0 but answers wrongly once τ ≤ −1.

Worked example: spans and their scores

spanstart + end logitP(span)EMF1
Nobel Prize in Physics4.2 + 4.6 = 8.80.29311.000
the Nobel Prize in Physics3.6 + 4.6 = 8.20.16111.000
Nobel Prize4.2 + 3.2 = 7.40.07200.667
Nobel Prize in Chemistry2.4 + 3.0 = 5.40.01000.750
Physics0.8 + 4.6 = 5.40.01000.400
null ([CLS])0.5 + 0.5 = 1.00.000——
Question “Which prize did Curie win in 1903?”, gold “the Nobel Prize in Physics”. P_start(Nobel) = 0.490, P_end(Physics) = 0.598.

The top two spans differ only by “the”, which normalisation removes, so both get EM 1. “Nobel Prize” shares 2 of the 4 gold tokens: P=2/2P=2/2, R=2/4R=2/4, F1 =0.667=0.667. The wrong prize, “Nobel Prize in Chemistry”, still matches 3 of 4 tokens: P=R=0.75P=R=0.75, F1 0.75. Token F1 gives partial credit even to a factually wrong answer.

python
1import collections, re, string
2from transformers import pipeline
3
4qa = pipeline("question-answering", model="deepset/roberta-base-squad2")
5ctx = "Marie Curie won the Nobel Prize in Physics in 1903 and the Nobel Prize in Chemistry in 1911."
6pred = qa(question="Which prize did Curie win in 1903?", context=ctx,
7 max_answer_len=30, handle_impossible_answer=True) # may return "" (no answer)
8
9def normalize(s):
10 s = "".join(ch for ch in s.lower() if ch not in set(string.punctuation))
11 return re.sub(r"\b(a|an|the)\b", " ", s).split()
12
13def f1(pred, gold):
14 p, g = normalize(pred), normalize(gold)
15 if not p or not g:
16 return float(p == g)
17 common = sum((collections.Counter(p) & collections.Counter(g)).values())
18 if common == 0:
19 return 0.0
20 P, R = common / len(p), common / len(g)
21 return 2 * P * R / (P + R)
22
23print(pred["answer"], pred["score"])
24print(f1("Nobel Prize", "the Nobel Prize in Physics")) # 0.667
25print(f1("the Nobel Prize in Chemistry", "the Nobel Prize in Physics")) # 0.75
A SQuAD 2.0 reader from the Hugging Face hub, and the official-style EM/F1 (the pipeline call was not run here; the f1 values were checked in plain Python)

Where it's used

Search

Answer boxes that highlight a span of a retrieved page.

Customer support and documentation

Answering from manuals and FAQs, where extractive spans can be checked against the source.

Reading-comprehension evaluation

SQuAD-style EM/F1 remains a standard probe of encoder models; see LLM evaluation and benchmarking for generative QA benchmarks.

Check yourself

A SQuAD 2.0 reader has a best span score of 5.1 and a null score of 6.0. With τ = 0, what does it output?

Check yourself

Gold answer: “the Nobel Prize in Physics”. Prediction: “Nobel Prize”. What are EM and F1?

Remember

  • ✓Extractive QA: start/end softmaxes over S⋅TiS\cdot T_i and E⋅TjE\cdot T_j; best span maximises their sum over i≤j<i+Lmax⁡i\le j<i+L_{\max}.
  • ✓SQuAD 2.0: answer only if the best span beats the [CLS] null score plus a tuned τ\tau.
  • ✓Open-domain QA = retriever (BM25 or DPR) + reader; the retriever's recall bounds the answer accuracy.
  • ✓Generative and closed-book QA write the answer; retrieval-backed generation is RAG.
  • ✓EM is all-or-nothing after normalisation; F1 is bag-of-tokens overlap and gives partial credit.
← back to Deep Learning roadmap