Definition
Question answering (QA) maps a natural-language question, and optionally a passage or a corpus, to an answer. Extractive QA returns a span of a given passage (SQuAD, Rajpurkar et al. 2016). Open-domain QA first retrieves passages from a large corpus such as Wikipedia, then reads them (DrQA, Chen et al. 2017). Generative QA writes the answer with a decoder, with or without retrieved evidence.
The task
Input: a question such as “Which prize did Curie win in 1903?” and the passage “Marie Curie won the Nobel Prize in Physics in 1903 and the Nobel Prize in Chemistry in 1911.” Output: “the Nobel Prize in Physics”.
The hard part is matching the question's meaning to the right place in the text. The passage holds two prizes and two years, so word overlap alone cannot decide. The model has to tie “1903” to the right prize. Some questions have no answer in the passage, and the model must notice that instead of guessing.
Earlier systems were pipelines: question-type classification, named-entity tagging, passage retrieval and hand-written answer-extraction rules. IBM Watson won Jeopardy! in 2011 with a large ensemble of such components. Neural readers replaced most of that with one model trained on question–passage–answer triples.
Extractive QA: start and end logits
BERT (Devlin et al. 2019) frames span extraction as two token classifications. The input is [CLS] question [SEP] passage [SEP]. The only new parameters are a start vector and an end vector , each in .
- final hidden state of token , in
- learned start and end vectors (a linear layer with 2 outputs)
- maximum answer length in tokens (30 in the BERT and Hugging Face SQuAD scripts)
Training maximises the log-likelihood of the gold start and the gold end, (the Hugging Face head averages the two). Start and end are independent softmaxes, so a span's log-probability is the sum of two logits, and the search is a double loop over .
Passages longer than the 512-token limit are cut into overlapping windows: 384 tokens with a stride of 128 in the original SQuAD script. Each window is read separately, and the best span over all windows wins.
Unanswerable questions (SQuAD 2.0) reuse [CLS] as the null span. Its score is , and the model answers only if the best span beats it: . The threshold is tuned on the dev set for F1.
Open-domain and generative QA
Without a given passage, a retriever first ranks millions of passages. DrQA used TF-IDF over hashed bigrams. Dense Passage Retrieval (DPR, Karpukhin et al. 2020) uses two BERT encoders and scores with maximum inner-product search over 21M 100-word Wikipedia passages. On Natural Questions, top-20 retrieval accuracy rose from 59.1% (BM25) to 78.4%.
A reader then extracts or generates the answer from the top . Fusion-in-Decoder (Izacard and Grave 2021) encodes each retrieved passage separately with T5 and lets the decoder attend to all of them. Generating from retrieved text is the subject of Retrieval-augmented generation (RAG). Closed-book QA drops the retriever: a model like T5 answers from its weights alone (Roberts, Raffel and Shazeer 2020).
- lower-case, strip punctuation, drop a / an / the, collapse whitespace
- bag-of-tokens overlap between prediction and gold answer; take the max over all gold answers
Benchmarks
- SQuAD 1.1
- Rajpurkar, Zhang, Lopyrev, Liang 2016: 107,785 crowd-written questions on 536 Wikipedia articles; every answer is a span. Human test 82.3 EM / 91.2 F1; BERT-large single model 84.1 / 90.9 on dev.
- SQuAD 2.0
- Rajpurkar, Jia, Liang 2018: adds 53,775 unanswerable questions written to resemble answerable ones. Human F1 89.5; BERT-large 80.0 EM / 83.1 F1 on test.
- Natural Questions
- Kwiatkowski et al. 2019: real anonymised Google search queries answered from Wikipedia pages, 307,373 training examples.
- TriviaQA
- Joshi et al. 2017: about 95K trivia question–answer pairs with independently gathered evidence documents.
- HotpotQA
- Yang et al. 2018: multi-hop questions whose answer needs two Wikipedia paragraphs.
Worked example: spans and their scores
| span | start + end logit | P(span) | EM | F1 |
|---|---|---|---|---|
| Nobel Prize in Physics | 4.2 + 4.6 = 8.8 | 0.293 | 1 | 1.000 |
| the Nobel Prize in Physics | 3.6 + 4.6 = 8.2 | 0.161 | 1 | 1.000 |
| Nobel Prize | 4.2 + 3.2 = 7.4 | 0.072 | 0 | 0.667 |
| Nobel Prize in Chemistry | 2.4 + 3.0 = 5.4 | 0.010 | 0 | 0.750 |
| Physics | 0.8 + 4.6 = 5.4 | 0.010 | 0 | 0.400 |
null ([CLS]) | 0.5 + 0.5 = 1.0 | 0.000 | — | — |
The top two spans differ only by “the”, which normalisation removes, so both get EM 1. “Nobel Prize” shares 2 of the 4 gold tokens: , , F1 . The wrong prize, “Nobel Prize in Chemistry”, still matches 3 of 4 tokens: , F1 0.75. Token F1 gives partial credit even to a factually wrong answer.
1import collections, re, string2from transformers import pipeline34qa = pipeline("question-answering", model="deepset/roberta-base-squad2")5ctx = "Marie Curie won the Nobel Prize in Physics in 1903 and the Nobel Prize in Chemistry in 1911."6pred = qa(question="Which prize did Curie win in 1903?", context=ctx,7 max_answer_len=30, handle_impossible_answer=True) # may return "" (no answer)89def normalize(s):10 s = "".join(ch for ch in s.lower() if ch not in set(string.punctuation))11 return re.sub(r"\b(a|an|the)\b", " ", s).split()1213def f1(pred, gold):14 p, g = normalize(pred), normalize(gold)15 if not p or not g:16 return float(p == g)17 common = sum((collections.Counter(p) & collections.Counter(g)).values())18 if common == 0:19 return 0.020 P, R = common / len(p), common / len(g)21 return 2 * P * R / (P + R)2223print(pred["answer"], pred["score"])24print(f1("Nobel Prize", "the Nobel Prize in Physics")) # 0.66725print(f1("the Nobel Prize in Chemistry", "the Nobel Prize in Physics")) # 0.75Where it's used
Search
Answer boxes that highlight a span of a retrieved page.
Customer support and documentation
Answering from manuals and FAQs, where extractive spans can be checked against the source.
Reading-comprehension evaluation
SQuAD-style EM/F1 remains a standard probe of encoder models; see LLM evaluation and benchmarking for generative QA benchmarks.
Check yourself
A SQuAD 2.0 reader has a best span score of 5.1 and a null score of 6.0. With τ = 0, what does it output?
Check yourself
Gold answer: “the Nobel Prize in Physics”. Prediction: “Nobel Prize”. What are EM and F1?
Remember
- ✓Extractive QA: start/end softmaxes over and ; best span maximises their sum over .
- ✓SQuAD 2.0: answer only if the best span beats the
[CLS]null score plus a tuned . - ✓Open-domain QA = retriever (BM25 or DPR) + reader; the retriever's recall bounds the answer accuracy.
- ✓Generative and closed-book QA write the answer; retrieval-backed generation is RAG.
- ✓EM is all-or-nothing after normalisation; F1 is bag-of-tokens overlap and gives partial credit.