Search and question answering

An answer engine retrieves, reranks, writes an answer that cites numbered sources, then checks each claim against its citations and abstains when evidence is weak; accuracy is capped by recall@k, and trust by citation precision.

8 min read839 wordscore0 viewsupdated 2026-10-21

Definition

Generative search and question answering is the application of retrieval plus an LLM to answer a user's question directly, in prose, with citations to the documents it used, instead of returning only a ranked list of links. The pattern was set out in WebGPT (Nakano et al., OpenAI, 2021), which fine-tuned GPT-3 to browse the web and collect references, and GopherCite (Menick et al., DeepMind, 2022), which backed answers with verbatim quotes and learned to decline. Web answer engines and enterprise "chat with your documents" tools are the two common deployments.

A search box gets three kinds of query: navigational ("github login"), where a link is the answer; exploratory ("hiking tents"), where a result list is better; and informational ("when did Webb launch?"), where a sentence is better. Answer engines target the third kind and keep the result list for the other two. The model internals and the extractive span models (SQuAD, exact match, token F1) are covered in Question answering; the generic pipeline is RAG architecture. This article is about the product: what the user sees, how citations are produced and checked, and when not to answer.

The pipeline

querywhen did webblaunch?rewriteJames Webb SpaceTelescope launchretrieveBM25 + densetop 50rerankcross-encoderkeep top 4generateLLM reads [1]–[4]cites each claimverifyNLI: claim vsits citationssources in the prompt (rerank score)[1] nasa.gov0.91Webb launched 25 Dec 2021 on anAriane 5 rocket[2] esa.int0.87...lifted off from Europe'sSpaceport in Kourou, French Guiana[3] encyclopedia0.80Webb reached its orbit aroundL2 on 24 January 2022[4] blog0.42Hubble launched in April 1990aboard the shuttle Discoveryanswer with inline citationsWebb launched on 25 December 2021on an Ariane 5 [1]✓ [1] entailsfrom Kourou, French Guiana [2].✓ [2] entailsIt reached L2 in January 2022 [3].✓ [3] entailscitation recall 3/3 · precision 3/3source [4] was retrieved but not citedgatetop score ≥ τ ?every claimsupported ?yesnoshow answer + clickable sources [1][2][3]abstain: "not found in sources" + plain result list
fig.One query through an answer engine. The rewritten query goes to hybrid retrieval (50 candidates), a cross-encoder keeps 4, and the LLM answers with a citation after each claim. A verifier checks every sentence against the sources it cites. Here all three claims are entailed (citation recall 3/3, precision 3/3), and the low-scoring Hubble passage [4] is retrieved but correctly left uncited. If the top score were below τ or a claim unsupported, the gate would abstain.
  1. 1Rewrite the query: expand abbreviations, resolve follow-ups ("and how much did it cost?") into a standalone question using the chat history, and optionally split multi-part questions. See Query rewriting and HyDE.
  2. 2Retrieve a wide candidate set (tens to hundreds) with lexical plus dense search, so exact names and paraphrases both match. See Hybrid search (BM25 + dense).
  3. 3Rerank with a cross-encoder and keep a handful of passages; the reranker score doubles as the retrieval-confidence signal for abstention. See Reranking.
  4. 4Generate from numbered passages with an instruction to cite every sentence and to say when the sources do not answer.
  5. 5Verify each sentence against its cited passages with an entailment (NLI) model or an LLM judge; drop or flag unsupported sentences, or abstain.
  6. 6Render the answer with clickable citations next to the plain result list, so the user can check the source and fall back to search.

Accuracy is capped by retrieval

P(correct)  ≈  R@k⏟gold passage retrieved⋅ actx(k)  +  (1−R@k)⋅amemP(\text{correct})\;\approx\;\underbrace{R@k}_{\text{gold passage retrieved}}\cdot\,a_{\text{ctx}}(k)\;+\;\big(1-R@k\big)\cdot a_{\text{mem}}
R@kR@k
recall@k: share of answerable questions whose gold passage is in the top kk
actx(k)a_{\text{ctx}}(k)
reader accuracy when the evidence is in the prompt; falls slowly with k because of distractors and position effects
amema_{\text{mem}}
accuracy from the model's parametric memory when the evidence is missing; for fresh or private data it is near 0

Raising kk raises R@kR@k but lowers actxa_\text{ctx}. Liu et al. ("Lost in the Middle", TACL 2024) placed the answer-bearing document at different positions among 20 retrieved documents and found a U-shaped curve: accuracy was highest when it came first or last and lowest in the middle. So more passages is not free. The usual compromise is a wide first stage, a reranker, and a short final context.

Reference points

WebGPT
Nakano et al., 2021. On ELI5 questions, the best model's answers were preferred to its human demonstrators' 56% of the time, and to the top-voted Reddit answer 69% of the time
Verifiability audit
Liu, Zhang and Liang (Findings of EMNLP 2023) audited four generative search engines (Bing Chat, NeevaAI, perplexity.ai, YouChat): on average 51.5% of sentences were fully supported by their citations and 74.5% of citations supported their sentence
ALCE
Gao, Yen, Yu and Chen (EMNLP 2023): a benchmark for answering with citations, scoring citation recall and precision with an NLI model
DPR
Karpukhin et al. (EMNLP 2020): on Natural Questions, top-20 retrieval accuracy 78.4% for dense passage retrieval vs 59.1% for BM25
Natural Questions
Kwiatkowski et al. (TACL 2019): real anonymised Google queries with answers marked in Wikipedia pages; 307,373 training examples

Citations you can check

CitRec=1∣S∣∑s∈S1[Cs⊨s],CitPrec=#{c:  c needed for a supported s}#{citations}\text{CitRec}=\frac{1}{|S|}\sum_{s\in S}\mathbb 1\big[C_s\models s\big],\qquad \text{CitPrec}=\frac{\#\{c:\;c\text{ needed for a supported }s\}}{\#\{\text{citations}\}}
SS
the answer's statements (sentences)
Cs⊨sC_s\models s
the passages cited by s jointly entail it, judged by an NLI model; an uncited sentence scores 0
c neededc\text{ needed}
c entails s alone, or removing it breaks the support (a simplification of ALCE's rule)

Citation recall asks whether each claim is backed; citation precision asks whether each citation earns its place. A system can get high precision by citing little and saying little, so report both, alongside answer correctness. The claim-level checks themselves are covered in Hallucination and groundedness evaluation.

Worked example: scoring one answer

StatementCitesNLI verdictSupportedPrecise citations
s1[1] [3][1] entails alone; [3] entails alone12 of 2
s2[2][2] does not entail00 of 1
s3[1] [4]neither alone; together they entail12 of 2 (each needed)
s4noneno citation0none
Four sentences, five citations. CitRec = 2/4 = 0.50; CitPrec = 4/5 = 0.80.

The fix differs by row. Statement s2 cited the wrong passage, or the claim came from memory. Statement s4 has no citation at all: either the generator added unsupported filler or the prompt did not insist on citing. A production verifier would drop s2 and s4 or regenerate them, which costs one NLI call per statement–citation pair.

When not to answer

cov(τ)=∣{i:ci≥τ}∣N,risk(τ)=#wrong among answered#answered,U(τ)=#correct−λ #wrongN\text{cov}(\tau)=\frac{|\{i:c_i\ge\tau\}|}{N},\qquad \text{risk}(\tau)=\frac{\#\text{wrong among answered}}{\#\text{answered}},\qquad U(\tau)=\frac{\#\text{correct}-\lambda\,\#\text{wrong}}{N}
cic_i
confidence for question i: top reranker score, verifier score, or the model's own calibrated estimate
τ\tau
abstention threshold; below it the engine shows results without an answer
λ\lambda
cost of a wrong answer relative to the value of a right one; abstaining scores 0

This is selective prediction (Geifman and El-Yaniv, 2017): trade coverage for lower risk along a risk–coverage curve. Kamath, Jia and Liang (ACL 2020) studied it for QA under domain shift and trained a separate calibrator to decide when to abstain. Questions the corpus cannot answer are the main reason to abstain; a model that always answers will make something up for them.

retrieve k passages, always answer0.00.51.015101520k (passages in the prompt)recall@8 = 0.879accuracy 0.589abstain below confidence τ (this k)0.000.350.700.00.51.0coverage (share answered)risk (wrong / answered) = 0.301200 questions (35 not answerable from the corpus, ringed) · k = 8, τ = 0.50fill: green = likely correct, red = likely wrong, blank = abstainedanswered 137/200 (coverage 0.685) · expected correct 95.8, wrong 41.2accuracy on answered 0.699 · unanswerable abstained 26/35utility = (correct − λ·wrong)/200 = 0.273 (λ = 1; always answering: 0.178)best τ for this k and λ: 0.53 → utility 0.286reader: 0.90 if gold passage is first/last, −0.20 in the middle, −0.008 per passage; 0.25 from memory
cost of a wrong answer λ
fig.200 synthetic questions, 35 of them not answerable from the corpus. Reader: 0.90 accuracy with the gold passage first or last, 0.20 lower in the middle, 0.008 lower per passage, 0.25 from memory. With no abstention, accuracy is 0.356 at k = 1, peaks at 0.589 at k = 8 (recall@8 = 0.879) and falls to 0.547 at k = 20 even though recall rises to 0.909. Defaults (k = 8, τ = 0.50, λ = 1): coverage 0.685, accuracy on answered 0.699, 26 of 35 unanswerable questions abstained, utility 0.273 vs 0.178 for always answering; the best τ is 0.53 (0.286). At λ = 4 the best τ is 0.79: coverage 0.095, accuracy 0.835, utility 0.017.

The viz shows the two levers separately. The left curve is the retrieval cap from the first equation: past k=8k=8 the extra recall is smaller than the loss from distractors and middle positions. The right curve is selective answering: each step left along it drops low-confidence questions and lowers risk. How far left to go depends only on λ\lambda, which is a product decision. A medical or legal tool might set λ=4\lambda=4; a casual web assistant might set λ=1\lambda=1.

In code

python
1import re
2
3def build_prompt(question, passages):
4 """Numbered sources + instructions to cite and to abstain."""
5 src = "\n".join(f"[{i}] ({p['url']}) {p['text']}" for i, p in enumerate(passages, 1))
6 return [
7 {"role": "system", "content": (
8 "Answer only from the sources. Cite every sentence with [n]. "
9 "If the sources do not contain the answer, reply exactly: NOT_FOUND")},
10 {"role": "user", "content": f"Sources:\n{src}\n\nQuestion: {question}"},
11 ]
12
13def check_citations(answer, n_sources):
14 """Every sentence needs >= 1 citation, and every citation must exist."""
15 sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer.strip()) if s]
16 report = []
17 for s in sentences:
18 ids = [int(x) for x in re.findall(r"\[(\d+)\]", s)]
19 ok = bool(ids) and all(1 <= i <= n_sources for i in ids)
20 report.append((s, ids, ok))
21 return report
22
23passages = [
24 {"url": "nasa.gov", "text": "Webb launched on 25 Dec 2021 on an Ariane 5.", "score": 0.91},
25 {"url": "esa.int", "text": "It lifted off from Kourou, French Guiana.", "score": 0.87},
26]
27TAU = 0.5
28if max(p["score"] for p in passages) < TAU:
29 print("abstain before calling the model")
30msgs = build_prompt("When did Webb launch?", passages)
31answer = "Webb launched on 25 December 2021 [1]. It launched from Kourou [2]. It cost $10B [3]."
32for s, ids, ok in check_citations(answer, len(passages)):
33 print("OK " if ok else "BAD", ids, s)
34# OK [1] Webb launched on 25 December 2021 [1].
35# OK [2] It launched from Kourou [2].
36# BAD [3] It cost $10B [3].
Prompt with numbered sources, a retrieval-confidence gate and a structural citation check (runs as is; the LLM call itself is omitted and the answer is hard-coded). The NLI entailment check would be a second pass over the OK sentences.

Use when

  • +One readable answer instead of ten links for factual questions.
  • +Fresh and private knowledge without retraining: update the index, not the model.
  • +Citations make every claim checkable, and abstention makes failures visible.

Avoid when

  • −Each answer adds an LLM call (and verifier calls) to the latency and cost of a search.
  • −Errors read as confident prose; a wrong answer can do more harm than a bad link.
  • −Answering on the page sends less traffic to the cited sources, which publishers notice.

Check yourself

Recall@20 is 0.91, but answer accuracy with k = 20 passages is lower than with k = 8. What is the most likely cause?

Check yourself

An answer has 4 sentences. Two are fully supported by their citations, one cites a passage that does not support it, and one has no citation. What is its citation recall?

Remember

  • ✓Answer engines run rewrite → hybrid retrieve → rerank → generate with numbered sources → verify → render or abstain.
  • ✓Accuracy ≈ R@k⋅actx(k)+(1−R@k) amemR@k\cdot a_\text{ctx}(k)+(1-R@k)\,a_\text{mem}: retrieval caps it, and too many passages lower actxa_\text{ctx}.
  • ✓Score citations with recall (is each claim backed?) and precision (is each citation needed?), using NLI; uncited sentences count as unsupported.
  • ✓A 2023 audit of four generative search engines: 51.5% of sentences fully supported, 74.5% of citations supportive.
  • ✓Abstain below a confidence threshold chosen from the risk–coverage curve and the cost λ of a wrong answer; unanswerable questions are why.
← back to Generative AI roadmap