Definition
Generative search and question answering is the application of retrieval plus an LLM to answer a user's question directly, in prose, with citations to the documents it used, instead of returning only a ranked list of links. The pattern was set out in WebGPT (Nakano et al., OpenAI, 2021), which fine-tuned GPT-3 to browse the web and collect references, and GopherCite (Menick et al., DeepMind, 2022), which backed answers with verbatim quotes and learned to decline. Web answer engines and enterprise "chat with your documents" tools are the two common deployments.
A search box gets three kinds of query: navigational ("github login"), where a link is the answer; exploratory ("hiking tents"), where a result list is better; and informational ("when did Webb launch?"), where a sentence is better. Answer engines target the third kind and keep the result list for the other two. The model internals and the extractive span models (SQuAD, exact match, token F1) are covered in Question answering; the generic pipeline is RAG architecture. This article is about the product: what the user sees, how citations are produced and checked, and when not to answer.
The pipeline
- 1Rewrite the query: expand abbreviations, resolve follow-ups ("and how much did it cost?") into a standalone question using the chat history, and optionally split multi-part questions. See Query rewriting and HyDE.
- 2Retrieve a wide candidate set (tens to hundreds) with lexical plus dense search, so exact names and paraphrases both match. See Hybrid search (BM25 + dense).
- 3Rerank with a cross-encoder and keep a handful of passages; the reranker score doubles as the retrieval-confidence signal for abstention. See Reranking.
- 4Generate from numbered passages with an instruction to cite every sentence and to say when the sources do not answer.
- 5Verify each sentence against its cited passages with an entailment (NLI) model or an LLM judge; drop or flag unsupported sentences, or abstain.
- 6Render the answer with clickable citations next to the plain result list, so the user can check the source and fall back to search.
Accuracy is capped by retrieval
- recall@k: share of answerable questions whose gold passage is in the top
- reader accuracy when the evidence is in the prompt; falls slowly with k because of distractors and position effects
- accuracy from the model's parametric memory when the evidence is missing; for fresh or private data it is near 0
Raising raises but lowers . Liu et al. ("Lost in the Middle", TACL 2024) placed the answer-bearing document at different positions among 20 retrieved documents and found a U-shaped curve: accuracy was highest when it came first or last and lowest in the middle. So more passages is not free. The usual compromise is a wide first stage, a reranker, and a short final context.
Reference points
- WebGPT
- Nakano et al., 2021. On ELI5 questions, the best model's answers were preferred to its human demonstrators' 56% of the time, and to the top-voted Reddit answer 69% of the time
- Verifiability audit
- Liu, Zhang and Liang (Findings of EMNLP 2023) audited four generative search engines (Bing Chat, NeevaAI, perplexity.ai, YouChat): on average 51.5% of sentences were fully supported by their citations and 74.5% of citations supported their sentence
- ALCE
- Gao, Yen, Yu and Chen (EMNLP 2023): a benchmark for answering with citations, scoring citation recall and precision with an NLI model
- DPR
- Karpukhin et al. (EMNLP 2020): on Natural Questions, top-20 retrieval accuracy 78.4% for dense passage retrieval vs 59.1% for BM25
- Natural Questions
- Kwiatkowski et al. (TACL 2019): real anonymised Google queries with answers marked in Wikipedia pages; 307,373 training examples
Citations you can check
- the answer's statements (sentences)
- the passages cited by s jointly entail it, judged by an NLI model; an uncited sentence scores 0
- c entails s alone, or removing it breaks the support (a simplification of ALCE's rule)
Citation recall asks whether each claim is backed; citation precision asks whether each citation earns its place. A system can get high precision by citing little and saying little, so report both, alongside answer correctness. The claim-level checks themselves are covered in Hallucination and groundedness evaluation.
Worked example: scoring one answer
| Statement | Cites | NLI verdict | Supported | Precise citations |
|---|---|---|---|---|
| s1 | [1] [3] | [1] entails alone; [3] entails alone | 1 | 2 of 2 |
| s2 | [2] | [2] does not entail | 0 | 0 of 1 |
| s3 | [1] [4] | neither alone; together they entail | 1 | 2 of 2 (each needed) |
| s4 | none | no citation | 0 | none |
The fix differs by row. Statement s2 cited the wrong passage, or the claim came from memory. Statement s4 has no citation at all: either the generator added unsupported filler or the prompt did not insist on citing. A production verifier would drop s2 and s4 or regenerate them, which costs one NLI call per statement–citation pair.
When not to answer
- confidence for question i: top reranker score, verifier score, or the model's own calibrated estimate
- abstention threshold; below it the engine shows results without an answer
- cost of a wrong answer relative to the value of a right one; abstaining scores 0
This is selective prediction (Geifman and El-Yaniv, 2017): trade coverage for lower risk along a risk–coverage curve. Kamath, Jia and Liang (ACL 2020) studied it for QA under domain shift and trained a separate calibrator to decide when to abstain. Questions the corpus cannot answer are the main reason to abstain; a model that always answers will make something up for them.
The viz shows the two levers separately. The left curve is the retrieval cap from the first equation: past the extra recall is smaller than the loss from distractors and middle positions. The right curve is selective answering: each step left along it drops low-confidence questions and lowers risk. How far left to go depends only on , which is a product decision. A medical or legal tool might set ; a casual web assistant might set .
In code
1import re23def build_prompt(question, passages):4 """Numbered sources + instructions to cite and to abstain."""5 src = "\n".join(f"[{i}] ({p['url']}) {p['text']}" for i, p in enumerate(passages, 1))6 return [7 {"role": "system", "content": (8 "Answer only from the sources. Cite every sentence with [n]. "9 "If the sources do not contain the answer, reply exactly: NOT_FOUND")},10 {"role": "user", "content": f"Sources:\n{src}\n\nQuestion: {question}"},11 ]1213def check_citations(answer, n_sources):14 """Every sentence needs >= 1 citation, and every citation must exist."""15 sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer.strip()) if s]16 report = []17 for s in sentences:18 ids = [int(x) for x in re.findall(r"\[(\d+)\]", s)]19 ok = bool(ids) and all(1 <= i <= n_sources for i in ids)20 report.append((s, ids, ok))21 return report2223passages = [24 {"url": "nasa.gov", "text": "Webb launched on 25 Dec 2021 on an Ariane 5.", "score": 0.91},25 {"url": "esa.int", "text": "It lifted off from Kourou, French Guiana.", "score": 0.87},26]27TAU = 0.528if max(p["score"] for p in passages) < TAU:29 print("abstain before calling the model")30msgs = build_prompt("When did Webb launch?", passages)31answer = "Webb launched on 25 December 2021 [1]. It launched from Kourou [2]. It cost $10B [3]."32for s, ids, ok in check_citations(answer, len(passages)):33 print("OK " if ok else "BAD", ids, s)34# OK [1] Webb launched on 25 December 2021 [1].35# OK [2] It launched from Kourou [2].36# BAD [3] It cost $10B [3].Use when
- +One readable answer instead of ten links for factual questions.
- +Fresh and private knowledge without retraining: update the index, not the model.
- +Citations make every claim checkable, and abstention makes failures visible.
Avoid when
- −Each answer adds an LLM call (and verifier calls) to the latency and cost of a search.
- −Errors read as confident prose; a wrong answer can do more harm than a bad link.
- −Answering on the page sends less traffic to the cited sources, which publishers notice.
Check yourself
Recall@20 is 0.91, but answer accuracy with k = 20 passages is lower than with k = 8. What is the most likely cause?
Check yourself
An answer has 4 sentences. Two are fully supported by their citations, one cites a passage that does not support it, and one has no citation. What is its citation recall?
Remember
- ✓Answer engines run rewrite → hybrid retrieve → rerank → generate with numbered sources → verify → render or abstain.
- ✓Accuracy ≈ : retrieval caps it, and too many passages lower .
- ✓Score citations with recall (is each claim backed?) and precision (is each citation needed?), using NLI; uncited sentences count as unsupported.
- ✓A 2023 audit of four generative search engines: 51.5% of sentences fully supported, 74.5% of citations supportive.
- ✓Abstain below a confidence threshold chosen from the risk–coverage curve and the cost λ of a wrong answer; unanswerable questions are why.