Definition
Hallucination is generated content that reads as fluent and confident but is not supported by the input the model was given (unfaithful) or by verifiable facts about the world (non-factual). Maynez, Narayan, Bohnet and McDonald ("On Faithfulness and Factuality in Abstractive Summarization", ACL 2020) split it into intrinsic hallucinations, which contradict the source, and extrinsic ones, which add information the source cannot confirm. Ji et al. (ACM Computing Surveys, 2023) and Huang et al. (2023) survey the field.
It matters wherever text is trusted without checking: a summary that changes a date, a RAG answer citing a passage that says the opposite, a legal brief with invented case law, code that imports a package that does not exist. The model's tone gives no signal. A wrong answer is generated by the same sampling process as a right one.
This article covers what hallucination is, why it happens and how systems reduce it. Measuring it is a separate job: see Hallucination and groundedness evaluation in this branch, and Hallucination detection and Factuality evaluation in the NLP module.
Two axes: source and world
Faithfulness asks whether each claim is supported by the context the model was given: the document to summarise, the retrieved passages, the conversation. Factuality asks whether the claim is true in the world. The two come apart. A faithful summary of an outdated source is non-factual. An extrinsic addition can be true and still a faithfulness failure, because the reader cannot check it against the source.
In practice: closed-domain tasks (summarisation, RAG, extraction) are judged mainly on faithfulness to the supplied context. Open-domain questions with no context are judged on factuality, and the model can rely only on what its weights store.
Why models hallucinate
A language model is trained to maximise the likelihood of the next token (see Language modeling). The objective rewards a plausible continuation; nothing in it rewards stopping to say "I don't know". For facts the model saw many times, the most plausible continuation is usually the right one. For facts it saw once or never (a minor person's birthday, a paper's exact title) it still produces a plausible-looking guess.
Kalai and Vempala ("Calibrated Language Models Must Hallucinate", STOC 2024) made this precise for arbitrary facts with no learnable pattern. Using a Good–Turing argument, they showed that a calibrated model must hallucinate on such facts at a rate set by how many of them appear exactly once in the training data. Simplified:
- monofact rate: share of training facts (of this arbitrary kind) seen exactly once, the Good–Turing estimate of the unseen mass
- how far the model's probabilities are from calibrated; the bound has further small terms omitted here
The bound does not apply to facts with structure the model can compute, like arithmetic, or to facts repeated often in the data. It explains why pretraining alone cannot remove hallucination on rare facts, and why the fix must involve abstention or outside sources.
Post-training and evaluation add an incentive to guess. Kalai, Nachum, Vempala and Zhang ("Why Language Models Hallucinate", 2025) point out that most benchmarks grade answers right or wrong, with 0 for "I don't know". Under that grading, guessing never scores worse than abstaining. If a wrong answer costs points instead, answering with confidence has expected score:
- probability the answer is correct (the model's confidence, if calibrated)
- penalty for a wrong answer; abstaining scores 0
- the confidence threshold above which answering pays: 0 for , 0.5 for , 0.75 for
Three more causes enter later. Gekhman et al. (EMNLP 2024) found that supervised fine-tuning on facts the base model did not know is learned slowly and makes the model more prone to hallucinate, so SFT data should match what the model already knows (see Supervised fine-tuning (SFT)). At decoding time, sampling can pick a low-probability wrong token, and later tokens then stay consistent with it. Zhang et al. (2023) call this snowballing: the model commits to an early error and justifies it. Long or cluttered contexts also hurt: Liu et al. ("Lost in the Middle", TACL 2024) found models use information in the middle of a long context less reliably than information at the start or end.
Checkable results
- TruthfulQA
- Lin, Hilton and Evans, ACL 2022. 817 questions in 38 categories built around common misconceptions. The best model tested (GPT-3 175B) was truthful on 58% of questions; humans on 94%. Larger models in the same family were less truthful.
- Snowballing
- Zhang et al., 2023. On questions where ChatGPT and GPT-4 gave wrong justifications, the same models identified 67% and 87% of those mistakes when asked about them separately.
- Mata v. Avianca
- S.D.N.Y., June 2023. Lawyers filed a brief citing six nonexistent court decisions generated by ChatGPT; the court fined them $5,000.
- Chain-of-Verification
- Dhuliawala et al. (Meta), 2023. Draft an answer, plan verification questions, answer them independently of the draft, then revise.
Abstention: trading coverage for accuracy
The score curve peaks near only when stated confidence is calibrated. Overconfident scores push the peak to a higher threshold, and you cannot know where without labelled data. So an abstention threshold needs a calibration set from your own traffic. Raising the rare-fact share to 0.6 makes the trade-off harsher: at the model answers only 38.4% of questions, with a 4.4% hallucination rate.
Worked example: when to answer
| p | k = 0 | k = 1 | k = 3 |
|---|---|---|---|
| 0.95 | 0.95 | 0.90 | 0.80 |
| 0.80 | 0.80 | 0.60 | 0.20 |
| 0.60 | 0.60 | 0.20 | −0.60 (abstain) |
| 0.40 | 0.40 | −0.20 (abstain) | −1.40 (abstain) |
| 0.20 | 0.20 | −0.60 (abstain) | −2.20 (abstain) |
| policy total | 2.95 (answer all) | 1.70 (answer 3) | 1.00 (answer 2) |
| guess all | 2.95 | 0.90 | −3.20 |
| expected wrong answers (policy) | 2.05 | 0.65 | 0.25 |
With the policy and guessing are identical and produce 2.05 expected wrong answers out of 5. With abstaining on three questions cuts expected errors to 0.25 and lifts the score from −3.20 to 1.00. The model did not get more accurate. It got better at not answering.
Mitigation, layer by layer
What production systems stack
- 1Ground the answer. Retrieve sources and instruct the model to answer only from them (see RAG architecture). This turns open-domain factuality into closed-domain faithfulness, which is easier to check.
- 2Make "I don't know" a valid output. Say so in the system prompt, include abstaining examples in fine-tuning data, and score errors worse than abstentions in your evaluations.
- 3Estimate confidence. Token log-probabilities, or agreement among several sampled answers (the idea behind Self-consistency and SelfCheckGPT). Abstain or escalate below a threshold tuned on labelled traffic.
- 4Decode conservatively for facts. Use low temperature or greedy decoding for extraction and QA (see Temperature sampling). Use Structured output (JSON mode) so fields can be validated.
- 5Verify before showing. Require citations, check that each cited passage entails its claim, run a chain-of-verification pass, or call tools (search, a calculator, a database) for facts the model should not recall from memory.
- 6Show provenance in the UI. Link each claim to its source so users can check it, and log flagged outputs for review.
1from collections import Counter23def answer_or_abstain(sample, question, n=5, k=3):4 """Return the majority answer only if agreement beats t* = k/(k+1)."""5 answers = [sample(question).strip().lower() for _ in range(n)]6 top, count = Counter(answers).most_common(1)[0]7 agreement = count / n # crude confidence estimate8 t_star = k / (k + 1) # wrong answers cost k, abstain costs 09 return top if agreement > t_star else "I don't know"1011# stub samplers standing in for an LLM call12from itertools import cycle13known = cycle(["1889", "1889", "1889", "1889", "1887"])14rare = cycle(["1912", "1915", "1912", "1908", "1921"])15print(answer_or_abstain(lambda q: next(known), "When was the Eiffel Tower completed?")) # 1889 (0.8 > 0.75)16print(answer_or_abstain(lambda q: next(rare), "When was this minor poet born?")) # I don't know (0.4)Check yourself
A benchmark gives 1 point per correct answer and 0 for both wrong answers and "I don't know". What does a model maximising this score do on a question it is 10% sure about?
Check yourself
The source says a drug was approved in 2019. The summary says it was approved in 2019 by the FDA's fast-track programme, which the source never mentions. Which label fits the added detail?
Remember
- ✓Intrinsic hallucinations contradict the source; extrinsic ones add unsupported content (Maynez et al., 2020). Faithfulness (vs source) and factuality (vs world) are separate checks.
- ✓Likelihood training always yields a guess; for arbitrary facts seen once, calibrated models must hallucinate at roughly the monofact rate (Kalai and Vempala, 2024).
- ✓Accuracy-only grading rewards guessing; with penalty per error, answer only when confidence exceeds .
- ✓SFT on facts the model does not know, sampling, error snowballing and long cluttered contexts all add hallucinations.
- ✓Stack mitigations: grounding, abstention, confidence estimation, conservative decoding, claim-level verification, visible provenance. Measure them with groundedness evaluation.