Hallucination

A hallucination is fluent output not supported by the source (intrinsic: contradicts it; extrinsic: adds to it) or by the world. Likelihood training and accuracy-only grading reward guessing; retrieval, abstention and verification reduce it.

8 min read1080 wordscore0 viewsupdated 2026-10-21

Definition

Hallucination is generated content that reads as fluent and confident but is not supported by the input the model was given (unfaithful) or by verifiable facts about the world (non-factual). Maynez, Narayan, Bohnet and McDonald ("On Faithfulness and Factuality in Abstractive Summarization", ACL 2020) split it into intrinsic hallucinations, which contradict the source, and extrinsic ones, which add information the source cannot confirm. Ji et al. (ACM Computing Surveys, 2023) and Huang et al. (2023) survey the field.

It matters wherever text is trusted without checking: a summary that changes a date, a RAG answer citing a passage that says the opposite, a legal brief with invented case law, code that imports a package that does not exist. The model's tone gives no signal. A wrong answer is generated by the same sampling process as a right one.

This article covers what hallucination is, why it happens and how systems reduce it. Measuring it is a separate job: see Hallucination and groundedness evaluation in this branch, and Hallucination detection and Factuality evaluation in the NLP module.

Two axes: source and world

Faithfulness asks whether each claim is supported by the context the model was given: the document to summarise, the retrieved passages, the conversation. Factuality asks whether the claim is true in the world. The two come apart. A faithful summary of an outdated source is non-factual. An extrinsic addition can be true and still a faithfulness failure, because the reader cannot check it against the source.

faithfulness: is each output claim supported by the source?source: “The Eiffel Tower was completed in 1889 and is 330 m tall.”summarisemodel summary: “The tower, finished in 1899, is 330 m tall and is repainted every 7 years.”“finished in 1899”intrinsic: contradicts source(source says 1889)“is 330 m tall”faithful: entailed by source(no hallucination)“repainted every 7 years”extrinsic: not in sourcetrue or not, unsupportedwhere it comes from (cause) and what helps (mitigation)datarare, wrong orstale factsretrieval (RAG),fresh sourcespretrainingnext-token loss:always a guesscalibratedconfidencefine-tuningSFT on unknownfacts; gradersreward guessingteach abstaining,penalise errorsdecodingsampling; oneerror snowballslow temperature,verify (CoVe)answerno source andno way to abstaincite sources,check claimsfactuality (vs the world) is a separate axis: a faithful summary of a wrong source is still false
fig.Top: one source sentence and a three-claim summary. “Finished in 1899” contradicts the source's 1889 (intrinsic). “330 m tall” is entailed (faithful). “Repainted every 7 years” is not in the source at all (extrinsic): it happens to be true, but nothing given to the model supports it. Bottom: the stage where each main cause enters the pipeline, and the mitigation aimed at it.

In practice: closed-domain tasks (summarisation, RAG, extraction) are judged mainly on faithfulness to the supplied context. Open-domain questions with no context are judged on factuality, and the model can rely only on what its weights store.

Why models hallucinate

A language model is trained to maximise the likelihood of the next token (see Language modeling). The objective rewards a plausible continuation; nothing in it rewards stopping to say "I don't know". For facts the model saw many times, the most plausible continuation is usually the right one. For facts it saw once or never (a minor person's birthday, a paper's exact title) it still produces a plausible-looking guess.

Kalai and Vempala ("Calibrated Language Models Must Hallucinate", STOC 2024) made this precise for arbitrary facts with no learnable pattern. Using a Good–Turing argument, they showed that a calibrated model must hallucinate on such facts at a rate set by how many of them appear exactly once in the training data. Simplified:

hallucination rate  ≳  MF^  −  miscalibration\text{hallucination rate}\;\gtrsim\;\widehat{\mathrm{MF}}\;-\;\text{miscalibration}
MF^\widehat{\mathrm{MF}}
monofact rate: share of training facts (of this arbitrary kind) seen exactly once, the Good–Turing estimate of the unseen mass
miscalibration\text{miscalibration}
how far the model's probabilities are from calibrated; the bound has further small terms omitted here

The bound does not apply to facts with structure the model can compute, like arithmetic, or to facts repeated often in the data. It explains why pretraining alone cannot remove hallucination on rare facts, and why the fix must involve abstention or outside sources.

Post-training and evaluation add an incentive to guess. Kalai, Nachum, Vempala and Zhang ("Why Language Models Hallucinate", 2025) point out that most benchmarks grade answers right or wrong, with 0 for "I don't know". Under that grading, guessing never scores worse than abstaining. If a wrong answer costs kk points instead, answering with confidence pp has expected score:

E[score∣answer]=p⋅1−(1−p)⋅k  >  0  ⟺  p  >  t∗=kk+1\mathbb E[\text{score}\mid\text{answer}]=p\cdot1-(1-p)\cdot k\;>\;0\iff p\;>\;t^*=\frac{k}{k+1}
pp
probability the answer is correct (the model's confidence, if calibrated)
kk
penalty for a wrong answer; abstaining scores 0
t∗t^*
the confidence threshold above which answering pays: 0 for k=0k=0, 0.5 for k=1k=1, 0.75 for k=3k=3

Three more causes enter later. Gekhman et al. (EMNLP 2024) found that supervised fine-tuning on facts the base model did not know is learned slowly and makes the model more prone to hallucinate, so SFT data should match what the model already knows (see Supervised fine-tuning (SFT)). At decoding time, sampling can pick a low-probability wrong token, and later tokens then stay consistent with it. Zhang et al. (2023) call this snowballing: the model commits to an early error and justifies it. Long or cluttered contexts also hurt: Liu et al. ("Lost in the Middle", TACL 2024) found models use information in the middle of a long context less reliably than information at the start or end.

Checkable results

TruthfulQA
Lin, Hilton and Evans, ACL 2022. 817 questions in 38 categories built around common misconceptions. The best model tested (GPT-3 175B) was truthful on 58% of questions; humans on 94%. Larger models in the same family were less truthful.
Snowballing
Zhang et al., 2023. On questions where ChatGPT and GPT-4 gave wrong justifications, the same models identified 67% and 87% of those mistakes when asked about them separately.
Mata v. Avianca
S.D.N.Y., June 2023. Lawyers filed a brief citing six nonexistent court decisions generated by ChatGPT; the court fined them $5,000.
Chain-of-Verification
Dhuliawala et al. (Meta), 2023. Draft an answer, plan verification questions, answer them independently of the draft, then revise.

Abstention: trading coverage for accuracy

answer iff confidence ≥ t = 0.00 · score = (correct − 1·wrong) / 500stated confidence of 500 answers: correct / wrong00.250.50.751at t = 0.00answered500 (100.0%)correct358hallucinated142 (28.4%)precision71.6%score0.432best t0.52 → 0.556146 rare-fact questionscoverage · halluc. rate · precision00.5100.50.95threshold tscore, k = 1 · t* = k/(k+1) = 0.500.000.5600.50.95threshold t (dashed: t*)
wrong-answer penalty k
stated confidence
fig.500 synthetic questions: common facts (true correctness p from 0.75 to 0.99) and rare facts (p from 0.05 to 0.6). The model answers when its stated confidence is at least t; grading is +1 correct, −k wrong, 0 abstain. Defaults (30% rare, 146 questions; k = 1; calibrated; t = 0): all 500 answered, 142 wrong (28.4%), score 0.432. At t = 0.5: 75.0% answered, 10.0% hallucinated, precision 86.7%, score 0.550; the best is 0.556 at t = 0.52. With k = 0 the best threshold is t = 0: accuracy-only grading rewards guessing. With k = 3 guessing everything scores −0.136; the best is 0.408 at t = 0.76. Overconfident mode shifts every stated confidence up, so t = 0.5 abstains on nothing.

The score curve peaks near t∗=k/(k+1)t^*=k/(k+1) only when stated confidence is calibrated. Overconfident scores push the peak to a higher threshold, and you cannot know where without labelled data. So an abstention threshold needs a calibration set from your own traffic. Raising the rare-fact share to 0.6 makes the trade-off harsher: at t=0.75t=0.75 the model answers only 38.4% of questions, with a 4.4% hallucination rate.

Worked example: when to answer

pk = 0k = 1k = 3
0.950.950.900.80
0.800.800.600.20
0.600.600.20−0.60 (abstain)
0.400.40−0.20 (abstain)−1.40 (abstain)
0.200.20−0.60 (abstain)−2.20 (abstain)
policy total2.95 (answer all)1.70 (answer 3)1.00 (answer 2)
guess all2.950.90−3.20
expected wrong answers (policy)2.050.650.25
Five questions with calibrated confidence p. Expected score of answering is p − k(1 − p); the policy answers only when that is positive (p > k/(k+1))

With k=0k=0 the policy and guessing are identical and produce 2.05 expected wrong answers out of 5. With k=3k=3 abstaining on three questions cuts expected errors to 0.25 and lifts the score from −3.20 to 1.00. The model did not get more accurate. It got better at not answering.

Mitigation, layer by layer

What production systems stack

  1. 1Ground the answer. Retrieve sources and instruct the model to answer only from them (see RAG architecture). This turns open-domain factuality into closed-domain faithfulness, which is easier to check.
  2. 2Make "I don't know" a valid output. Say so in the system prompt, include abstaining examples in fine-tuning data, and score errors worse than abstentions in your evaluations.
  3. 3Estimate confidence. Token log-probabilities, or agreement among several sampled answers (the idea behind Self-consistency and SelfCheckGPT). Abstain or escalate below a threshold tuned on labelled traffic.
  4. 4Decode conservatively for facts. Use low temperature or greedy decoding for extraction and QA (see Temperature sampling). Use Structured output (JSON mode) so fields can be validated.
  5. 5Verify before showing. Require citations, check that each cited passage entails its claim, run a chain-of-verification pass, or call tools (search, a calculator, a database) for facts the model should not recall from memory.
  6. 6Show provenance in the UI. Link each claim to its source so users can check it, and log flagged outputs for review.
python
1from collections import Counter
2
3def answer_or_abstain(sample, question, n=5, k=3):
4 """Return the majority answer only if agreement beats t* = k/(k+1)."""
5 answers = [sample(question).strip().lower() for _ in range(n)]
6 top, count = Counter(answers).most_common(1)[0]
7 agreement = count / n # crude confidence estimate
8 t_star = k / (k + 1) # wrong answers cost k, abstain costs 0
9 return top if agreement > t_star else "I don't know"
10
11# stub samplers standing in for an LLM call
12from itertools import cycle
13known = cycle(["1889", "1889", "1889", "1889", "1887"])
14rare = cycle(["1912", "1915", "1912", "1908", "1921"])
15print(answer_or_abstain(lambda q: next(known), "When was the Eiffel Tower completed?")) # 1889 (0.8 > 0.75)
16print(answer_or_abstain(lambda q: next(rare), "When was this minor poet born?")) # I don't know (0.4)
Abstain unless sampled answers agree beyond the break-even threshold k/(k+1); `sample` is any function returning one model answer at moderate temperature (runs as is with a stub)

Check yourself

A benchmark gives 1 point per correct answer and 0 for both wrong answers and "I don't know". What does a model maximising this score do on a question it is 10% sure about?

Check yourself

The source says a drug was approved in 2019. The summary says it was approved in 2019 by the FDA's fast-track programme, which the source never mentions. Which label fits the added detail?

Remember

  • ✓Intrinsic hallucinations contradict the source; extrinsic ones add unsupported content (Maynez et al., 2020). Faithfulness (vs source) and factuality (vs world) are separate checks.
  • ✓Likelihood training always yields a guess; for arbitrary facts seen once, calibrated models must hallucinate at roughly the monofact rate (Kalai and Vempala, 2024).
  • ✓Accuracy-only grading rewards guessing; with penalty kk per error, answer only when confidence exceeds k/(k+1)k/(k+1).
  • ✓SFT on facts the model does not know, sampling, error snowballing and long cluttered contexts all add hallucinations.
  • ✓Stack mitigations: grounding, abstention, confidence estimation, conservative decoding, claim-level verification, visible provenance. Measure them with groundedness evaluation.
← back to Generative AI roadmap