# Hallucination

> A hallucination is fluent output not supported by the source (intrinsic: contradicts it; extrinsic: adds to it) or by the world. Likelihood training and accuracy-only grading reward guessing; retrieval, abstention and verification reduce it.

Source: https://www.mlfoundry.in/learn/hallucination  
Section: Generative AI › Foundations of Generative AI  
Level: core · 8 min read · updated 2026-10-21

**Hallucination:** is generated content that reads as fluent and confident but is not supported by the input the model was given (unfaithful) or by verifiable facts about the world (non-factual). Maynez, Narayan, Bohnet and McDonald ("On Faithfulness and Factuality in Abstractive Summarization", ACL 2020) split it into **intrinsic** hallucinations, which contradict the source, and **extrinsic** ones, which add information the source cannot confirm. Ji et al. (*ACM Computing Surveys*, 2023) and Huang et al. (2023) survey the field.

It matters wherever text is trusted without checking: a summary that changes a date, a RAG answer citing a passage that says the opposite, a legal brief with invented case law, code that imports a package that does not exist. The model's tone gives no signal. A wrong answer is generated by the same sampling process as a right one.

This article covers what hallucination is, why it happens and how systems reduce it. Measuring it is a separate job: see **Hallucination and groundedness evaluation** in this branch, and **Hallucination detection** and **Factuality evaluation** in the NLP module.

## Two axes: source and world

**Faithfulness** asks whether each claim is supported by the context the model was given: the document to summarise, the retrieved passages, the conversation. **Factuality** asks whether the claim is true in the world. The two come apart. A faithful summary of an outdated source is non-factual. An extrinsic addition can be true and still a faithfulness failure, because the reader cannot check it against the source.

> Figure: Top: one source sentence and a three-claim summary. “Finished in 1899” contradicts the source's 1889 (intrinsic). “330 m tall” is entailed (faithful). “Repainted every 7 years” is not in the source at all (extrinsic): it happens to be true, but nothing given to the model supports it. Bottom: the stage where each main cause enters the pipeline, and the mitigation aimed at it.

In practice: closed-domain tasks (summarisation, RAG, extraction) are judged mainly on faithfulness to the supplied context. Open-domain questions with no context are judged on factuality, and the model can rely only on what its weights store.

## Why models hallucinate

A language model is trained to maximise the likelihood of the next token (see **Language modeling**). The objective rewards a plausible continuation; nothing in it rewards stopping to say "I don't know". For facts the model saw many times, the most plausible continuation is usually the right one. For facts it saw once or never (a minor person's birthday, a paper's exact title) it still produces a plausible-looking guess.

Kalai and Vempala ("Calibrated Language Models Must Hallucinate", STOC 2024) made this precise for arbitrary facts with no learnable pattern. Using a Good–Turing argument, they showed that a calibrated model must hallucinate on such facts at a rate set by how many of them appear exactly once in the training data. Simplified:

$$
\text{hallucination rate}\;\gtrsim\;\widehat{\mathrm{MF}}\;-\;\text{miscalibration}
$$

- $\widehat{\mathrm{MF}}$: monofact rate: share of training facts (of this arbitrary kind) seen exactly once, the Good–Turing estimate of the unseen mass
- $\text{miscalibration}$: how far the model's probabilities are from calibrated; the bound has further small terms omitted here

The bound does not apply to facts with structure the model can compute, like arithmetic, or to facts repeated often in the data. It explains why pretraining alone cannot remove hallucination on rare facts, and why the fix must involve abstention or outside sources.

Post-training and evaluation add an incentive to guess. Kalai, Nachum, Vempala and Zhang ("Why Language Models Hallucinate", 2025) point out that most benchmarks grade answers right or wrong, with 0 for "I don't know". Under that grading, guessing never scores worse than abstaining. If a wrong answer costs $k$ points instead, answering with confidence $p$ has expected score:

$$
\mathbb E[\text{score}\mid\text{answer}]=p\cdot1-(1-p)\cdot k\;>\;0\iff p\;>\;t^*=\frac{k}{k+1}
$$

- $p$: probability the answer is correct (the model's confidence, if calibrated)
- $k$: penalty for a wrong answer; abstaining scores 0
- $t^*$: the confidence threshold above which answering pays: 0 for $k=0$, 0.5 for $k=1$, 0.75 for $k=3$

Three more causes enter later. Gekhman et al. (EMNLP 2024) found that supervised fine-tuning on facts the base model did not know is learned slowly and makes the model more prone to hallucinate, so SFT data should match what the model already knows (see **Supervised fine-tuning (SFT)**). At decoding time, sampling can pick a low-probability wrong token, and later tokens then stay consistent with it. Zhang et al. (2023) call this **snowballing**: the model commits to an early error and justifies it. Long or cluttered contexts also hurt: Liu et al. ("Lost in the Middle", *TACL* 2024) found models use information in the middle of a long context less reliably than information at the start or end.

**Checkable results**

- **TruthfulQA:** Lin, Hilton and Evans, ACL 2022. 817 questions in 38 categories built around common misconceptions. The best model tested (GPT-3 175B) was truthful on 58% of questions; humans on 94%. Larger models in the same family were less truthful.
- **Snowballing:** Zhang et al., 2023. On questions where ChatGPT and GPT-4 gave wrong justifications, the same models identified 67% and 87% of those mistakes when asked about them separately.
- **Mata v. Avianca:** S.D.N.Y., June 2023. Lawyers filed a brief citing six nonexistent court decisions generated by ChatGPT; the court fined them $5,000.
- **Chain-of-Verification:** Dhuliawala et al. (Meta), 2023. Draft an answer, plan verification questions, answer them independently of the draft, then revise.

## Abstention: trading coverage for accuracy

> Figure: 500 synthetic questions: common facts (true correctness p from 0.75 to 0.99) and rare facts (p from 0.05 to 0.6). The model answers when its stated confidence is at least t; grading is +1 correct, −k wrong, 0 abstain. Defaults (30% rare, 146 questions; k = 1; calibrated; t = 0): all 500 answered, 142 wrong (28.4%), score 0.432. At t = 0.5: 75.0% answered, 10.0% hallucinated, precision 86.7%, score 0.550; the best is 0.556 at t = 0.52. With k = 0 the best threshold is t = 0: accuracy-only grading rewards guessing. With k = 3 guessing everything scores −0.136; the best is 0.408 at t = 0.76. Overconfident mode shifts every stated confidence up, so t = 0.5 abstains on nothing.

The score curve peaks near $t^*=k/(k+1)$ only when stated confidence is calibrated. Overconfident scores push the peak to a higher threshold, and you cannot know where without labelled data. So an abstention threshold needs a calibration set from your own traffic. Raising the rare-fact share to 0.6 makes the trade-off harsher: at $t=0.75$ the model answers only 38.4% of questions, with a 4.4% hallucination rate.

## Worked example: when to answer

Five questions with calibrated confidence p. Expected score of answering is p − k(1 − p); the policy answers only when that is positive (p > k/(k+1))

| p | k = 0 | k = 1 | k = 3 |
| --- | --- | --- | --- |
| 0.95 | 0.95 | 0.90 | 0.80 |
| 0.80 | 0.80 | 0.60 | 0.20 |
| 0.60 | 0.60 | 0.20 | −0.60 (abstain) |
| 0.40 | 0.40 | −0.20 (abstain) | −1.40 (abstain) |
| 0.20 | 0.20 | −0.60 (abstain) | −2.20 (abstain) |
| policy total | 2.95 (answer all) | 1.70 (answer 3) | 1.00 (answer 2) |
| guess all | 2.95 | 0.90 | −3.20 |
| expected wrong answers (policy) | 2.05 | 0.65 | 0.25 |

With $k=0$ the policy and guessing are identical and produce 2.05 expected wrong answers out of 5. With $k=3$ abstaining on three questions cuts expected errors to 0.25 and lifts the score from −3.20 to 1.00. The model did not get more accurate. It got better at not answering.

## Mitigation, layer by layer

**What production systems stack**

1. **Ground the answer.** Retrieve sources and instruct the model to answer only from them (see **RAG architecture**). This turns open-domain factuality into closed-domain faithfulness, which is easier to check.
2. **Make "I don't know" a valid output.** Say so in the system prompt, include abstaining examples in fine-tuning data, and score errors worse than abstentions in your evaluations.
3. **Estimate confidence.** Token log-probabilities, or agreement among several sampled answers (the idea behind **Self-consistency** and SelfCheckGPT). Abstain or escalate below a threshold tuned on labelled traffic.
4. **Decode conservatively for facts.** Use low temperature or greedy decoding for extraction and QA (see **Temperature sampling**). Use **Structured output (JSON mode)** so fields can be validated.
5. **Verify before showing.** Require citations, check that each cited passage entails its claim, run a chain-of-verification pass, or call tools (search, a calculator, a database) for facts the model should not recall from memory.
6. **Show provenance in the UI.** Link each claim to its source so users can check it, and log flagged outputs for review.

```python
from collections import Counter

def answer_or_abstain(sample, question, n=5, k=3):
    """Return the majority answer only if agreement beats t* = k/(k+1)."""
    answers = [sample(question).strip().lower() for _ in range(n)]
    top, count = Counter(answers).most_common(1)[0]
    agreement = count / n                 # crude confidence estimate
    t_star = k / (k + 1)                  # wrong answers cost k, abstain costs 0
    return top if agreement > t_star else "I don't know"

# stub samplers standing in for an LLM call
from itertools import cycle
known = cycle(["1889", "1889", "1889", "1889", "1887"])
rare = cycle(["1912", "1915", "1912", "1908", "1921"])
print(answer_or_abstain(lambda q: next(known), "When was the Eiffel Tower completed?"))  # 1889 (0.8 > 0.75)
print(answer_or_abstain(lambda q: next(rare), "When was this minor poet born?"))        # I don't know (0.4)
```

Abstain unless sampled answers agree beyond the break-even threshold k/(k+1); `sample` is any function returning one model answer at moderate temperature (runs as is with a stub)

> **Retrieval does not end hallucination:** A RAG system can still ignore a retrieved passage, merge two passages into a claim neither makes, or attach a real citation to an unsupported sentence. It can also faithfully repeat a wrong document. Check claim-level support against the cited text, not just whether a citation is present.

> **Consistency catches guesses, not stable errors:** Sampling agreement flags answers the model is unsure of, because guesses vary between samples. A misconception the model learned firmly comes out the same every time and passes the check. TruthfulQA-style errors need outside evidence, not self-agreement.

**Question:** A benchmark gives 1 point per correct answer and 0 for both wrong answers and "I don't know". What does a model maximising this score do on a question it is 10% sure about?

- A. Abstains, because 10% is low
- B. Guesses, because guessing has expected score 0.1 > 0
- C. Abstains only if it is calibrated
- D. Refuses the whole benchmark

**Answer:** B. Guesses, because guessing has expected score 0.1 > 0 — With $k=0$ the break-even threshold $k/(k+1)$ is 0, so any guess beats abstaining. This is the incentive Kalai et al. (2025) identify: accuracy-only leaderboards reward bluffing.

**Question:** The source says a drug was approved in 2019. The summary says it was approved in 2019 by the FDA's fast-track programme, which the source never mentions. Which label fits the added detail?

- A. Intrinsic hallucination
- B. Extrinsic hallucination
- C. Faithful content
- D. A factuality error by definition

**Answer:** B. Extrinsic hallucination — It does not contradict the source; it adds information the source cannot confirm. That is extrinsic. Whether it is also non-factual depends on the world, which a faithfulness check cannot decide.

**Key points**

- Intrinsic hallucinations contradict the source; extrinsic ones add unsupported content (Maynez et al., 2020). Faithfulness (vs source) and factuality (vs world) are separate checks.
- Likelihood training always yields a guess; for arbitrary facts seen once, calibrated models must hallucinate at roughly the monofact rate (Kalai and Vempala, 2024).
- Accuracy-only grading rewards guessing; with penalty $k$ per error, answer only when confidence exceeds $k/(k+1)$.
- SFT on facts the model does not know, sampling, error snowballing and long cluttered contexts all add hallucinations.
- Stack mitigations: grounding, abstention, confidence estimation, conservative decoding, claim-level verification, visible provenance. Measure them with groundedness evaluation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
