Definition
Hallucination detection means assigning each model output (an answer, a sentence, a claim) a score that is high when the content is not supported by the source or by the world. The output is then flagged, withheld or sent to retrieval. Main methods: token-probability scores, sampling consistency (SelfCheckGPT, Manakul, Liusie and Gales, EMNLP 2023), semantic entropy (Kuhn, Gal and Farquhar, ICLR 2023; Farquhar, Kossen, Kuhn and Gal, Nature 2024), NLI checks against a source, and probes on hidden states.
A hallucination is fluent text that is not grounded. Maynez et al. (ACL 2020), studying abstractive summarisation, split hallucinations into two kinds. Intrinsic ones contradict the source: the source says Warsaw, the summary says Kraków. Extrinsic ones add content the source cannot confirm or refute. Ji et al.'s 2023 survey (ACM Computing Surveys) uses the same split for every generation task.
Extrinsic does not mean false. An added fact can be true about the world but still unfaithful to the document. So detection needs a reference point: the given source (faithfulness) or world knowledge (factuality). Scoring a model's overall factual precision is a separate job, covered in Factuality evaluation.
Why is detection hard? The model's decoding objective rewards plausible text, not true text, and a wrong answer reads exactly like a right one. Before LLMs, summarisation used entailment classifiers and QA-based consistency checks against the source. Open-ended generation has no source to check against, so most modern detectors look at the model's own uncertainty.
Uncertainty from one pass: token probabilities
The cheapest signal is the model's probability for its own answer. You can take the mean negative log-probability per token, or the maximum over tokens, which catches a single shaky entity inside an otherwise confident sentence:
- the generated answer, T tokens
- the generating model's next-token distribution (needs logit access)
The weakness is that probability is spread over wordings, not meanings. If the model writes "Canberra", "Canberra." and "canberra", each string gets only part of the mass. Its confidence in the fact is high, yet each surface form looks uncertain. Kadavath et al. (2022) found that large models are reasonably calibrated when asked directly whether their own answer is true (P(True)), which is a separate, prompt-based signal.
Sampling consistency and semantic entropy
SelfCheckGPT needs only black-box access. Sample extra responses at temperature 1. Then, for each sentence of the main response, measure how far the samples support it, using BERTScore, QA, n-gram overlap, NLI or an LLM prompt. The idea: when the model knows a fact, its samples repeat it. When it is guessing, the samples scatter. The paper tested this on GPT-3 biographies from WikiBio, annotated sentence by sentence.
Semantic entropy measures that scatter in meaning space. Two answers are put in the same cluster when each entails the other (bidirectional entailment, checked with an NLI model). Then the entropy is taken over clusters instead of strings:
- meaning clusters found among the N sampled answers
- sequence probability of sample s; the discrete variant replaces it by counts and needs no logits
The Nature paper calls the errors this catches confabulations: claims that are both wrong and arbitrary, changing from sample to sample. It does not catch a model that gives the same wrong answer every time. Probes target that case. A linear classifier on one layer's hidden state is trained on answers labelled true or false (Azaria and Mitchell, 2023), or trained to predict semantic entropy from a single pass (semantic entropy probes, Kossen et al., 2024).
When a source exists (summaries, RAG answers), use NLI against it. Split the output into sentences or claims and ask an entailment model whether the retrieved or given text entails each one. Contradiction marks an intrinsic hallucination. "Neutral" marks an extrinsic one.
| Detector | Signal | Access | Extra cost |
|---|---|---|---|
| Token probability | mean or max −log p of the answer | logits | none (same pass) |
| P(True) self-evaluation | model's probability that its answer is true | logits + a prompt | 1 pass |
| SelfCheckGPT | support of each sentence by N samples | black-box | N generations + checks |
| Semantic entropy | entropy over meaning clusters | black-box (+ an NLI model) | N generations + up to NLI calls |
| NLI vs source | entail / neutral / contradict per claim | a source document | 1 NLI call per claim–passage pair |
| Hidden-state probe | activations + labelled data | a dot product |
Scoring the detector: AUROC
A detector outputs a score, not a verdict, so it is judged over a labelled set of correct and hallucinated answers by the area under the ROC curve:
- a random hallucinated answer and a random correct one
- chance ranking / perfect separation; independent of any threshold
AUROC ignores the threshold and the base rate. In deployment you still choose , and the cost of a false alarm (refusing a correct answer) sets it. The Nature paper also reports AURAC, the accuracy of the answers kept when the most uncertain ones are rejected.
Worked example
| Question | Sample counts | Lexical H | Semantic H | SelfCheck 1 − agree |
|---|---|---|---|---|
| Capital of Australia (correct) | Canberra 4, Canberra. 4, Sydney 2 | 1 − 8/10 = 0.20 | ||
| Author of The Master and Margarita (correct) | Bulgakov 4, M.Bulgakov 4, Bulgakov. 2 | 1.055 | 0 | 0 |
| Marie Curie's birthplace (greedy Kraków, wrong) | Kraków 3, Paris 3, Lublin 2, Warsaw 1, Warszawa 1 | 1 − 3/10 = 0.70 |
Lexical entropy gives both correct questions the same 1.055, because rewordings look like disagreement. Semantic entropy gives 0.500, 0 and 1.366, which ranks the hallucination last. AUROC on a tiny set: hallucinated scores , correct scores . Of the 9 pairs, 8 are ordered correctly and 1 is tied, so .
1import math2from transformers import pipeline3from sklearn.metrics import roc_auc_score45nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")67def entails(q, a, b):8 out = nli([{"text": f"{q} {a}", "text_pair": f"{q} {b}"}])[0]9 return out["label"] == "ENTAILMENT"1011def semantic_entropy(q, answers):12 clusters = [] # greedy clustering by two-way entailment13 for a in answers:14 for c in clusters:15 if entails(q, c[0], a) and entails(q, a, c[0]):16 c.append(a)17 break18 else:19 clusters.append([a])20 p = [len(c) / len(answers) for c in clusters]21 return -sum(pi * math.log(pi) for pi in p)2223ans = ["Paris", "Warszawa", "Warsaw", "Kraków", "Lublin",24 "Kraków", "Kraków", "Lublin", "Paris", "Paris"]25print(semantic_entropy("Where was Marie Curie born?", ans)) # 1.366 if Warsaw = Warszawa2627# over a labelled set: roc_auc_score(is_hallucinated, scores)Use when
- +Black-box methods (SelfCheckGPT, discrete semantic entropy) work through an API with no logits.
- +Uncertainty scores need no labelled data and no reference text.
- +Probes and token scores add almost no latency.
Avoid when
- −Sampling multiplies inference cost by about , plus the NLI calls.
- −Confident, repeated errors get low uncertainty scores.
- −Probes need labelled data and access to activations, and may not transfer across domains or models.
Check yourself
Ten samples read "Bulgakov" ×4, "M.Bulgakov" ×4, "Bulgakov." ×2. What are the lexical and the (discrete) semantic entropy?
Check yourself
A detector has AUROC 0.5 on a balanced test set. What does that mean?
Remember
- ✓Intrinsic hallucinations contradict the source. Extrinsic ones cannot be checked against it, and they may still be true.
- ✓Token score: . It is cheap but confounded by wording.
- ✓SelfCheckGPT: sample N responses and check whether they support each sentence. Black-box.
- ✓Semantic entropy: cluster samples by bidirectional entailment, then . It catches arbitrary confabulations, not consistent errors.
- ✓NLI against a source catches intrinsic errors. Hidden-state probes can catch errors without sampling.
- ✓Detectors are compared by AUROC . The deployment threshold is set by the cost of false alarms.