Hallucination detection

Flag a generation as unsupported or false by scoring it: low token probability, disagreement between sampled answers, high semantic entropy, contradiction with the source, or a probe on hidden states. Compare detectors by AUROC.

8 min read876 wordsadvanced0 viewsupdated 2026-10-15

Definition

Hallucination detection means assigning each model output (an answer, a sentence, a claim) a score s(x)s(x) that is high when the content is not supported by the source or by the world. The output is then flagged, withheld or sent to retrieval. Main methods: token-probability scores, sampling consistency (SelfCheckGPT, Manakul, Liusie and Gales, EMNLP 2023), semantic entropy (Kuhn, Gal and Farquhar, ICLR 2023; Farquhar, Kossen, Kuhn and Gal, Nature 2024), NLI checks against a source, and probes on hidden states.

A hallucination is fluent text that is not grounded. Maynez et al. (ACL 2020), studying abstractive summarisation, split hallucinations into two kinds. Intrinsic ones contradict the source: the source says Warsaw, the summary says Kraków. Extrinsic ones add content the source cannot confirm or refute. Ji et al.'s 2023 survey (ACM Computing Surveys) uses the same split for every generation task.

Extrinsic does not mean false. An added fact can be true about the world but still unfaithful to the document. So detection needs a reference point: the given source (faithfulness) or world knowledge (factuality). Scoring a model's overall factual precision is a separate job, covered in Factuality evaluation.

Why is detection hard? The model's decoding objective rewards plausible text, not true text, and a wrong answer reads exactly like a right one. Before LLMs, summarisation used entailment classifiers and QA-based consistency checks against the source. Open-ended generation has no source to check against, so most modern detectors look at the model's own uncertainty.

two kinds of hallucination, relative to a source documentsource Marie Curie was born in Warsaw in 1867.output Curie was born in Kraków in 1867 and loved chess.intrinsic: contradicts the sourceextrinsic: source can't confirm or refuteQ: Where was MarieCurie born?LLMgreedy + T=1greedy: Kraków p = 0.4 × 0.7 = 0.2810 samples: Kraków 3 · Paris 3Warsaw 2 (1 as Warszawa) · Lublin 2token probability−(1/T)Σ log p(y_t)greedy “Kraków”−log 0.28 = 1.27needs logitsSelfCheckGPTdo the samplessupport it?1 − 3/10 = 0.70black-boxsemantic entropycluster samples bytwo-way entailment4 meanings: 1.37black-box + NLINLI vs sourcesource ⊨ claim?Warsaw vs Kraków→ contradictionneeds a sourceactivation probeσ(wᵀh_ℓ) on ahidden layertrained on labelsneeds activationsscore s(x) → flag if s ≥ τ → abstain, retrieve, or warn the userevaluated with AUROC = P(s(hallucinated) > s(correct)); 0.5 = chance, 1 = perfecttwo-way entailment: a ⊨ b and b ⊨ a, so “Warsaw” and “Warszawa” share one meaning
fig.Top: intrinsic vs extrinsic hallucination relative to a source. Bottom: one question, a greedy answer and ten samples feed five detector families. Each produces a score s(x). A threshold turns the score into a flag, and AUROC measures how well the score ranks hallucinated answers above correct ones.

Uncertainty from one pass: token probabilities

The cheapest signal is the model's probability for its own answer. You can take the mean negative log-probability per token, or the maximum over tokens, which catches a single shaky entity inside an otherwise confident sentence:

stok(x)=−1T∑t=1Tlog⁡pθ ⁣(yt∣y<t,x),smax⁡(x)=max⁡t(−log⁡pθ(yt∣y<t,x))s_{\text{tok}}(x)=-\frac{1}{T}\sum_{t=1}^{T}\log p_\theta\!\left(y_t\mid y_{<t},x\right),\qquad s_{\max}(x)=\max_t\big(-\log p_\theta(y_t\mid y_{<t},x)\big)
y1:Ty_{1:T}
the generated answer, T tokens
pθp_\theta
the generating model's next-token distribution (needs logit access)

The weakness is that probability is spread over wordings, not meanings. If the model writes "Canberra", "Canberra." and "canberra", each string gets only part of the mass. Its confidence in the fact is high, yet each surface form looks uncertain. Kadavath et al. (2022) found that large models are reasonably calibrated when asked directly whether their own answer is true (P(True)), which is a separate, prompt-based signal.

Sampling consistency and semantic entropy

SelfCheckGPT needs only black-box access. Sample NN extra responses at temperature 1. Then, for each sentence of the main response, measure how far the samples support it, using BERTScore, QA, n-gram overlap, NLI or an LLM prompt. The idea: when the model knows a fact, its samples repeat it. When it is guessing, the samples scatter. The paper tested this on GPT-3 biographies from WikiBio, annotated sentence by sentence.

Semantic entropy measures that scatter in meaning space. Two answers are put in the same cluster when each entails the other (bidirectional entailment, checked with an NLI model). Then the entropy is taken over clusters instead of strings:

SE(x)=−∑c∈Cp(c∣x)log⁡p(c∣x),p(c∣x)=∑s∈cp(s∣x)  ≈  ∣c∣N\mathrm{SE}(x)=-\sum_{c\in\mathcal{C}}p(c\mid x)\log p(c\mid x),\qquad p(c\mid x)=\sum_{s\in c}p(s\mid x)\;\approx\;\frac{|c|}{N}
C\mathcal{C}
meaning clusters found among the N sampled answers
p(s∣x)p(s\mid x)
sequence probability of sample s; the discrete variant replaces it by counts and needs no logits

The Nature paper calls the errors this catches confabulations: claims that are both wrong and arbitrary, changing from sample to sample. It does not catch a model that gives the same wrong answer every time. Probes target that case. A linear classifier σ(w⊤hℓ)\sigma(w^\top h_\ell) on one layer's hidden state is trained on answers labelled true or false (Azaria and Mitchell, 2023), or trained to predict semantic entropy from a single pass (semantic entropy probes, Kossen et al., 2024).

When a source exists (summaries, RAG answers), use NLI against it. Split the output into sentences or claims and ask an entailment model whether the retrieved or given text entails each one. Contradiction marks an intrinsic hallucination. "Neutral" marks an extrinsic one.

DetectorSignalAccessExtra cost
Token probabilitymean or max −log p of the answerlogitsnone (same pass)
P(True) self-evaluationmodel's probability that its answer is truelogits + a prompt1 pass
SelfCheckGPTsupport of each sentence by N samplesblack-boxN generations + checks
Semantic entropyentropy over meaning clustersblack-box (+ an NLI model)N generations + up to O(N2)O(N^2) NLI calls
NLI vs sourceentail / neutral / contradict per claima source document1 NLI call per claim–passage pair
Hidden-state probeσ(w⊤hℓ)\sigma(w^\top h_\ell)activations + labelled dataa dot product
Detector families: what they need and what they cost per answer

Scoring the detector: AUROC

A detector outputs a score, not a verdict, so it is judged over a labelled set of correct and hallucinated answers by the area under the ROC curve:

AUROC=P(s(x+)>s(x−))+12 P(s(x+)=s(x−))\mathrm{AUROC}=P\big(s(x^{+})>s(x^{-})\big)+\tfrac12\,P\big(s(x^{+})=s(x^{-})\big)
x+, x−x^{+},\,x^{-}
a random hallucinated answer and a random correct one
0.5/10.5 / 1
chance ranking / perfect separation; independent of any threshold

AUROC ignores the threshold and the base rate. In deployment you still choose τ\tau, and the cost of a false alarm (refusing a correct answer) sets it. The Nature paper also reports AURAC, the accuracy of the answers kept when the most uncertain ones are rejected.

Where was Marie Curie born? · greedy “Kraków”hallucinated (truth: Warsaw)ParisWarszawaWarsawKrakówLublinKrakówKrakówLublinParisParisKrakowKrakówWarsawWarsawLublinKrakówWarszawaLublinWarsawKrakow10 samples → meanings: Kraków 3 · Warsaw 2 · Paris 3 · Lublin 25 distinct strings → lexical H = 1.505 · 4 meanings → semantic H = 1.366SelfCheck = 1 − 3/10 = 0.700 · −log p = −log(0.40·0.70) = 1.273semantic entropy 1.366 ≥ 0.752 → flagged (right)000.50.511false-alarm rate (correct answers flagged)hallucinations caughtAUROC over 64 questions, N = 10−log p(greedy)0.685lexical entropy0.747semantic entropy0.882SelfCheck disagree0.828bar: 0.5 (chance) → 1flag if semantic entropy ≥ 0.752hallucinated293correct626flaggedpassedcaught 91% · false alarms 19% · precision 83%semantic entropy of all 64 questions (● hallucinated, ● correct, ring = this question)01.50
detector
fig.A toy QA model over 64 questions, 32 of them hallucinated. At the default N = 10 samples, semantic entropy reaches AUROC 0.882, SelfCheck disagreement 0.828, lexical (string) entropy 0.747 and −log p of the greedy answer 0.685. With the threshold at 50% (0.752 nats), semantic entropy catches 29 of the 32 hallucinations and raises 6 false alarms. On question 1 the ten samples are 3 strings with one meaning, so lexical entropy is 1.055 but semantic entropy is 0.500. At N = 2, semantic entropy can only be 0 or ln 2 and its AUROC falls to 0.625. At N = 20 it rises to 0.897.

Worked example

QuestionSample countsLexical HSemantic HSelfCheck 1 − agree
Capital of Australia (correct)Canberra 4, Canberra. 4, Sydney 2H(.4,.4,.2)=1.055H(.4,.4,.2)=1.055H(.8,.2)=0.500H(.8,.2)=0.5001 − 8/10 = 0.20
Author of The Master and Margarita (correct)Bulgakov 4, M.Bulgakov 4, Bulgakov. 21.05500
Marie Curie's birthplace (greedy Kraków, wrong)Kraków 3, Paris 3, Lublin 2, Warsaw 1, Warszawa 1H(.3,.3,.2,.1,.1)=1.505H(.3,.3,.2,.1,.1)=1.505H(.3,.3,.2,.2)=1.366H(.3,.3,.2,.2)=1.3661 − 3/10 = 0.70
Ten samples per question (natural log). The Curie answers are the viz's question 3: Paris, Warszawa, Warsaw, Kraków, Lublin, Kraków, Kraków, Lublin, Paris, Paris.

Lexical entropy gives both correct questions the same 1.055, because rewordings look like disagreement. Semantic entropy gives 0.500, 0 and 1.366, which ranks the hallucination last. AUROC on a tiny set: hallucinated scores (1.37,0.95,0.50)(1.37, 0.95, 0.50), correct scores (0.50,0.30,0)(0.50, 0.30, 0). Of the 9 pairs, 8 are ordered correctly and 1 is tied, so AUROC=(8+0.5)/9=0.944\mathrm{AUROC}=(8+0.5)/9=0.944.

python
1import math
2from transformers import pipeline
3from sklearn.metrics import roc_auc_score
4
5nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")
6
7def entails(q, a, b):
8 out = nli([{"text": f"{q} {a}", "text_pair": f"{q} {b}"}])[0]
9 return out["label"] == "ENTAILMENT"
10
11def semantic_entropy(q, answers):
12 clusters = [] # greedy clustering by two-way entailment
13 for a in answers:
14 for c in clusters:
15 if entails(q, c[0], a) and entails(q, a, c[0]):
16 c.append(a)
17 break
18 else:
19 clusters.append([a])
20 p = [len(c) / len(answers) for c in clusters]
21 return -sum(pi * math.log(pi) for pi in p)
22
23ans = ["Paris", "Warszawa", "Warsaw", "Kraków", "Lublin",
24 "Kraków", "Kraków", "Lublin", "Paris", "Paris"]
25print(semantic_entropy("Where was Marie Curie born?", ans)) # 1.366 if Warsaw = Warszawa
26
27# over a labelled set: roc_auc_score(is_hallucinated, scores)
Discrete semantic entropy with an off-the-shelf MNLI model (sketch; not run here)

Use when

  • +Black-box methods (SelfCheckGPT, discrete semantic entropy) work through an API with no logits.
  • +Uncertainty scores need no labelled data and no reference text.
  • +Probes and token scores add almost no latency.

Avoid when

  • −Sampling multiplies inference cost by about NN, plus the NLI calls.
  • −Confident, repeated errors get low uncertainty scores.
  • −Probes need labelled data and access to activations, and may not transfer across domains or models.

Check yourself

Ten samples read "Bulgakov" ×4, "M.Bulgakov" ×4, "Bulgakov." ×2. What are the lexical and the (discrete) semantic entropy?

Check yourself

A detector has AUROC 0.5 on a balanced test set. What does that mean?

Remember

  • ✓Intrinsic hallucinations contradict the source. Extrinsic ones cannot be checked against it, and they may still be true.
  • ✓Token score: −1T∑log⁡p(yt)-\frac1T\sum\log p(y_t). It is cheap but confounded by wording.
  • ✓SelfCheckGPT: sample N responses and check whether they support each sentence. Black-box.
  • ✓Semantic entropy: cluster samples by bidirectional entailment, then −∑cp(c)log⁡p(c)-\sum_c p(c)\log p(c). It catches arbitrary confabulations, not consistent errors.
  • ✓NLI against a source catches intrinsic errors. Hidden-state probes can catch errors without sampling.
  • ✓Detectors are compared by AUROC =P(s(x+)>s(x−))=P(s(x^+)>s(x^-)). The deployment threshold is set by the cost of false alarms.
← back to Deep Learning roadmap