# Hallucination detection

> Flag a generation as unsupported or false by scoring it: low token probability, disagreement between sampled answers, high semantic entropy, contradiction with the source, or a probe on hidden states. Compare detectors by AUROC.

Source: https://www.mlfoundry.in/learn/hallucination-detection  
Section: Deep Learning › Deep Learning for NLP  
Level: advanced · 8 min read · updated 2026-10-15

**Hallucination detection:** means assigning each model output (an answer, a sentence, a claim) a score $s(x)$ that is high when the content is not supported by the source or by the world. The output is then flagged, withheld or sent to retrieval. Main methods: token-probability scores, sampling consistency (**SelfCheckGPT**, Manakul, Liusie and Gales, EMNLP 2023), **semantic entropy** (Kuhn, Gal and Farquhar, ICLR 2023; Farquhar, Kossen, Kuhn and Gal, *Nature* 2024), NLI checks against a source, and probes on hidden states.

A hallucination is fluent text that is not grounded. Maynez et al. (ACL 2020), studying abstractive summarisation, split hallucinations into two kinds. **Intrinsic** ones contradict the source: the source says Warsaw, the summary says Kraków. **Extrinsic** ones add content the source cannot confirm or refute. Ji et al.'s 2023 survey (*ACM Computing Surveys*) uses the same split for every generation task.

Extrinsic does not mean false. An added fact can be true about the world but still unfaithful to the document. So detection needs a reference point: the given source (faithfulness) or world knowledge (factuality). Scoring a model's overall factual precision is a separate job, covered in **Factuality evaluation**.

Why is detection hard? The model's decoding objective rewards plausible text, not true text, and a wrong answer reads exactly like a right one. Before LLMs, summarisation used entailment classifiers and QA-based consistency checks against the source. Open-ended generation has no source to check against, so most modern detectors look at the model's own uncertainty.

> Figure: Top: intrinsic vs extrinsic hallucination relative to a source. Bottom: one question, a greedy answer and ten samples feed five detector families. Each produces a score s(x). A threshold turns the score into a flag, and AUROC measures how well the score ranks hallucinated answers above correct ones.

## Uncertainty from one pass: token probabilities

The cheapest signal is the model's probability for its own answer. You can take the mean negative log-probability per token, or the maximum over tokens, which catches a single shaky entity inside an otherwise confident sentence:

$$
s_{\text{tok}}(x)=-\frac{1}{T}\sum_{t=1}^{T}\log p_\theta\!\left(y_t\mid y_{<t},x\right),\qquad s_{\max}(x)=\max_t\big(-\log p_\theta(y_t\mid y_{<t},x)\big)
$$

- $y_{1:T}$: the generated answer, T tokens
- $p_\theta$: the generating model's next-token distribution (needs logit access)

The weakness is that probability is spread over **wordings**, not meanings. If the model writes "Canberra", "Canberra." and "canberra", each string gets only part of the mass. Its confidence in the fact is high, yet each surface form looks uncertain. Kadavath et al. (2022) found that large models are reasonably calibrated when asked directly whether their own answer is true (P(True)), which is a separate, prompt-based signal.

## Sampling consistency and semantic entropy

**SelfCheckGPT** needs only black-box access. Sample $N$ extra responses at temperature 1. Then, for each sentence of the main response, measure how far the samples support it, using BERTScore, QA, n-gram overlap, NLI or an LLM prompt. The idea: when the model knows a fact, its samples repeat it. When it is guessing, the samples scatter. The paper tested this on GPT-3 biographies from WikiBio, annotated sentence by sentence.

**Semantic entropy** measures that scatter in meaning space. Two answers are put in the same cluster when each entails the other (bidirectional entailment, checked with an NLI model). Then the entropy is taken over clusters instead of strings:

$$
\mathrm{SE}(x)=-\sum_{c\in\mathcal{C}}p(c\mid x)\log p(c\mid x),\qquad p(c\mid x)=\sum_{s\in c}p(s\mid x)\;\approx\;\frac{|c|}{N}
$$

- $\mathcal{C}$: meaning clusters found among the N sampled answers
- $p(s\mid x)$: sequence probability of sample s; the discrete variant replaces it by counts and needs no logits

The *Nature* paper calls the errors this catches **confabulations**: claims that are both wrong and arbitrary, changing from sample to sample. It does not catch a model that gives the same wrong answer every time. **Probes** target that case. A linear classifier $\sigma(w^\top h_\ell)$ on one layer's hidden state is trained on answers labelled true or false (Azaria and Mitchell, 2023), or trained to predict semantic entropy from a single pass (semantic entropy probes, Kossen et al., 2024).

When a source exists (summaries, RAG answers), use NLI against it. Split the output into sentences or claims and ask an entailment model whether the retrieved or given text entails each one. Contradiction marks an intrinsic hallucination. "Neutral" marks an extrinsic one.

Detector families: what they need and what they cost per answer

| Detector | Signal | Access | Extra cost |
| --- | --- | --- | --- |
| Token probability | mean or max −log p of the answer | logits | none (same pass) |
| P(True) self-evaluation | model's probability that its answer is true | logits + a prompt | 1 pass |
| SelfCheckGPT | support of each sentence by N samples | black-box | N generations + checks |
| Semantic entropy | entropy over meaning clusters | black-box (+ an NLI model) | N generations + up to $O(N^2)$ NLI calls |
| NLI vs source | entail / neutral / contradict per claim | a source document | 1 NLI call per claim–passage pair |
| Hidden-state probe | $\sigma(w^\top h_\ell)$ | activations + labelled data | a dot product |

## Scoring the detector: AUROC

A detector outputs a score, not a verdict, so it is judged over a labelled set of correct and hallucinated answers by the area under the ROC curve:

$$
\mathrm{AUROC}=P\big(s(x^{+})>s(x^{-})\big)+\tfrac12\,P\big(s(x^{+})=s(x^{-})\big)
$$

- $x^{+},\,x^{-}$: a random hallucinated answer and a random correct one
- $0.5 / 1$: chance ranking / perfect separation; independent of any threshold

AUROC ignores the threshold and the base rate. In deployment you still choose $\tau$, and the cost of a false alarm (refusing a correct answer) sets it. The Nature paper also reports AURAC, the accuracy of the answers kept when the most uncertain ones are rejected.

> Figure: A toy QA model over 64 questions, 32 of them hallucinated. At the default N = 10 samples, semantic entropy reaches AUROC 0.882, SelfCheck disagreement 0.828, lexical (string) entropy 0.747 and −log p of the greedy answer 0.685. With the threshold at 50% (0.752 nats), semantic entropy catches 29 of the 32 hallucinations and raises 6 false alarms. On question 1 the ten samples are 3 strings with one meaning, so lexical entropy is 1.055 but semantic entropy is 0.500. At N = 2, semantic entropy can only be 0 or ln 2 and its AUROC falls to 0.625. At N = 20 it rises to 0.897.

## Worked example

Ten samples per question (natural log). The Curie answers are the viz's question 3: Paris, Warszawa, Warsaw, Kraków, Lublin, Kraków, Kraków, Lublin, Paris, Paris.

| Question | Sample counts | Lexical H | Semantic H | SelfCheck 1 − agree |
| --- | --- | --- | --- | --- |
| Capital of Australia (correct) | Canberra 4, Canberra. 4, Sydney 2 | $H(.4,.4,.2)=1.055$ | $H(.8,.2)=0.500$ | 1 − 8/10 = 0.20 |
| Author of The Master and Margarita (correct) | Bulgakov 4, M.Bulgakov 4, Bulgakov. 2 | 1.055 | 0 | 0 |
| Marie Curie's birthplace (greedy Kraków, wrong) | Kraków 3, Paris 3, Lublin 2, Warsaw 1, Warszawa 1 | $H(.3,.3,.2,.1,.1)=1.505$ | $H(.3,.3,.2,.2)=1.366$ | 1 − 3/10 = 0.70 |

Lexical entropy gives both correct questions the same 1.055, because rewordings look like disagreement. Semantic entropy gives 0.500, 0 and 1.366, which ranks the hallucination last. AUROC on a tiny set: hallucinated scores $(1.37, 0.95, 0.50)$, correct scores $(0.50, 0.30, 0)$. Of the 9 pairs, 8 are ordered correctly and 1 is tied, so $\mathrm{AUROC}=(8+0.5)/9=0.944$.

```python
import math
from transformers import pipeline
from sklearn.metrics import roc_auc_score

nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")

def entails(q, a, b):
    out = nli([{"text": f"{q} {a}", "text_pair": f"{q} {b}"}])[0]
    return out["label"] == "ENTAILMENT"

def semantic_entropy(q, answers):
    clusters = []                               # greedy clustering by two-way entailment
    for a in answers:
        for c in clusters:
            if entails(q, c[0], a) and entails(q, a, c[0]):
                c.append(a)
                break
        else:
            clusters.append([a])
    p = [len(c) / len(answers) for c in clusters]
    return -sum(pi * math.log(pi) for pi in p)

ans = ["Paris", "Warszawa", "Warsaw", "Kraków", "Lublin",
       "Kraków", "Kraków", "Lublin", "Paris", "Paris"]
print(semantic_entropy("Where was Marie Curie born?", ans))  # 1.366 if Warsaw = Warszawa

# over a labelled set: roc_auc_score(is_hallucinated, scores)
```

Discrete semantic entropy with an off-the-shelf MNLI model (sketch; not run here)

> **Consistent errors are invisible to sampling:** If the model answers "Kraków" in all 10 samples, semantic entropy is 0 and SelfCheck sees full agreement. Sampling methods detect *arbitrary* errors. Errors the model learned (a misconception repeated in the training data, the kind **TruthfulQA** targets) need a source, retrieval or a probe.

> **Cluster meanings, not strings:** Probability and entropy over strings mix two uncertainties: which fact, and which wording. Only the first predicts errors. Merging paraphrases with bidirectional entailment removes the second, which is why semantic entropy beats lexical entropy in the figure at every N.

**Strengths**

- Black-box methods (SelfCheckGPT, discrete semantic entropy) work through an API with no logits.
- Uncertainty scores need no labelled data and no reference text.
- Probes and token scores add almost no latency.

**Limitations**

- Sampling multiplies inference cost by about $N$, plus the NLI calls.
- Confident, repeated errors get low uncertainty scores.
- Probes need labelled data and access to activations, and may not transfer across domains or models.

**Question:** Ten samples read "Bulgakov" ×4, "M.Bulgakov" ×4, "Bulgakov." ×2. What are the lexical and the (discrete) semantic entropy?

- A. Both 0
- B. Lexical 1.055, semantic 0
- C. Lexical 0, semantic 1.055
- D. Both 1.055

**Answer:** B. Lexical 1.055, semantic 0 — Three distinct strings with frequencies $(0.4, 0.4, 0.2)$ give $-\sum p\log p=1.055$ nats. All three entail each other, so they form one cluster with $p=1$, and the entropy is $0$. String-level scores would wrongly flag this correct answer.

**Question:** A detector has AUROC 0.5 on a balanced test set. What does that mean?

- A. It flags half of all answers
- B. Its scores rank a random hallucinated answer above a random correct one only half the time, i.e. no better than chance
- C. Its precision is 50% at every threshold
- D. Half of its flags are hallucinations at the best threshold

**Answer:** B. Its scores rank a random hallucinated answer above a random correct one only half the time, i.e. no better than chance — AUROC is the probability $P(s(x^+)>s(x^-))$ (ties count one half), computed over all thresholds. 0.5 is a coin flip. The flag rate and precision depend on the threshold you pick, which AUROC does not fix.

**Key points**

- Intrinsic hallucinations contradict the source. Extrinsic ones cannot be checked against it, and they may still be true.
- Token score: $-\frac1T\sum\log p(y_t)$. It is cheap but confounded by wording.
- SelfCheckGPT: sample N responses and check whether they support each sentence. Black-box.
- Semantic entropy: cluster samples by bidirectional entailment, then $-\sum_c p(c)\log p(c)$. It catches arbitrary confabulations, not consistent errors.
- NLI against a source catches intrinsic errors. Hidden-state probes can catch errors without sampling.
- Detectors are compared by AUROC $=P(s(x^+)>s(x^-))$. The deployment threshold is set by the cost of false alarms.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
