Hallucination and groundedness evaluation

Split answers into atomic claims, verify each against the supplied context (NLI or LLM judge), and report claim support, the share of responses with any unsupported claim, and rates corrected for verifier error.

8 min read711 wordsadvanced0 viewsupdated 2026-10-21

Definition

Groundedness evaluation measures how much of a generated answer is supported by a given source: the retrieved passages in RAG, the document in summarisation, the tool output in an agent. The answer is split into atomic claims, a verifier labels each claim as entailed, contradicted or absent from the source, and the labels are aggregated. The decompose-then-verify recipe was popularised by FActScore (Min et al., EMNLP 2023) for facts against Wikipedia, and by the faithfulness metric of RAGAS (Es et al., 2023) for answers against retrieved context.

Hallucination explains why models produce unsupported text. This article is about measuring it in a system you ship. The question is narrower than "is it true?". It is "does the source the model was given back up what it said?". That makes the measurement checkable without world knowledge, and it is the property a RAG product promises: answers come from your documents.

Two failure types matter. An intrinsic hallucination contradicts the source. An extrinsic one adds information the source does not contain. Groundedness counts both as unsupported, even when an extrinsic claim happens to be true, because the user cannot verify it from the cited passages. World truthfulness is a separate measurement (see Factuality evaluation).

groundedness = share of the answer's claims that the retrieved context entailsretrieved context (refund policy)§1 Items can be returned within 30 days of delivery.§2 Refunds reach the original payment method in 5–7business days.§3 Sale items are final and cannot be returned.questionCan I return a sale itemI bought two weeks ago?generated answerYes: returns are accepted within 30 days and sale items qualify.The refund arrives in 5–7 business days; return shipping is free.1 · split into atomic claims → 2 · verify each against the context (NLI)claimevidencep(entail)verdict (τ = 0.5)c1 returns accepted within 30 days§10.97supportedc2 sale items can be returned§30.02contradictedc3 refund arrives in 5–7 business days§20.94supportedc4 return shipping is free–0.31not in contextclaim levelgroundedness 2/4 = 0.50intrinsic 1 (c2)extrinsic 1 (c4)response levelany claim < τ → flaggedhallucinated: yesmean p = 0.56 would passcorpus levelflag rate p̂ over all claimscorrect for verifierSe, Sp from human labelsintrinsic = contradicts the context · extrinsic = adds facts the context does not containboth count against groundedness, even when an extrinsic claim happens to be true
fig.One RAG answer about a refund policy, split into four claims and checked against the three retrieved sections by an NLI verifier. c2 contradicts §3 (intrinsic) and c4 has no support (extrinsic), so claim-level groundedness is 2/4 = 0.50 and the response counts as hallucinated. A whole-response score would hide this: the mean entailment probability is 0.56, above τ = 0.5.

Three levels of measurement

G=1∣C∣∑c∈C1[s(c,x)≥τ],Hresp=1∣R∣∑r∈R1[∃ c∈r: s(c,xr)<τ],E[Hresp]=1−(1−q)nG=\frac{1}{|C|}\sum_{c\in C}\mathbb 1\big[s(c,x)\ge\tau\big],\qquad H_{\text{resp}}=\frac{1}{|R|}\sum_{r\in R}\mathbb 1\big[\exists\,c\in r:\ s(c,x_r)<\tau\big],\qquad \mathbb E[H_{\text{resp}}]=1-(1-q)^{n}
C, RC,\ R
all claims, all responses in the eval set
s(c,x)s(c,x)
verifier support score of claim cc given source xx (e.g. NLI entailment probability)
τ\tau
threshold below which a claim is flagged unsupported
q, nq,\ n
per-claim unsupported rate and claims per response, if errors were independent

GG is the claim-level groundedness, the RAGAS-style faithfulness averaged over a set. HrespH_\text{resp} is what a user experiences: the share of answers that contain at least one unsupported claim. The two diverge quickly. At q=0.02q=0.02, a 10-claim answer is hallucinated 18.3% of the time and a 30-claim answer 45.5% of the time. Report both, together with the average number of claims, or long answers will look better per claim and worse per response.

Per-claim rate qn = 5n = 10n = 30
0.020.0960.1830.455
0.050.2260.4010.785
Response-level hallucination rate 1 − (1 − q)ⁿ for independent claim errors

Verifiers

VerifierHow it scores a claimStrengthWeakness
NLI cross-encoderp(entailment) of claim given each passage, max over passagescheap, local, deterministicmisses support spread over several sentences
LLM judgeprompted to label supported / contradicted / not found, citing a spanhandles paraphrase and multi-hopcostly; position and self-preference biases
QA-basedgenerate questions from the claim, answer them from the source, comparelocalises the errorslow; depends on question quality
Lexical overlapshare of claim tokens or n-grams found in the sourcenearly freerewards copying, misses negation
Ways to score s(c, x); every one of them must be validated against human labels

Long sources need chunking: SummaC (Laban et al., TACL 2022) scores every (document sentence, summary sentence) pair with NLI and aggregates the matrix, because NLI models were trained on short premises. Reference-free methods such as SelfCheckGPT need no source at all; they belong to Hallucination detection and measure model uncertainty, not groundedness. LLM-as-a-judge covers the biases of judge-based verifiers.

Benchmarks and methods

FActScore
Min et al., EMNLP 2023: atomic facts in generated biographies checked against Wikipedia; ChatGPT scored about 58% supported facts.
RAGAS faithfulness
Es et al., 2023: an LLM extracts statements from the answer and judges each against the retrieved context; score = supported / total.
SummaC
Laban et al., TACL 2022: sentence-level NLI for summary consistency, with a benchmark that unifies six consistency datasets.
TRUE
Honovich et al., NAACL 2022: 11 consistency datasets in one binary format, scored by ROC AUC; NLI- and QA-based metrics led.
TruthfulQA
Lin, Hilton and Evans, ACL 2022: 817 questions in 38 categories that test world truthfulness, not groundedness to a source.

Correcting for the verifier

A verifier with false positives and false negatives gives a biased rate. If it flags a fraction p^\hat p of claims, with sensitivity SeSe (unsupported claims caught) and specificity SpSp (supported claims passed), then E[p^]=Se π+(1−Sp)(1−π)\mathbb E[\hat p]=Se\,\pi+(1-Sp)(1-\pi). Solving for the true unsupported rate π\pi gives the Rogan–Gladen estimator (Rogan and Gladen, 1978), borrowed from epidemiology, where imperfect diagnostic tests raise the same problem:

π^=p^+Sp−1Se+Sp−1,G^=1−π^\hat\pi=\frac{\hat p+Sp-1}{Se+Sp-1},\qquad \hat G=1-\hat\pi
p^\hat p
fraction of claims the verifier flags
Se, SpSe,\ Sp
measured on a human-labelled calibration set at the same τ\tau
Se+Sp−1Se+Sp-1
Youden's J; near 0 the correction divides by almost nothing and amplifies errors
40 RAG responses · 197 claims · 28 unsupported · NLI verifiersupported claims ↑unsupported claims ↓00.250.50.751τ = 0.50← flagverifier support score (each class scaled to its tallest bin)claim-level scoringflagged claims41 = 27 true + 14 falsemissed unsupported1 of 28precision / recall0.659 / 0.964true groundedness0.858measured (1 − flag rate)0.792calibration Se / Sp0.940 / 0.940Rogan–Gladen corrected0.832responses, flagged if any claim scores < τtruly hallucinated 20/40 · flagged 26 · caught 20 · false alarms 6 · missed 0fill = truth (has an unsupported claim / fully grounded), blue outline = flaggedresponse hallucination rate 0.500 vs claim rate 0.142: one bad claim ina ~5-claim answer makes the whole answer count; averaging over claims hides it.
verifier
granularity
fig.40 synthetic RAG responses with 197 claims, 28 unsupported (true groundedness 0.858). Defaults (NLI verifier, τ = 0.50): 41 claims flagged (27 true, 14 false; precision 0.659, recall 0.964), measured groundedness 0.792; Se = Sp = 0.940 from a 200-claim calibration set give a corrected 0.832. At response level 20 of 40 answers are truly hallucinated; flagging on any claim catches all 20 with 6 false alarms, while a whole-response mean flags only 4. The lexical verifier at τ = 0.50 has precision 0.301 and measures 0.629; with Se + Sp − 1 = 0.50 the correction reaches only 0.679.

The threshold moves the measured number as much as the system does. With the NLI verifier at τ=0.40\tau=0.40 the measured rate equals the truth (0.858) only because 4 false alarms cancel 4 misses. At τ=0.60\tau=0.60 it drops to 0.685. Fix τ\tau before comparing systems, and use the same verifier version for every run.

Worked example: correcting the default run

  1. 1Flag rate: p^=41/197=0.208\hat p=41/197=0.208, so the naive groundedness is 1−0.208=0.7921-0.208=0.792.
  2. 2Calibration on 100 supported and 100 unsupported human-labelled claims: Se=Sp=0.94Se=Sp=0.94, so Se+Sp−1=0.88Se+Sp-1=0.88.
  3. 3Correct: π^=(0.208+0.94−1)/0.88=0.148/0.88=0.168\hat\pi=(0.208+0.94-1)/0.88=0.148/0.88=0.168, so G^=0.832\hat G=0.832.
  4. 4Compare: the truth is 0.8580.858. With the Se and Sp realised on these 197 claims (27/28=0.96427/28=0.964 and 155/169=0.917155/169=0.917) the formula returns exactly π=0.142\pi=0.142. The remaining gap is calibration-sample noise.
  5. 5Sampling error alone: 1.960.858⋅0.142/197=±0.0491.96\sqrt{0.858\cdot0.142/197}=\pm0.049. Two systems 0.03 apart on 200 claims are not distinguishable.
python
1import numpy as np
2
3# verifier support scores per claim, grouped by response (NLI entailment prob or judge score)
4responses = [[0.97, 0.02, 0.94, 0.31], [0.91, 0.88], [0.95, 0.72, 0.99]]
5tau = 0.5
6
7flags = [np.array(r) < tau for r in responses]
8claims = np.concatenate(flags)
9groundedness = 1 - claims.mean() # claim level
10halluc_rate = np.mean([f.any() for f in flags]) # response level
11
12# verifier sensitivity / specificity at this tau, from human-labelled claims
13se, sp = 0.94, 0.94
14p_hat = claims.mean()
15pi = np.clip((p_hat + sp - 1) / (se + sp - 1), 0, 1) # Rogan–Gladen
16print(f"groundedness {groundedness:.3f} hallucinated responses {halluc_rate:.3f} corrected {1 - pi:.3f}")
17# groundedness 0.778 hallucinated responses 0.333 corrected 0.816
Claim-, response- and corrected corpus-level scores from per-claim verifier scores (runs as is; the scores would come from an NLI model or judge)

Check yourself

A verifier with Se = 0.70 and Sp = 0.80 flags 30% of claims. What is the Rogan–Gladen estimate of the unsupported rate?

Check yourself

The answer says "return shipping is free". That is true on the company website but absent from the retrieved context. How should groundedness treat it?

Remember

  • ✓Groundedness asks whether the given source supports each claim, not whether the claim is true in the world.
  • ✓Decompose into atomic claims, verify each (NLI, LLM judge, QA), then aggregate.
  • ✓Report claim-level GG and response-level rate together: 1−(1−q)n1-(1-q)^n grows fast with answer length.
  • ✓Flag on any unsupported claim; averaging scores over a response hides single errors.
  • ✓Correct raw flag rates with the verifier's SeSe and SpSp: π^=(p^+Sp−1)/(Se+Sp−1)\hat\pi=(\hat p+Sp-1)/(Se+Sp-1); weak verifiers make the correction unstable.
  • ✓Freeze decomposer, verifier and threshold, and pair the metric with retrieval recall and answer rate.
← back to Generative AI roadmap