Definition
Groundedness evaluation measures how much of a generated answer is supported by a given source: the retrieved passages in RAG, the document in summarisation, the tool output in an agent. The answer is split into atomic claims, a verifier labels each claim as entailed, contradicted or absent from the source, and the labels are aggregated. The decompose-then-verify recipe was popularised by FActScore (Min et al., EMNLP 2023) for facts against Wikipedia, and by the faithfulness metric of RAGAS (Es et al., 2023) for answers against retrieved context.
Hallucination explains why models produce unsupported text. This article is about measuring it in a system you ship. The question is narrower than "is it true?". It is "does the source the model was given back up what it said?". That makes the measurement checkable without world knowledge, and it is the property a RAG product promises: answers come from your documents.
Two failure types matter. An intrinsic hallucination contradicts the source. An extrinsic one adds information the source does not contain. Groundedness counts both as unsupported, even when an extrinsic claim happens to be true, because the user cannot verify it from the cited passages. World truthfulness is a separate measurement (see Factuality evaluation).
Three levels of measurement
- all claims, all responses in the eval set
- verifier support score of claim given source (e.g. NLI entailment probability)
- threshold below which a claim is flagged unsupported
- per-claim unsupported rate and claims per response, if errors were independent
is the claim-level groundedness, the RAGAS-style faithfulness averaged over a set. is what a user experiences: the share of answers that contain at least one unsupported claim. The two diverge quickly. At , a 10-claim answer is hallucinated 18.3% of the time and a 30-claim answer 45.5% of the time. Report both, together with the average number of claims, or long answers will look better per claim and worse per response.
| Per-claim rate q | n = 5 | n = 10 | n = 30 |
|---|---|---|---|
| 0.02 | 0.096 | 0.183 | 0.455 |
| 0.05 | 0.226 | 0.401 | 0.785 |
Verifiers
| Verifier | How it scores a claim | Strength | Weakness |
|---|---|---|---|
| NLI cross-encoder | p(entailment) of claim given each passage, max over passages | cheap, local, deterministic | misses support spread over several sentences |
| LLM judge | prompted to label supported / contradicted / not found, citing a span | handles paraphrase and multi-hop | costly; position and self-preference biases |
| QA-based | generate questions from the claim, answer them from the source, compare | localises the error | slow; depends on question quality |
| Lexical overlap | share of claim tokens or n-grams found in the source | nearly free | rewards copying, misses negation |
Long sources need chunking: SummaC (Laban et al., TACL 2022) scores every (document sentence, summary sentence) pair with NLI and aggregates the matrix, because NLI models were trained on short premises. Reference-free methods such as SelfCheckGPT need no source at all; they belong to Hallucination detection and measure model uncertainty, not groundedness. LLM-as-a-judge covers the biases of judge-based verifiers.
Benchmarks and methods
- FActScore
- Min et al., EMNLP 2023: atomic facts in generated biographies checked against Wikipedia; ChatGPT scored about 58% supported facts.
- RAGAS faithfulness
- Es et al., 2023: an LLM extracts statements from the answer and judges each against the retrieved context; score = supported / total.
- SummaC
- Laban et al., TACL 2022: sentence-level NLI for summary consistency, with a benchmark that unifies six consistency datasets.
- TRUE
- Honovich et al., NAACL 2022: 11 consistency datasets in one binary format, scored by ROC AUC; NLI- and QA-based metrics led.
- TruthfulQA
- Lin, Hilton and Evans, ACL 2022: 817 questions in 38 categories that test world truthfulness, not groundedness to a source.
Correcting for the verifier
A verifier with false positives and false negatives gives a biased rate. If it flags a fraction of claims, with sensitivity (unsupported claims caught) and specificity (supported claims passed), then . Solving for the true unsupported rate gives the Rogan–Gladen estimator (Rogan and Gladen, 1978), borrowed from epidemiology, where imperfect diagnostic tests raise the same problem:
- fraction of claims the verifier flags
- measured on a human-labelled calibration set at the same
- Youden's J; near 0 the correction divides by almost nothing and amplifies errors
The threshold moves the measured number as much as the system does. With the NLI verifier at the measured rate equals the truth (0.858) only because 4 false alarms cancel 4 misses. At it drops to 0.685. Fix before comparing systems, and use the same verifier version for every run.
Worked example: correcting the default run
- 1Flag rate: , so the naive groundedness is .
- 2Calibration on 100 supported and 100 unsupported human-labelled claims: , so .
- 3Correct: , so .
- 4Compare: the truth is . With the Se and Sp realised on these 197 claims ( and ) the formula returns exactly . The remaining gap is calibration-sample noise.
- 5Sampling error alone: . Two systems 0.03 apart on 200 claims are not distinguishable.
1import numpy as np23# verifier support scores per claim, grouped by response (NLI entailment prob or judge score)4responses = [[0.97, 0.02, 0.94, 0.31], [0.91, 0.88], [0.95, 0.72, 0.99]]5tau = 0.567flags = [np.array(r) < tau for r in responses]8claims = np.concatenate(flags)9groundedness = 1 - claims.mean() # claim level10halluc_rate = np.mean([f.any() for f in flags]) # response level1112# verifier sensitivity / specificity at this tau, from human-labelled claims13se, sp = 0.94, 0.9414p_hat = claims.mean()15pi = np.clip((p_hat + sp - 1) / (se + sp - 1), 0, 1) # Rogan–Gladen16print(f"groundedness {groundedness:.3f} hallucinated responses {halluc_rate:.3f} corrected {1 - pi:.3f}")17# groundedness 0.778 hallucinated responses 0.333 corrected 0.816Check yourself
A verifier with Se = 0.70 and Sp = 0.80 flags 30% of claims. What is the Rogan–Gladen estimate of the unsupported rate?
Check yourself
The answer says "return shipping is free". That is true on the company website but absent from the retrieved context. How should groundedness treat it?
Remember
- ✓Groundedness asks whether the given source supports each claim, not whether the claim is true in the world.
- ✓Decompose into atomic claims, verify each (NLI, LLM judge, QA), then aggregate.
- ✓Report claim-level and response-level rate together: grows fast with answer length.
- ✓Flag on any unsupported claim; averaging scores over a response hides single errors.
- ✓Correct raw flag rates with the verifier's and : ; weak verifiers make the correction unstable.
- ✓Freeze decomposer, verifier and threshold, and pair the metric with retrieval recall and answer rate.