# Hallucination and groundedness evaluation

> Split answers into atomic claims, verify each against the supplied context (NLI or LLM judge), and report claim support, the share of responses with any unsupported claim, and rates corrected for verifier error.

Source: https://www.mlfoundry.in/learn/hallucination-and-groundedness-evaluation  
Section: Generative AI › Evaluation of Generative Models  
Level: advanced · 8 min read · updated 2026-10-21

**Groundedness evaluation:** measures how much of a generated answer is supported by a given source: the retrieved passages in RAG, the document in summarisation, the tool output in an agent. The answer is split into atomic claims, a verifier labels each claim as entailed, contradicted or absent from the source, and the labels are aggregated. The decompose-then-verify recipe was popularised by FActScore (Min et al., EMNLP 2023) for facts against Wikipedia, and by the faithfulness metric of RAGAS (Es et al., 2023) for answers against retrieved context.

**Hallucination** explains why models produce unsupported text. This article is about measuring it in a system you ship. The question is narrower than "is it true?". It is "does the source the model was given back up what it said?". That makes the measurement checkable without world knowledge, and it is the property a RAG product promises: answers come from your documents.

Two failure types matter. An **intrinsic** hallucination contradicts the source. An **extrinsic** one adds information the source does not contain. Groundedness counts both as unsupported, even when an extrinsic claim happens to be true, because the user cannot verify it from the cited passages. World truthfulness is a separate measurement (see **Factuality evaluation**).

> Figure: One RAG answer about a refund policy, split into four claims and checked against the three retrieved sections by an NLI verifier. c2 contradicts §3 (intrinsic) and c4 has no support (extrinsic), so claim-level groundedness is 2/4 = 0.50 and the response counts as hallucinated. A whole-response score would hide this: the mean entailment probability is 0.56, above τ = 0.5.

## Three levels of measurement

$$
G=\frac{1}{|C|}\sum_{c\in C}\mathbb 1\big[s(c,x)\ge\tau\big],\qquad H_{\text{resp}}=\frac{1}{|R|}\sum_{r\in R}\mathbb 1\big[\exists\,c\in r:\ s(c,x_r)<\tau\big],\qquad \mathbb E[H_{\text{resp}}]=1-(1-q)^{n}
$$

- $C,\ R$: all claims, all responses in the eval set
- $s(c,x)$: verifier support score of claim $c$ given source $x$ (e.g. NLI entailment probability)
- $\tau$: threshold below which a claim is flagged unsupported
- $q,\ n$: per-claim unsupported rate and claims per response, if errors were independent

$G$ is the claim-level groundedness, the RAGAS-style faithfulness averaged over a set. $H_\text{resp}$ is what a user experiences: the share of answers that contain at least one unsupported claim. The two diverge quickly. At $q=0.02$, a 10-claim answer is hallucinated 18.3% of the time and a 30-claim answer 45.5% of the time. Report both, together with the average number of claims, or long answers will look better per claim and worse per response.

Response-level hallucination rate 1 − (1 − q)ⁿ for independent claim errors

| Per-claim rate q | n = 5 | n = 10 | n = 30 |
| --- | --- | --- | --- |
| 0.02 | 0.096 | 0.183 | 0.455 |
| 0.05 | 0.226 | 0.401 | 0.785 |

## Verifiers

Ways to score s(c, x); every one of them must be validated against human labels

| Verifier | How it scores a claim | Strength | Weakness |
| --- | --- | --- | --- |
| NLI cross-encoder | p(entailment) of claim given each passage, max over passages | cheap, local, deterministic | misses support spread over several sentences |
| LLM judge | prompted to label supported / contradicted / not found, citing a span | handles paraphrase and multi-hop | costly; position and self-preference biases |
| QA-based | generate questions from the claim, answer them from the source, compare | localises the error | slow; depends on question quality |
| Lexical overlap | share of claim tokens or n-grams found in the source | nearly free | rewards copying, misses negation |

Long sources need chunking: SummaC (Laban et al., TACL 2022) scores every (document sentence, summary sentence) pair with NLI and aggregates the matrix, because NLI models were trained on short premises. Reference-free methods such as SelfCheckGPT need no source at all; they belong to **Hallucination detection** and measure model uncertainty, not groundedness. **LLM-as-a-judge** covers the biases of judge-based verifiers.

**Benchmarks and methods**

- **FActScore:** Min et al., EMNLP 2023: atomic facts in generated biographies checked against Wikipedia; ChatGPT scored about 58% supported facts.
- **RAGAS faithfulness:** Es et al., 2023: an LLM extracts statements from the answer and judges each against the retrieved context; score = supported / total.
- **SummaC:** Laban et al., TACL 2022: sentence-level NLI for summary consistency, with a benchmark that unifies six consistency datasets.
- **TRUE:** Honovich et al., NAACL 2022: 11 consistency datasets in one binary format, scored by ROC AUC; NLI- and QA-based metrics led.
- **TruthfulQA:** Lin, Hilton and Evans, ACL 2022: 817 questions in 38 categories that test world truthfulness, not groundedness to a source.

## Correcting for the verifier

A verifier with false positives and false negatives gives a biased rate. If it flags a fraction $\hat p$ of claims, with sensitivity $Se$ (unsupported claims caught) and specificity $Sp$ (supported claims passed), then $\mathbb E[\hat p]=Se\,\pi+(1-Sp)(1-\pi)$. Solving for the true unsupported rate $\pi$ gives the Rogan–Gladen estimator (Rogan and Gladen, 1978), borrowed from epidemiology, where imperfect diagnostic tests raise the same problem:

$$
\hat\pi=\frac{\hat p+Sp-1}{Se+Sp-1},\qquad \hat G=1-\hat\pi
$$

- $\hat p$: fraction of claims the verifier flags
- $Se,\ Sp$: measured on a human-labelled calibration set at the same $\tau$
- $Se+Sp-1$: Youden's J; near 0 the correction divides by almost nothing and amplifies errors

> Figure: 40 synthetic RAG responses with 197 claims, 28 unsupported (true groundedness 0.858). Defaults (NLI verifier, τ = 0.50): 41 claims flagged (27 true, 14 false; precision 0.659, recall 0.964), measured groundedness 0.792; Se = Sp = 0.940 from a 200-claim calibration set give a corrected 0.832. At response level 20 of 40 answers are truly hallucinated; flagging on any claim catches all 20 with 6 false alarms, while a whole-response mean flags only 4. The lexical verifier at τ = 0.50 has precision 0.301 and measures 0.629; with Se + Sp − 1 = 0.50 the correction reaches only 0.679.

The threshold moves the measured number as much as the system does. With the NLI verifier at $\tau=0.40$ the measured rate equals the truth (0.858) only because 4 false alarms cancel 4 misses. At $\tau=0.60$ it drops to 0.685. Fix $\tau$ before comparing systems, and use the same verifier version for every run.

## Worked example: correcting the default run

1. Flag rate: $\hat p=41/197=0.208$, so the naive groundedness is $1-0.208=0.792$.
2. Calibration on 100 supported and 100 unsupported human-labelled claims: $Se=Sp=0.94$, so $Se+Sp-1=0.88$.
3. Correct: $\hat\pi=(0.208+0.94-1)/0.88=0.148/0.88=0.168$, so $\hat G=0.832$.
4. Compare: the truth is $0.858$. With the Se and Sp realised on these 197 claims ($27/28=0.964$ and $155/169=0.917$) the formula returns exactly $\pi=0.142$. The remaining gap is calibration-sample noise.
5. Sampling error alone: $1.96\sqrt{0.858\cdot0.142/197}=\pm0.049$. Two systems 0.03 apart on 200 claims are not distinguishable.

```python
import numpy as np

# verifier support scores per claim, grouped by response (NLI entailment prob or judge score)
responses = [[0.97, 0.02, 0.94, 0.31], [0.91, 0.88], [0.95, 0.72, 0.99]]
tau = 0.5

flags = [np.array(r) < tau for r in responses]
claims = np.concatenate(flags)
groundedness = 1 - claims.mean()                      # claim level
halluc_rate = np.mean([f.any() for f in flags])       # response level

# verifier sensitivity / specificity at this tau, from human-labelled claims
se, sp = 0.94, 0.94
p_hat = claims.mean()
pi = np.clip((p_hat + sp - 1) / (se + sp - 1), 0, 1)  # Rogan–Gladen
print(f"groundedness {groundedness:.3f}  hallucinated responses {halluc_rate:.3f}  corrected {1 - pi:.3f}")
# groundedness 0.778  hallucinated responses 0.333  corrected 0.816
```

Claim-, response- and corrected corpus-level scores from per-claim verifier scores (runs as is; the scores would come from an NLI model or judge)

> **Grounded is not the same as correct:** A perfectly grounded answer can be wrong if retrieval returned an outdated or wrong passage, and "I don't know" is vacuously grounded. Report groundedness next to retrieval recall (see **RAG evaluation**) and the answer rate. Otherwise a system that refuses more often, or copies more, wins the metric.

> **Decomposition is part of the metric:** The number depends on how claims are cut. Splitting "returns within 30 days and sale items qualify" into one claim or two changes $G$, and an LLM decomposer can drop or merge claims. Freeze the decomposer prompt and model along with the verifier, and spot-check that every claim in a sample actually appears in the answer.

**Question:** A verifier with Se = 0.70 and Sp = 0.80 flags 30% of claims. What is the Rogan–Gladen estimate of the unsupported rate?

- A. 0.10
- B. 0.20
- C. 0.30
- D. 0.43

**Answer:** B. 0.20 — $\hat\pi=(0.30+0.80-1)/(0.70+0.80-1)=0.10/0.50=0.20$. A third of the raw flags are false alarms on supported claims. With Youden's J only 0.5, small errors in Se or Sp double in the estimate.

**Question:** The answer says "return shipping is free". That is true on the company website but absent from the retrieved context. How should groundedness treat it?

- A. Supported, because it is true
- B. Unsupported (extrinsic): the source does not back it
- C. Contradicted (intrinsic)
- D. Ignored, because it is not about the question

**Answer:** B. Unsupported (extrinsic): the source does not back it — Groundedness is relative to the supplied source. The claim is extrinsic: not contradicted, but not supported either. The user cannot check it in the citations, and the next time the model makes such a claim it may be false.

**Key points**

- Groundedness asks whether the given source supports each claim, not whether the claim is true in the world.
- Decompose into atomic claims, verify each (NLI, LLM judge, QA), then aggregate.
- Report claim-level $G$ and response-level rate together: $1-(1-q)^n$ grows fast with answer length.
- Flag on any unsupported claim; averaging scores over a response hides single errors.
- Correct raw flag rates with the verifier's $Se$ and $Sp$: $\hat\pi=(\hat p+Sp-1)/(Se+Sp-1)$; weak verifiers make the correction unstable.
- Freeze decomposer, verifier and threshold, and pair the metric with retrieval recall and answer rate.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
