# RAG evaluation

> Score retrieval and generation separately: recall@k, MRR and nDCG against labelled gold chunks, then faithfulness and answer relevance with an LLM judge (RAGAS), on a versioned eval set big enough that confidence intervals separate the systems.

Source: https://www.mlfoundry.in/learn/rag-evaluation  
Section: Generative AI › Retrieval-Augmented Generation (RAG)  
Level: core · 8 min read · updated 2026-10-21

**RAG evaluation:** measures a retrieval-augmented system stage by stage. **Retrieval** is scored with information-retrieval metrics (recall@k, MRR, nDCG) against human- or LLM-labelled relevant chunks. **Generation** is scored for faithfulness to the retrieved context and relevance to the question, usually by an LLM judge. The best-known framework is **RAGAS** (Es, James, Espinosa-Anke and Schockaert, 2023; EACL 2024 demo track); **ARES** (Saad-Falcon, Khattab, Potts and Zaharia, NAACL 2024) adds trained judges and confidence intervals.

A RAG answer can fail in two places, and the fixes are different. If the right chunk never reached the prompt, no prompt or model change helps; you fix chunking, retrieval or reranking. If it did reach the prompt and the answer still contradicts it, you fix the prompt or the model. One end-to-end accuracy number cannot tell these apart, so every serious RAG eval scores both stages on the same items.

The pipeline itself is described in **RAG architecture**; the component articles (**Document chunking**, **Hybrid search (BM25 + dense)**, **Reranking**) each move one of the metrics below. Claim-level hallucination metrics get their own article, **Hallucination and groundedness evaluation**.

> Figure: One eval-set item holds a question, graded gold chunks and a reference answer. The system retrieves c1, c2, c5 and answers. Retrieval metrics compare the ranked ids with the gold labels (recall@3 = 2/4, nDCG@3 = 0.947). Generation metrics compare the answer with the context and the question. Reading the two together says which stage to fix.

## Retrieval metrics

$$
\text{recall@}k=\frac{|\mathrm{Rel}\cap\mathrm{Top}_k|}{|\mathrm{Rel}|},\qquad \text{MRR@}k=\frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\operatorname{rank}_q},\qquad \text{nDCG@}k=\frac{\sum_{i=1}^{k}\frac{2^{g_i}-1}{\log_2(i+1)}}{\text{IDCG@}k}
$$

- $\mathrm{Rel}$: the chunks labelled relevant for the query (the qrels)
- $\operatorname{rank}_q$: position of the first relevant chunk; the term is 0 if it is below $k$
- $g_i$: graded relevance of the chunk at position i (0–3 here); IDCG is the same sum for the ideal order, so nDCG ∈ [0, 1]

Each answers a different question. **Recall@k** asks whether the evidence reached the prompt at all, and it caps the answer quality, so it is the metric to watch for a fixed $k$. **MRR** asks how high the first good chunk is, which matters when only the top one or two are used. **nDCG** (Järvelin and Kekäläinen, 2002) rewards putting the *most* relevant chunks first; it is the standard metric for graded judgments. Precision@k matters less for RAG than for search, because the LLM can ignore an irrelevant chunk, though too many distractors do hurt.

## Generation metrics

$$
\text{Faithfulness}=\frac{|V|}{|S|},\qquad \text{AnswerRelevance}=\frac{1}{n}\sum_{i=1}^{n}\cos\!\big(E(q),\,E(\hat q_i)\big)
$$

- $S,\;V$: statements an LLM extracts from the answer; the subset it judges supported by the retrieved context
- $\hat q_i$: questions an LLM generates from the answer alone (n of them); E is an embedding model

These are the RAGAS definitions. Faithfulness is a groundedness check: it says nothing about whether the context was right. Answer relevance penalises evasive or padded answers, but not wrong ones, because a confident false answer still regenerates the original question. RAGAS also defined **context relevance** (the share of context sentences needed to answer). The library later added **context precision** and **context recall**, which compare the context with a reference answer and so need labels.

**Reference points**

- **RAGAS:** Es et al., 2023. Reference-free: faithfulness, answer relevance and context relevance computed with LLM prompts; validated against human judgments on WikiEval, a set built from Wikipedia pages on events after the models' training data
- **ARES:** Saad-Falcon et al., NAACL 2024. Fine-tunes small judges on synthetic data for context relevance, faithfulness and answer relevance, then combines them with a small human-labelled set through prediction-powered inference to give confidence intervals
- **BEIR:** Thakur et al., NeurIPS 2021 Datasets and Benchmarks. 18 datasets for zero-shot retrieval, reported with nDCG@10
- **MS MARCO:** The passage-ranking task's official metric is MRR@10
- **Pooling:** TREC practice: label the union of the top results from several systems, so a new system is not penalised for finding unlabelled relevant chunks (it still can be, if it finds chunks no pooled system found)

> Figure: Default (hybrid + rerank, k = 3, 50 queries, grounded answer): the worked query scores recall@3 0.500, RR 1.000, nDCG@3 0.947; BM25 gets 0.500, 0.500, 0.440. Over 50 queries nDCG@3 is 0.386 ± 0.104 vs 0.531 ± 0.105, and the paired difference 0.145 ± 0.139 barely excludes 0. With 20 queries it is 0.062 ± 0.253, not significant; with 200, 0.141 ± 0.068. The hallucinated answer has faithfulness 0.000 but answer relevance 0.688, above the grounded answer's 0.648. The evasive answer scores relevance 0.180 and reaches faithfulness 1.000 only at k = 6.

## Worked example: one query, two retrievers

Gold grades c1 = 3, c2 = 2, c3 = 1, c4 = 1, so IDCG@3 = 7 + 3/log₂3 + 1/2 = 9.393. Checked in Python.

| Retriever | Top 3 (grades) | recall@3 | RR@3 | DCG@3 | nDCG@3 |
| --- | --- | --- | --- | --- | --- |
| BM25 | c6 (0), c4 (1), c1 (3) | 2/4 = 0.50 | 1/2 = 0.50 | 0 + 1/1.585 + 7/2 = 4.131 | 0.440 |
| hybrid + rerank | c1 (3), c2 (2), c5 (0) | 2/4 = 0.50 | 1/1 = 1.00 | 7 + 3/1.585 + 0 = 8.893 | 0.947 |

Recall cannot tell these retrievers apart: both put two relevant chunks in the top 3. nDCG can, because the hybrid puts the grade-3 chunk first. Which number matters depends on the generator. If the model reads all three chunks equally well, recall is what counts. If it attends mostly to the first chunk, or you pass only one, rank matters.

## Building the eval set

1. **Sample real queries** from logs (or from domain experts before launch), stratified by type: single-fact, multi-hop, comparison, unanswerable from the corpus. Unanswerable items test abstention.
2. **Label relevance** by pooling the top 10–20 chunks from several retrievers and grading each (0–3). An LLM judge can pre-label; humans check a sample and the disagreements.
3. **Add synthetic items** where logs are thin: give an LLM one chunk, ask for a question it answers, keep that chunk as gold, and filter out questions that copy the chunk's wording (they flatter BM25).
4. **Write reference answers** for correctness and context-recall metrics; keep them short and factual.
5. **Size it** so the confidence interval is smaller than the differences you care about: the half-width is about $1.96\,s/\sqrt{n}$. Compare systems with a *paired* difference on the same queries, which has a smaller variance than two separate means.
6. **Version and refresh**: freeze each release of the set, store run outputs, and re-label when the corpus changes, since gold chunk ids go stale after re-chunking.

```python
import math

def recall_at_k(ranked, gold, k):
    return len(set(ranked[:k]) & set(gold)) / len(gold)

def rr_at_k(ranked, gold, k):
    return next((1 / (i + 1) for i, d in enumerate(ranked[:k]) if d in gold), 0.0)

def ndcg_at_k(ranked, grades, k):          # grades: {chunk_id: 0..3}
    dcg = sum((2 ** grades.get(d, 0) - 1) / math.log2(i + 2) for i, d in enumerate(ranked[:k]))
    ideal = sorted(grades.values(), reverse=True)[:k]
    idcg = sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(ideal))
    return dcg / idcg

grades = {"c1": 3, "c2": 2, "c3": 1, "c4": 1}
runs = {"bm25":   ["c6", "c4", "c1", "c2", "c5", "c8", "c3", "c7"],
        "hybrid": ["c1", "c2", "c5", "c3", "c6", "c4", "c7", "c8"]}
for name, ranked in runs.items():
    print(name, recall_at_k(ranked, grades, 3), rr_at_k(ranked, grades, 3),
          round(ndcg_at_k(ranked, grades, 3), 3))

def faithfulness(answer, context, llm):
    claims = llm(f"List each factual statement in:\n{answer}").splitlines()
    ok = [llm(f"Context:\n{context}\nIs this supported? yes/no: {c}") == "yes" for c in claims]
    return sum(ok) / len(claims)
```

Retrieval metrics on the worked query (run: prints bm25 0.5 0.5 0.44 and hybrid 0.5 1.0 0.947), plus the shape of an LLM-judge faithfulness check (llm is a placeholder; that part not run)

> **The judge is a model too:** LLM-judged faithfulness and relevance inherit the judge's errors: leniency, position and length bias, and drift when the judge model is updated. Pin the judge model and prompt, report them with the scores, and check agreement with human labels on a sample before trusting a difference (see **LLM-as-a-judge**). In the figure, answer relevance ranks a hallucinated answer above a grounded one, so it must never be read alone.

> **Paired comparisons need fewer queries:** Two systems run on the same queries make correlated errors. Testing the per-query difference cancels the shared difficulty: at 50 queries the separate nDCG@3 intervals are ±0.104 and ±0.105 and overlap, yet the paired difference 0.145 ± 0.139 still excludes zero.

- **Retriever and chunking choices:** Recall@k and nDCG@10 on a fixed labelled set decide chunk size, hybrid weights, k and whether a reranker pays for its latency.
- **Release gating:** Run the frozen eval set in CI; block a deploy when faithfulness or recall drops by more than the confidence interval.
- **Production monitoring:** Reference-free metrics (faithfulness, answer relevance) can run on a sample of live traffic, where no gold labels exist.

**Question:** Retrieval recall@5 is 0.95 but many answers are wrong, and faithfulness is 0.60. Where should you look first?

- A. Chunking, since recall is too low
- B. The generation stage: prompt, model, context ordering
- C. The embedding model
- D. The eval set is too small; nothing can be concluded

**Answer:** B. The generation stage: prompt, model, context ordering — The evidence reaches the prompt 95% of the time, yet 40% of claims are not supported by it. That is the 'gold chunk in top-k, answer bad' cell: a generation problem. Retrieval changes would barely move it.

**Question:** Ranking A puts grades (0, 1, 3) in its top 3, ranking B puts (3, 2, 0). Gold grades are 3, 2, 1, 1. Which statement is right?

- A. A has higher recall@3
- B. B has higher recall@3
- C. Equal recall@3; B has higher nDCG@3 (0.947 vs 0.440)
- D. Equal recall@3 and equal nDCG@3

**Answer:** C. Equal recall@3; B has higher nDCG@3 (0.947 vs 0.440) — Both contain two of the four relevant chunks, so recall@3 is 0.5 for both. B's DCG is $7+3/\log_2 3=8.893$ and A's is $1/\log_2 3+7/2=4.131$; dividing by IDCG $9.393$ gives 0.947 and 0.440.

**Key points**

- Evaluate retrieval and generation separately on the same items; the 2×2 of retrieval hit/miss × answer good/bad says which stage to fix.
- Retrieval: recall@k (did evidence arrive), MRR (rank of first hit), nDCG@k $=\text{DCG}/\text{IDCG}$ with gain $2^g-1$ (graded order).
- RAGAS: faithfulness $=|V|/|S|$ supported statements; answer relevance $=$ mean cosine between the question and questions regenerated from the answer.
- Faithfulness ignores whether the context was right; answer relevance ignores whether the answer is true. Read them together.
- Eval sets: real stratified queries, pooled graded labels, synthetic items with care, unanswerable cases; size by the $1.96\,s/\sqrt n$ half-width and compare systems paired.
- Pin and validate the judge; version the eval set and re-label after re-chunking.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
