RAG evaluation

Score retrieval and generation separately: recall@k, MRR and nDCG against labelled gold chunks, then faithfulness and answer relevance with an LLM judge (RAGAS), on a versioned eval set big enough that confidence intervals separate the systems.

8 min read798 wordscore0 viewsupdated 2026-10-21

Definition

RAG evaluation measures a retrieval-augmented system stage by stage. Retrieval is scored with information-retrieval metrics (recall@k, MRR, nDCG) against human- or LLM-labelled relevant chunks. Generation is scored for faithfulness to the retrieved context and relevance to the question, usually by an LLM judge. The best-known framework is RAGAS (Es, James, Espinosa-Anke and Schockaert, 2023; EACL 2024 demo track); ARES (Saad-Falcon, Khattab, Potts and Zaharia, NAACL 2024) adds trained judges and confidence intervals.

A RAG answer can fail in two places, and the fixes are different. If the right chunk never reached the prompt, no prompt or model change helps; you fix chunking, retrieval or reranking. If it did reach the prompt and the answer still contradicts it, you fix the prompt or the model. One end-to-end accuracy number cannot tell these apart, so every serious RAG eval scores both stages on the same items.

The pipeline itself is described in RAG architecture; the component articles (Document chunking, Hybrid search (BM25 + dense), Reranking) each move one of the metrics below. Claim-level hallucination metrics get their own article, Hallucination and groundedness evaluation.

eval-set item (1 of n)q: How long is the X200 warranty?gold: c1 g3 · c2 g2 · c3 g1 · c4 g1ref: 2 years from purchasesystem under testretrieve top-3: c1 c2 c5generate: “2-year warranty frompurchase; file claims online.”retrieval: ranked list vs gold idsrecall@3 = 2/4 = 0.50MRR: 1/rank of first gold = 1/1nDCG@3 = 8.893/9.393 = 0.947precision@3 = 2/3 = 0.67no LLM needed; cheap, exactgeneration: answer vs context, qfaithfulness = supported claims/ all claims = 3/3 (vs context)answer relevance = mean cos(q, q̂ᵢ)context precision / recall (vs ref)needs an LLM judge (RAGAS)read both together: which stage failed?answer goodanswer badgold chunkin top-kgold chunkmissedanswer correct, faithful→ working as intendedanswer wrong or unfaithful→ generation problem:prompt, model, long contextanswer correct anyway→ parametric memory ora gap in the gold labelsanswer wrong→ retrieval problem:chunking, k, hybrid, rerank
fig.One eval-set item holds a question, graded gold chunks and a reference answer. The system retrieves c1, c2, c5 and answers. Retrieval metrics compare the ranked ids with the gold labels (recall@3 = 2/4, nDCG@3 = 0.947). Generation metrics compare the answer with the context and the question. Reading the two together says which stage to fix.

Retrieval metrics

recall@k=∣Rel∩Topk∣∣Rel∣,MRR@k=1∣Q∣∑q∈Q1rank⁡q,nDCG@k=∑i=1k2gi−1log⁡2(i+1)IDCG@k\text{recall@}k=\frac{|\mathrm{Rel}\cap\mathrm{Top}_k|}{|\mathrm{Rel}|},\qquad \text{MRR@}k=\frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\operatorname{rank}_q},\qquad \text{nDCG@}k=\frac{\sum_{i=1}^{k}\frac{2^{g_i}-1}{\log_2(i+1)}}{\text{IDCG@}k}
Rel\mathrm{Rel}
the chunks labelled relevant for the query (the qrels)
rank⁡q\operatorname{rank}_q
position of the first relevant chunk; the term is 0 if it is below kk
gig_i
graded relevance of the chunk at position i (0–3 here); IDCG is the same sum for the ideal order, so nDCG ∈ [0, 1]

Each answers a different question. Recall@k asks whether the evidence reached the prompt at all, and it caps the answer quality, so it is the metric to watch for a fixed kk. MRR asks how high the first good chunk is, which matters when only the top one or two are used. nDCG (Järvelin and Kekäläinen, 2002) rewards putting the most relevant chunks first; it is the standard metric for graded judgments. Precision@k matters less for RAG than for search, because the LLM can ignore an irrelevant chunk, though too many distractors do hurt.

Generation metrics

Faithfulness=∣V∣∣S∣,AnswerRelevance=1n∑i=1ncos⁡ ⁣(E(q), E(q^i))\text{Faithfulness}=\frac{|V|}{|S|},\qquad \text{AnswerRelevance}=\frac{1}{n}\sum_{i=1}^{n}\cos\!\big(E(q),\,E(\hat q_i)\big)
S,  VS,\;V
statements an LLM extracts from the answer; the subset it judges supported by the retrieved context
q^i\hat q_i
questions an LLM generates from the answer alone (n of them); E is an embedding model

These are the RAGAS definitions. Faithfulness is a groundedness check: it says nothing about whether the context was right. Answer relevance penalises evasive or padded answers, but not wrong ones, because a confident false answer still regenerates the original question. RAGAS also defined context relevance (the share of context sentences needed to answer). The library later added context precision and context recall, which compare the context with a reference answer and so need labels.

Reference points

RAGAS
Es et al., 2023. Reference-free: faithfulness, answer relevance and context relevance computed with LLM prompts; validated against human judgments on WikiEval, a set built from Wikipedia pages on events after the models' training data
ARES
Saad-Falcon et al., NAACL 2024. Fine-tunes small judges on synthetic data for context relevance, faithfulness and answer relevance, then combines them with a small human-labelled set through prediction-powered inference to give confidence intervals
BEIR
Thakur et al., NeurIPS 2021 Datasets and Benchmarks. 18 datasets for zero-shot retrieval, reported with nDCG@10
MS MARCO
The passage-ranking task's official metric is MRR@10
Pooling
TREC practice: label the union of the top results from several systems, so a new system is not penalised for finding unlabelled relevant chunks (it still can be, if it finds chunks no pooled system found)
query: “How long is the warranty on the X200 router?”# hybrid + rerank ranking · grade g (0–3)(2^g−1)/log₂(i+1)1c1 X200 warranty: 2 years from purchase37.0002c2 Warranty claims: file online with receipt21.8933c5 X100 warranty: 1 year00.0004c3 Extended protection plans1·5c6 X200 router specifications0·6c4 Returns accepted within 30 days1·7c7 Router setup guide0·8c8 Shipping times by region0·this query, k = 3 (4 relevant chunks)recall@3 = 0.500relevant in top-k / 4RR@3 = 1.0001 / rank of first relevantDCG@3 = 8.893sum of the gain columnIDCG@3 = 9.393same, ideal order 3,2,1,1nDCG@3 = 0.947DCG / IDCGeval set: 50 queries, k = 3 · mean ± 95% CI (1.96·sd/√n)recall@3BM25hybrid+rr0.367 ± 0.0970.493 ± 0.103MRR@3BM25hybrid+rr0.477 ± 0.1210.587 ± 0.117nDCG@3BM25hybrid+rr0.386 ± 0.1040.531 ± 0.105paired nDCG@3 difference (hybrid − BM25) = 0.145 ± 0.139 → significantanswer (grounded): “The X200 has a 2-year warranty from the date of purchase; file claims online.”faithfulness: claims vs top-3 context✓ X200 warranty is 2 years (c1)✓ it starts at purchase (c1)✓ claims are filed online (c2)faithfulness = 3/3 = 1.000answer relevance: questions made from the answer0.866 How long is the X200 warranty?0.577 When does the X200 warranty start?0.500 How do I file an X200 warranty claim?mean cosine to the query = 0.648(bag-of-words cosine stands in for embeddings)
retriever
eval set size
answer
fig.Default (hybrid + rerank, k = 3, 50 queries, grounded answer): the worked query scores recall@3 0.500, RR 1.000, nDCG@3 0.947; BM25 gets 0.500, 0.500, 0.440. Over 50 queries nDCG@3 is 0.386 ± 0.104 vs 0.531 ± 0.105, and the paired difference 0.145 ± 0.139 barely excludes 0. With 20 queries it is 0.062 ± 0.253, not significant; with 200, 0.141 ± 0.068. The hallucinated answer has faithfulness 0.000 but answer relevance 0.688, above the grounded answer's 0.648. The evasive answer scores relevance 0.180 and reaches faithfulness 1.000 only at k = 6.

Worked example: one query, two retrievers

RetrieverTop 3 (grades)recall@3RR@3DCG@3nDCG@3
BM25c6 (0), c4 (1), c1 (3)2/4 = 0.501/2 = 0.500 + 1/1.585 + 7/2 = 4.1310.440
hybrid + rerankc1 (3), c2 (2), c5 (0)2/4 = 0.501/1 = 1.007 + 3/1.585 + 0 = 8.8930.947
Gold grades c1 = 3, c2 = 2, c3 = 1, c4 = 1, so IDCG@3 = 7 + 3/log₂3 + 1/2 = 9.393. Checked in Python.

Recall cannot tell these retrievers apart: both put two relevant chunks in the top 3. nDCG can, because the hybrid puts the grade-3 chunk first. Which number matters depends on the generator. If the model reads all three chunks equally well, recall is what counts. If it attends mostly to the first chunk, or you pass only one, rank matters.

Building the eval set

  1. 1Sample real queries from logs (or from domain experts before launch), stratified by type: single-fact, multi-hop, comparison, unanswerable from the corpus. Unanswerable items test abstention.
  2. 2Label relevance by pooling the top 10–20 chunks from several retrievers and grading each (0–3). An LLM judge can pre-label; humans check a sample and the disagreements.
  3. 3Add synthetic items where logs are thin: give an LLM one chunk, ask for a question it answers, keep that chunk as gold, and filter out questions that copy the chunk's wording (they flatter BM25).
  4. 4Write reference answers for correctness and context-recall metrics; keep them short and factual.
  5. 5Size it so the confidence interval is smaller than the differences you care about: the half-width is about 1.96 s/n1.96\,s/\sqrt{n}. Compare systems with a paired difference on the same queries, which has a smaller variance than two separate means.
  6. 6Version and refresh: freeze each release of the set, store run outputs, and re-label when the corpus changes, since gold chunk ids go stale after re-chunking.
python
1import math
2
3def recall_at_k(ranked, gold, k):
4 return len(set(ranked[:k]) & set(gold)) / len(gold)
5
6def rr_at_k(ranked, gold, k):
7 return next((1 / (i + 1) for i, d in enumerate(ranked[:k]) if d in gold), 0.0)
8
9def ndcg_at_k(ranked, grades, k): # grades: {chunk_id: 0..3}
10 dcg = sum((2 ** grades.get(d, 0) - 1) / math.log2(i + 2) for i, d in enumerate(ranked[:k]))
11 ideal = sorted(grades.values(), reverse=True)[:k]
12 idcg = sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(ideal))
13 return dcg / idcg
14
15grades = {"c1": 3, "c2": 2, "c3": 1, "c4": 1}
16runs = {"bm25": ["c6", "c4", "c1", "c2", "c5", "c8", "c3", "c7"],
17 "hybrid": ["c1", "c2", "c5", "c3", "c6", "c4", "c7", "c8"]}
18for name, ranked in runs.items():
19 print(name, recall_at_k(ranked, grades, 3), rr_at_k(ranked, grades, 3),
20 round(ndcg_at_k(ranked, grades, 3), 3))
21
22def faithfulness(answer, context, llm):
23 claims = llm(f"List each factual statement in:\n{answer}").splitlines()
24 ok = [llm(f"Context:\n{context}\nIs this supported? yes/no: {c}") == "yes" for c in claims]
25 return sum(ok) / len(claims)
Retrieval metrics on the worked query (run: prints bm25 0.5 0.5 0.44 and hybrid 0.5 1.0 0.947), plus the shape of an LLM-judge faithfulness check (llm is a placeholder; that part not run)

Where it's used

Retriever and chunking choices

Recall@k and nDCG@10 on a fixed labelled set decide chunk size, hybrid weights, k and whether a reranker pays for its latency.

Release gating

Run the frozen eval set in CI; block a deploy when faithfulness or recall drops by more than the confidence interval.

Production monitoring

Reference-free metrics (faithfulness, answer relevance) can run on a sample of live traffic, where no gold labels exist.

Check yourself

Retrieval recall@5 is 0.95 but many answers are wrong, and faithfulness is 0.60. Where should you look first?

Check yourself

Ranking A puts grades (0, 1, 3) in its top 3, ranking B puts (3, 2, 0). Gold grades are 3, 2, 1, 1. Which statement is right?

Remember

  • ✓Evaluate retrieval and generation separately on the same items; the 2×2 of retrieval hit/miss × answer good/bad says which stage to fix.
  • ✓Retrieval: recall@k (did evidence arrive), MRR (rank of first hit), nDCG@k =DCG/IDCG=\text{DCG}/\text{IDCG} with gain 2g−12^g-1 (graded order).
  • ✓RAGAS: faithfulness =∣V∣/∣S∣=|V|/|S| supported statements; answer relevance == mean cosine between the question and questions regenerated from the answer.
  • ✓Faithfulness ignores whether the context was right; answer relevance ignores whether the answer is true. Read them together.
  • ✓Eval sets: real stratified queries, pooled graded labels, synthetic items with care, unanswerable cases; size by the 1.96 s/n1.96\,s/\sqrt n half-width and compare systems paired.
  • ✓Pin and validate the judge; version the eval set and re-label after re-chunking.
← back to Generative AI roadmap