Definition
RAG evaluation measures a retrieval-augmented system stage by stage. Retrieval is scored with information-retrieval metrics (recall@k, MRR, nDCG) against human- or LLM-labelled relevant chunks. Generation is scored for faithfulness to the retrieved context and relevance to the question, usually by an LLM judge. The best-known framework is RAGAS (Es, James, Espinosa-Anke and Schockaert, 2023; EACL 2024 demo track); ARES (Saad-Falcon, Khattab, Potts and Zaharia, NAACL 2024) adds trained judges and confidence intervals.
A RAG answer can fail in two places, and the fixes are different. If the right chunk never reached the prompt, no prompt or model change helps; you fix chunking, retrieval or reranking. If it did reach the prompt and the answer still contradicts it, you fix the prompt or the model. One end-to-end accuracy number cannot tell these apart, so every serious RAG eval scores both stages on the same items.
The pipeline itself is described in RAG architecture; the component articles (Document chunking, Hybrid search (BM25 + dense), Reranking) each move one of the metrics below. Claim-level hallucination metrics get their own article, Hallucination and groundedness evaluation.
Retrieval metrics
- the chunks labelled relevant for the query (the qrels)
- position of the first relevant chunk; the term is 0 if it is below
- graded relevance of the chunk at position i (0–3 here); IDCG is the same sum for the ideal order, so nDCG ∈ [0, 1]
Each answers a different question. Recall@k asks whether the evidence reached the prompt at all, and it caps the answer quality, so it is the metric to watch for a fixed . MRR asks how high the first good chunk is, which matters when only the top one or two are used. nDCG (Järvelin and Kekäläinen, 2002) rewards putting the most relevant chunks first; it is the standard metric for graded judgments. Precision@k matters less for RAG than for search, because the LLM can ignore an irrelevant chunk, though too many distractors do hurt.
Generation metrics
- statements an LLM extracts from the answer; the subset it judges supported by the retrieved context
- questions an LLM generates from the answer alone (n of them); E is an embedding model
These are the RAGAS definitions. Faithfulness is a groundedness check: it says nothing about whether the context was right. Answer relevance penalises evasive or padded answers, but not wrong ones, because a confident false answer still regenerates the original question. RAGAS also defined context relevance (the share of context sentences needed to answer). The library later added context precision and context recall, which compare the context with a reference answer and so need labels.
Reference points
- RAGAS
- Es et al., 2023. Reference-free: faithfulness, answer relevance and context relevance computed with LLM prompts; validated against human judgments on WikiEval, a set built from Wikipedia pages on events after the models' training data
- ARES
- Saad-Falcon et al., NAACL 2024. Fine-tunes small judges on synthetic data for context relevance, faithfulness and answer relevance, then combines them with a small human-labelled set through prediction-powered inference to give confidence intervals
- BEIR
- Thakur et al., NeurIPS 2021 Datasets and Benchmarks. 18 datasets for zero-shot retrieval, reported with nDCG@10
- MS MARCO
- The passage-ranking task's official metric is MRR@10
- Pooling
- TREC practice: label the union of the top results from several systems, so a new system is not penalised for finding unlabelled relevant chunks (it still can be, if it finds chunks no pooled system found)
Worked example: one query, two retrievers
| Retriever | Top 3 (grades) | recall@3 | RR@3 | DCG@3 | nDCG@3 |
|---|---|---|---|---|---|
| BM25 | c6 (0), c4 (1), c1 (3) | 2/4 = 0.50 | 1/2 = 0.50 | 0 + 1/1.585 + 7/2 = 4.131 | 0.440 |
| hybrid + rerank | c1 (3), c2 (2), c5 (0) | 2/4 = 0.50 | 1/1 = 1.00 | 7 + 3/1.585 + 0 = 8.893 | 0.947 |
Recall cannot tell these retrievers apart: both put two relevant chunks in the top 3. nDCG can, because the hybrid puts the grade-3 chunk first. Which number matters depends on the generator. If the model reads all three chunks equally well, recall is what counts. If it attends mostly to the first chunk, or you pass only one, rank matters.
Building the eval set
- 1Sample real queries from logs (or from domain experts before launch), stratified by type: single-fact, multi-hop, comparison, unanswerable from the corpus. Unanswerable items test abstention.
- 2Label relevance by pooling the top 10–20 chunks from several retrievers and grading each (0–3). An LLM judge can pre-label; humans check a sample and the disagreements.
- 3Add synthetic items where logs are thin: give an LLM one chunk, ask for a question it answers, keep that chunk as gold, and filter out questions that copy the chunk's wording (they flatter BM25).
- 4Write reference answers for correctness and context-recall metrics; keep them short and factual.
- 5Size it so the confidence interval is smaller than the differences you care about: the half-width is about . Compare systems with a paired difference on the same queries, which has a smaller variance than two separate means.
- 6Version and refresh: freeze each release of the set, store run outputs, and re-label when the corpus changes, since gold chunk ids go stale after re-chunking.
1import math23def recall_at_k(ranked, gold, k):4 return len(set(ranked[:k]) & set(gold)) / len(gold)56def rr_at_k(ranked, gold, k):7 return next((1 / (i + 1) for i, d in enumerate(ranked[:k]) if d in gold), 0.0)89def ndcg_at_k(ranked, grades, k): # grades: {chunk_id: 0..3}10 dcg = sum((2 ** grades.get(d, 0) - 1) / math.log2(i + 2) for i, d in enumerate(ranked[:k]))11 ideal = sorted(grades.values(), reverse=True)[:k]12 idcg = sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(ideal))13 return dcg / idcg1415grades = {"c1": 3, "c2": 2, "c3": 1, "c4": 1}16runs = {"bm25": ["c6", "c4", "c1", "c2", "c5", "c8", "c3", "c7"],17 "hybrid": ["c1", "c2", "c5", "c3", "c6", "c4", "c7", "c8"]}18for name, ranked in runs.items():19 print(name, recall_at_k(ranked, grades, 3), rr_at_k(ranked, grades, 3),20 round(ndcg_at_k(ranked, grades, 3), 3))2122def faithfulness(answer, context, llm):23 claims = llm(f"List each factual statement in:\n{answer}").splitlines()24 ok = [llm(f"Context:\n{context}\nIs this supported? yes/no: {c}") == "yes" for c in claims]25 return sum(ok) / len(claims)Where it's used
Retriever and chunking choices
Recall@k and nDCG@10 on a fixed labelled set decide chunk size, hybrid weights, k and whether a reranker pays for its latency.
Release gating
Run the frozen eval set in CI; block a deploy when faithfulness or recall drops by more than the confidence interval.
Production monitoring
Reference-free metrics (faithfulness, answer relevance) can run on a sample of live traffic, where no gold labels exist.
Check yourself
Retrieval recall@5 is 0.95 but many answers are wrong, and faithfulness is 0.60. Where should you look first?
Check yourself
Ranking A puts grades (0, 1, 3) in its top 3, ranking B puts (3, 2, 0). Gold grades are 3, 2, 1, 1. Which statement is right?
Remember
- ✓Evaluate retrieval and generation separately on the same items; the 2×2 of retrieval hit/miss × answer good/bad says which stage to fix.
- ✓Retrieval: recall@k (did evidence arrive), MRR (rank of first hit), nDCG@k with gain (graded order).
- ✓RAGAS: faithfulness supported statements; answer relevance mean cosine between the question and questions regenerated from the answer.
- ✓Faithfulness ignores whether the context was right; answer relevance ignores whether the answer is true. Read them together.
- ✓Eval sets: real stratified queries, pooled graded labels, synthetic items with care, unanswerable cases; size by the half-width and compare systems paired.
- ✓Pin and validate the judge; version the eval set and re-label after re-chunking.