# RAG architecture

> A RAG system is two pipelines: offline, documents are parsed, chunked, embedded and indexed with metadata; online, each query is rewritten, retrieved, reranked and packed into a cited prompt. Retrieval recall caps answer quality.

Source: https://www.mlfoundry.in/learn/rag-architecture  
Section: Generative AI › Retrieval-Augmented Generation (RAG)  
Level: core · 8 min read · updated 2026-10-21

**RAG architecture:** is the system design of retrieval-augmented generation: the components, data flow and budgets that turn a document collection and a general-purpose LLM into an assistant that answers from those documents and cites them. The term RAG comes from Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020). The survey by Gao et al. ("Retrieval-Augmented Generation for Large Language Models: A Survey", 2023) sorts systems into **naive** (index, retrieve, generate), **advanced** (adds query rewriting, hybrid retrieval and reranking) and **modular** RAG (components routed and repeated).

An LLM alone cannot know your private documents or anything after its training cutoff, and it gives no source to check. Fine-tuning can change behaviour, but it is a poor way to store facts that change weekly or must be deleted on request (see **Fine-tuning vs RAG vs prompting**). RAG keeps the facts in an index that you can update, filter and audit. The model only reads what retrieval hands it. The model-side view, with the marginal likelihood over passages, is in **Retrieval-augmented generation (RAG)**; this article is about the system around the model.

## Two pipelines

The **offline** path runs in batch whenever sources change. Parse each document to clean text, keep its title, date, URL and access list, split it into chunks (see **Document chunking**), embed every chunk (see **Embedding models**) and write vectors, text and metadata to an index (see **Vector databases**). A lexical index over the same chunks is built alongside.

> Figure: Top: the offline indexing path, from raw sources to an index of 10,000 chunks of 300 tokens with 50-token overlap, each stored with its vector, text, terms and metadata. Bottom: one query, "reset my router?", from user team-a. An optional query step rewrites it and adds the access filter, hybrid retrieval returns 50 candidates, a cross-encoder keeps 5, and the prompt (rules, sources [1]–[5], question) is 2,000 tokens. The answer cites [2]. Stage latencies match the budget figure below. Hover a lane to highlight it.

The **online** path runs per query. An optional query step rewrites follow-ups into stand-alone questions or generates hypothetical answers to search with (see **Query rewriting and HyDE**). Retrieval runs BM25 and dense search and fuses them (see **Hybrid search (BM25 + dense)**). A cross-encoder reorders the top 50–100 (see **Reranking**). The best few chunks go into a prompt with numbered sources and an instruction to answer only from them, and the model streams an answer with citations.

## Where quality and latency go

$$
P(\text{correct})\;\approx\;R@k\cdot p_{\text{read}}(k)\;+\;\big(1-R@k\big)\,p_{\text{closed}}
$$

- $R@k$: end-to-end recall: probability that a gold chunk is among the $k$ chunks in the prompt (after fusion and reranking)
- $p_{\text{read}}(k)$: probability that the model answers correctly when the gold chunk is present; it falls slowly with $k$ as distractors and length grow
- $p_{\text{closed}}$: what the model gets right with no useful context (from its weights, or by luck)

The first term is why retrieval is the usual bottleneck: if $R@k=0.7$, no model can answer more than about 70% of questions from evidence. Raising $k$ lifts $R@k$ but lowers $p_{\text{read}}$ and costs tokens. Liu et al. ("Lost in the Middle", TACL 2024) measured the second effect: in multi-document QA, accuracy was highest when the relevant passage was at the start or end of the context and dropped when it sat in the middle.

$$
\text{TTFT}\approx t_{\text{embed}}+t_{\text{retrieve}}+n\,t_{\text{pair}}+\frac{T_0+k\,T_{\text{chunk}}}{r_{\text{prefill}}},\qquad \text{cost}=T_{\text{in}}\,c_{\text{in}}+T_{\text{out}}\,c_{\text{out}}
$$

- $n\,t_{\text{pair}}$: reranking $n$ candidates with a cross-encoder at $t_{\text{pair}}$ per pair
- $T_0,\;T_{\text{chunk}}$: fixed prompt tokens (rules, question, history) and tokens per chunk
- $r_{\text{prefill}}$: prompt tokens processed per second before the first output token
- $c_{\text{in}},\,c_{\text{out}}$: price per input and output token; input tokens dominate when k is large

Components, their main knob, and where each is covered

| Stage | Typical choice | Main knob | See |
| --- | --- | --- | --- |
| Chunking | split on headings or paragraphs, title prepended | chunk size, overlap | Document chunking |
| Embedding | bi-encoder, unit-normalised vectors | model, dimension | Embedding models |
| Index | HNSW or IVF vectors + inverted index + metadata | recall vs latency | Vector databases |
| Retrieval | BM25 + dense, fused by RRF | candidates per lane | Hybrid search (BM25 + dense) |
| Reranking | cross-encoder over the top n | n, model size | Reranking |
| Prompting | numbered sources, answer-only-from-sources rule | k, order, format | Prompt templates |
| Evaluation | retrieval and answer scored separately | labelled query set | RAG evaluation |

**Reference points**

- **RAG (2020):** Lewis et al.: DPR retriever over 21M 100-word Wikipedia passages, BART-large generator, top 5 or 10 passages; query encoder and generator fine-tuned jointly, document index kept fixed
- **Lost in the middle:** Liu et al. (TACL 2024): with 10, 20 or 30 retrieved documents, multi-document QA accuracy was U-shaped in the position of the gold document
- **Survey taxonomy:** Gao et al. (2023): naive, advanced and modular RAG; advanced adds pre-retrieval (query rewriting) and post-retrieval (reranking, compression) steps
- **Metadata:** access-control lists, source, date and language are stored per chunk so that retrieval can filter before ranking, not after generation

> Figure: An exact recall model (one gold chunk among 10,000, gold score N(μ, 1) against N(0, 1) distractors) and an assumed reader, latency and price model. Defaults (hybrid μ = 4.6, reranker μ = 4 over the top 50, k = 5): recall@5 = 0.9737, accuracy 0.8764, 2,000 prompt tokens, TTFT 260 ms, 9.75 dollars per 1,000 queries; the best k is 3 (0.8831). Turn the reranker off: recall@5 drops to 0.8983, the best k moves to 7 and accuracy peaks at 0.8222. With BM25 only and no reranker, recall@5 is 0.684. Push k to 30 with the reranker on: recall barely moves (0.9782), accuracy falls to 0.7330 and the prompt grows to 9,500 tokens (TTFT 1,010 ms, 32.25 dollars).

## Worked example: sizing k

**Default setup (hybrid, rerank top 50, k = 5)**

1. Hybrid retrieval puts the gold chunk in the top 50 with probability $0.9782$. Among those 50 the reranker puts it in its top 5 with probability $0.9953$, so $R@5=0.9782\times0.9953=0.9737$.
2. Reader: $p_{\text{read}}(5)=0.92-0.006\cdot4=0.896$. Accuracy $=0.9737\cdot0.896+0.0263\cdot0.15=0.8724+0.0039=0.8764$.
3. Prompt $=500+5\cdot300=2{,}000$ tokens. TTFT $=15+20+50\cdot0.5+2000/10{,}000\,\text{s}=15+20+25+200=260$ ms.
4. Cost per 1,000 queries, at the assumed 3 and 15 dollars per million input and output tokens: $(2000\cdot3+250\cdot15)/1000=6.00+3.75=9.75$ dollars.

Configurations compared in the figure's model (prices and speeds assumed)

| Setup | k | recall@k | Accuracy | Prompt tokens | TTFT (ms) | Dollars per 1k queries |
| --- | --- | --- | --- | --- | --- | --- |
| BM25, no reranker | 5 | 0.6840 | 0.6602 | 2,000 | 235 | 9.75 |
| hybrid, no reranker | 5 | 0.8983 | 0.8201 | 2,000 | 235 | 9.75 |
| hybrid, no reranker | 7 | 0.9158 | 0.8222 | 2,600 | 295 | 11.55 |
| hybrid + rerank 50 | 5 | 0.9737 | 0.8764 | 2,000 | 260 | 9.75 |
| hybrid + rerank 50 | 3 | 0.9672 | 0.8831 | 1,400 | 200 | 7.95 |
| hybrid + rerank 50 | 30 | 0.9782 | 0.7330 | 9,500 | 1,010 | 32.25 |

The reranker costs 25 ms and pays for itself: it lets $k$ drop from 7 to 3, which saves 1,200 prompt tokens per query, cuts TTFT by 95 ms and raises accuracy by 6 points. The exact numbers depend on the assumed reader curve; the shape, a peak at small $k$ once the ranking is good, is what to look for on your own data.

```python
SYSTEM = ("Answer only from the numbered sources. Cite them as [n]. "
          "If the sources do not contain the answer, say so.")

def answer(question: str, user, history=(), k=5, n=50):
    q = rewrite(question, history) if history else question     # stand-alone query
    acl = {"acl": {"any_of": user.groups}}                        # filter BEFORE ranking
    cands = index.hybrid_search(q, top_n=n, filter=acl)           # BM25 + dense, RRF
    top = reranker.rank(q, cands)[:k]                             # cross-encoder
    sources = "\n\n".join(f"[{i+1}] {c.title} ({c.updated})\n{c.text}"
                           for i, c in enumerate(top))
    out = llm.generate(system=SYSTEM,
                       user=f"Sources:\n{sources}\n\nQuestion: {question}",
                       max_tokens=250, stream=True)
    log(query=q, chunk_ids=[c.id for c in top], answer=out)      # for RAG evaluation
    return out, [(i + 1, c.url) for i, c in enumerate(top)]
```

The online path as one function (sketch: index, reranker and llm stand for your search and model clients; not run here)

> **Security and freshness live in the index:** Apply access filters inside the retrieval query, not by asking the model to ignore chunks the user may not see: anything in the prompt can leak into the answer. Retrieved text is also untrusted input. A web page or ticket can contain instructions aimed at the model (see **Prompt injection**). When a document is deleted or edited, delete or re-embed its chunks; a stale index answers confidently from text that no longer exists.

> **Debug stage by stage:** Log the chunk ids behind every answer. For a wrong answer, first ask: was a gold chunk in the top 50? In the top $k$? If not, fix chunking, retrieval or reranking. If it was there and the answer still contradicts it, the problem is reading or faithfulness (see **Hallucination and groundedness evaluation**). The two need different fixes and different metrics (see **RAG evaluation**).

- **Naive RAG:** Embed, top-k vector search, stuff the prompt. Good for prototypes; fails on exact identifiers, follow-up questions and long-tail queries.
- **Advanced RAG:** Adds query rewriting, hybrid retrieval, reranking and citation checks. The default for production help desks and document assistants.
- **Agentic and graph RAG:** The model decides when and what to retrieve and may retrieve repeatedly (**Agentic RAG**); or retrieval runs over an entity graph and community summaries for corpus-wide questions (**GraphRAG**).

**Question:** A RAG assistant answers 62% of test questions correctly. Logs show a gold chunk was in the prompt for 64% of questions, and the answer was right in 93% of those cases. What should you fix first?

- A. The prompt template
- B. Retrieval: the gold chunk is missing for 36% of questions
- C. Switch to a larger LLM
- D. Lower the temperature

**Answer:** B. Retrieval: the gold chunk is missing for 36% of questions — With the gold chunk present the model is right 93% of the time, so reading is fine. Recall caps the system: $0.64\cdot0.93=0.595$, and the remaining 2.5 points come from questions answered without evidence. Better chunking, hybrid search or reranking raise $R@k$; a bigger model cannot read a chunk it never receives.

**Question:** In the budget figure, why does accuracy fall when k rises from 5 to 30 with the reranker on?

- A. Recall@k decreases as k grows
- B. Recall is already about 0.97 at k = 5, so extra chunks add almost no recall but lower the reader's accuracy and add 7,500 prompt tokens
- C. The reranker stops working above k = 5
- D. TTFT limits are exceeded

**Answer:** B. Recall is already about 0.97 at k = 5, so extra chunks add almost no recall but lower the reader's accuracy and add 7,500 prompt tokens — Recall@k can only rise with k: here from 0.9737 to 0.9782. The assumed reader loses 0.006 per extra chunk, 0.15 over 25 chunks, which outweighs the 0.0045 gain in recall.

**Key points**

- Two pipelines: offline parse, chunk, embed, index (with metadata and ACLs); online rewrite, retrieve, rerank, prompt, generate with citations.
- $P(\text{correct})\approx R@k\,p_{\text{read}}(k)+(1-R@k)\,p_{\text{closed}}$: retrieval recall caps quality.
- More chunks raise recall but cost tokens and latency and can lower reading accuracy; a reranker lets k shrink.
- Latency budget: embed + retrieve + $n\,t_{\text{pair}}$ + prefill of $T_0+kT_{\text{chunk}}$ tokens.
- Filter by access before ranking, treat retrieved text as untrusted, re-index on change, and evaluate retrieval and answers separately.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
