Definition
RAG architecture is the system design of retrieval-augmented generation: the components, data flow and budgets that turn a document collection and a general-purpose LLM into an assistant that answers from those documents and cites them. The term RAG comes from Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020). The survey by Gao et al. ("Retrieval-Augmented Generation for Large Language Models: A Survey", 2023) sorts systems into naive (index, retrieve, generate), advanced (adds query rewriting, hybrid retrieval and reranking) and modular RAG (components routed and repeated).
An LLM alone cannot know your private documents or anything after its training cutoff, and it gives no source to check. Fine-tuning can change behaviour, but it is a poor way to store facts that change weekly or must be deleted on request (see Fine-tuning vs RAG vs prompting). RAG keeps the facts in an index that you can update, filter and audit. The model only reads what retrieval hands it. The model-side view, with the marginal likelihood over passages, is in Retrieval-augmented generation (RAG); this article is about the system around the model.
Two pipelines
The offline path runs in batch whenever sources change. Parse each document to clean text, keep its title, date, URL and access list, split it into chunks (see Document chunking), embed every chunk (see Embedding models) and write vectors, text and metadata to an index (see Vector databases). A lexical index over the same chunks is built alongside.
The online path runs per query. An optional query step rewrites follow-ups into stand-alone questions or generates hypothetical answers to search with (see Query rewriting and HyDE). Retrieval runs BM25 and dense search and fuses them (see Hybrid search (BM25 + dense)). A cross-encoder reorders the top 50–100 (see Reranking). The best few chunks go into a prompt with numbered sources and an instruction to answer only from them, and the model streams an answer with citations.
Where quality and latency go
- end-to-end recall: probability that a gold chunk is among the chunks in the prompt (after fusion and reranking)
- probability that the model answers correctly when the gold chunk is present; it falls slowly with as distractors and length grow
- what the model gets right with no useful context (from its weights, or by luck)
The first term is why retrieval is the usual bottleneck: if , no model can answer more than about 70% of questions from evidence. Raising lifts but lowers and costs tokens. Liu et al. ("Lost in the Middle", TACL 2024) measured the second effect: in multi-document QA, accuracy was highest when the relevant passage was at the start or end of the context and dropped when it sat in the middle.
- reranking candidates with a cross-encoder at per pair
- fixed prompt tokens (rules, question, history) and tokens per chunk
- prompt tokens processed per second before the first output token
- price per input and output token; input tokens dominate when k is large
| Stage | Typical choice | Main knob | See |
|---|---|---|---|
| Chunking | split on headings or paragraphs, title prepended | chunk size, overlap | Document chunking |
| Embedding | bi-encoder, unit-normalised vectors | model, dimension | Embedding models |
| Index | HNSW or IVF vectors + inverted index + metadata | recall vs latency | Vector databases |
| Retrieval | BM25 + dense, fused by RRF | candidates per lane | Hybrid search (BM25 + dense) |
| Reranking | cross-encoder over the top n | n, model size | Reranking |
| Prompting | numbered sources, answer-only-from-sources rule | k, order, format | Prompt templates |
| Evaluation | retrieval and answer scored separately | labelled query set | RAG evaluation |
Reference points
- RAG (2020)
- Lewis et al.: DPR retriever over 21M 100-word Wikipedia passages, BART-large generator, top 5 or 10 passages; query encoder and generator fine-tuned jointly, document index kept fixed
- Lost in the middle
- Liu et al. (TACL 2024): with 10, 20 or 30 retrieved documents, multi-document QA accuracy was U-shaped in the position of the gold document
- Survey taxonomy
- Gao et al. (2023): naive, advanced and modular RAG; advanced adds pre-retrieval (query rewriting) and post-retrieval (reranking, compression) steps
- Metadata
- access-control lists, source, date and language are stored per chunk so that retrieval can filter before ranking, not after generation
Worked example: sizing k
Default setup (hybrid, rerank top 50, k = 5)
- 1Hybrid retrieval puts the gold chunk in the top 50 with probability . Among those 50 the reranker puts it in its top 5 with probability , so .
- 2Reader: . Accuracy .
- 3Prompt tokens. TTFT ms.
- 4Cost per 1,000 queries, at the assumed 3 and 15 dollars per million input and output tokens: dollars.
| Setup | k | recall@k | Accuracy | Prompt tokens | TTFT (ms) | Dollars per 1k queries |
|---|---|---|---|---|---|---|
| BM25, no reranker | 5 | 0.6840 | 0.6602 | 2,000 | 235 | 9.75 |
| hybrid, no reranker | 5 | 0.8983 | 0.8201 | 2,000 | 235 | 9.75 |
| hybrid, no reranker | 7 | 0.9158 | 0.8222 | 2,600 | 295 | 11.55 |
| hybrid + rerank 50 | 5 | 0.9737 | 0.8764 | 2,000 | 260 | 9.75 |
| hybrid + rerank 50 | 3 | 0.9672 | 0.8831 | 1,400 | 200 | 7.95 |
| hybrid + rerank 50 | 30 | 0.9782 | 0.7330 | 9,500 | 1,010 | 32.25 |
The reranker costs 25 ms and pays for itself: it lets drop from 7 to 3, which saves 1,200 prompt tokens per query, cuts TTFT by 95 ms and raises accuracy by 6 points. The exact numbers depend on the assumed reader curve; the shape, a peak at small once the ranking is good, is what to look for on your own data.
1SYSTEM = ("Answer only from the numbered sources. Cite them as [n]. "2 "If the sources do not contain the answer, say so.")34def answer(question: str, user, history=(), k=5, n=50):5 q = rewrite(question, history) if history else question # stand-alone query6 acl = {"acl": {"any_of": user.groups}} # filter BEFORE ranking7 cands = index.hybrid_search(q, top_n=n, filter=acl) # BM25 + dense, RRF8 top = reranker.rank(q, cands)[:k] # cross-encoder9 sources = "\n\n".join(f"[{i+1}] {c.title} ({c.updated})\n{c.text}"10 for i, c in enumerate(top))11 out = llm.generate(system=SYSTEM,12 user=f"Sources:\n{sources}\n\nQuestion: {question}",13 max_tokens=250, stream=True)14 log(query=q, chunk_ids=[c.id for c in top], answer=out) # for RAG evaluation15 return out, [(i + 1, c.url) for i, c in enumerate(top)]Where it's used
Naive RAG
Embed, top-k vector search, stuff the prompt. Good for prototypes; fails on exact identifiers, follow-up questions and long-tail queries.
Advanced RAG
Adds query rewriting, hybrid retrieval, reranking and citation checks. The default for production help desks and document assistants.
Agentic and graph RAG
The model decides when and what to retrieve and may retrieve repeatedly (Agentic RAG); or retrieval runs over an entity graph and community summaries for corpus-wide questions (GraphRAG).
Check yourself
A RAG assistant answers 62% of test questions correctly. Logs show a gold chunk was in the prompt for 64% of questions, and the answer was right in 93% of those cases. What should you fix first?
Check yourself
In the budget figure, why does accuracy fall when k rises from 5 to 30 with the reranker on?
Remember
- ✓Two pipelines: offline parse, chunk, embed, index (with metadata and ACLs); online rewrite, retrieve, rerank, prompt, generate with citations.
- ✓: retrieval recall caps quality.
- ✓More chunks raise recall but cost tokens and latency and can lower reading accuracy; a reranker lets k shrink.
- ✓Latency budget: embed + retrieve + + prefill of tokens.
- ✓Filter by access before ranking, treat retrieved text as untrusted, re-index on change, and evaluate retrieval and answers separately.