RAG architecture

A RAG system is two pipelines: offline, documents are parsed, chunked, embedded and indexed with metadata; online, each query is rewritten, retrieved, reranked and packed into a cited prompt. Retrieval recall caps answer quality.

8 min read730 wordscore0 viewsupdated 2026-10-21

Definition

RAG architecture is the system design of retrieval-augmented generation: the components, data flow and budgets that turn a document collection and a general-purpose LLM into an assistant that answers from those documents and cites them. The term RAG comes from Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020). The survey by Gao et al. ("Retrieval-Augmented Generation for Large Language Models: A Survey", 2023) sorts systems into naive (index, retrieve, generate), advanced (adds query rewriting, hybrid retrieval and reranking) and modular RAG (components routed and repeated).

An LLM alone cannot know your private documents or anything after its training cutoff, and it gives no source to check. Fine-tuning can change behaviour, but it is a poor way to store facts that change weekly or must be deleted on request (see Fine-tuning vs RAG vs prompting). RAG keeps the facts in an index that you can update, filter and audit. The model only reads what retrieval hands it. The model-side view, with the marginal likelihood over passages, is in Retrieval-augmented generation (RAG); this article is about the system around the model.

Two pipelines

The offline path runs in batch whenever sources change. Parse each document to clean text, keep its title, date, URL and access list, split it into chunks (see Document chunking), embed every chunk (see Embedding models) and write vectors, text and metadata to an index (see Vector databases). A lexical index over the same chunks is built alongside.

OFFLINE: indexing pipeline (batch, re-run when sources change)sourcesPDF, HTML, wikitickets, DB rowsre-run on changeparse + cleantext, tablestitles, datesdrop boilerplatechunk300 tokens50 overlap+ doc titleembed + tokenise1 vector/chunkterms → postingsbatch, offlineindex (10,000 chunks)vector | text | termsmeta: doc_id, acl, updated, urlread both indexes, filtered by aclONLINE: per queryquery"reset myrouter?"user: team-aquery steprewrite, HyDEacl = team-ahistory merge0–1 LLM callretrieveBM25 + denseRRF fuse→ top 5015 + 20 msrerankcross-encoder50 pairs→ top 525 msbuild promptrules + [1]-[5]+ question= 2,000 tokensgenerate"Hold reset10 s [2]"streamedprefill 200 msevaluate separately:retrieval → recall@50 after fusion, recall@5 after rerankanswer → faithfulness to [1]-[5], citation precision, refusalsguard:retrieved text is untrusted input; acl filter before ranking
fig.Top: the offline indexing path, from raw sources to an index of 10,000 chunks of 300 tokens with 50-token overlap, each stored with its vector, text, terms and metadata. Bottom: one query, "reset my router?", from user team-a. An optional query step rewrites it and adds the access filter, hybrid retrieval returns 50 candidates, a cross-encoder keeps 5, and the prompt (rules, sources [1]–[5], question) is 2,000 tokens. The answer cites [2]. Stage latencies match the budget figure below. Hover a lane to highlight it.

The online path runs per query. An optional query step rewrites follow-ups into stand-alone questions or generates hypothetical answers to search with (see Query rewriting and HyDE). Retrieval runs BM25 and dense search and fuses them (see Hybrid search (BM25 + dense)). A cross-encoder reorders the top 50–100 (see Reranking). The best few chunks go into a prompt with numbered sources and an instruction to answer only from them, and the model streams an answer with citations.

Where quality and latency go

P(correct)  ≈  R@k⋅pread(k)  +  (1−R@k) pclosedP(\text{correct})\;\approx\;R@k\cdot p_{\text{read}}(k)\;+\;\big(1-R@k\big)\,p_{\text{closed}}
R@kR@k
end-to-end recall: probability that a gold chunk is among the kk chunks in the prompt (after fusion and reranking)
pread(k)p_{\text{read}}(k)
probability that the model answers correctly when the gold chunk is present; it falls slowly with kk as distractors and length grow
pclosedp_{\text{closed}}
what the model gets right with no useful context (from its weights, or by luck)

The first term is why retrieval is the usual bottleneck: if R@k=0.7R@k=0.7, no model can answer more than about 70% of questions from evidence. Raising kk lifts R@kR@k but lowers preadp_{\text{read}} and costs tokens. Liu et al. ("Lost in the Middle", TACL 2024) measured the second effect: in multi-document QA, accuracy was highest when the relevant passage was at the start or end of the context and dropped when it sat in the middle.

TTFT≈tembed+tretrieve+n tpair+T0+k Tchunkrprefill,cost=Tin cin+Tout cout\text{TTFT}\approx t_{\text{embed}}+t_{\text{retrieve}}+n\,t_{\text{pair}}+\frac{T_0+k\,T_{\text{chunk}}}{r_{\text{prefill}}},\qquad \text{cost}=T_{\text{in}}\,c_{\text{in}}+T_{\text{out}}\,c_{\text{out}}
n tpairn\,t_{\text{pair}}
reranking nn candidates with a cross-encoder at tpairt_{\text{pair}} per pair
T0,  TchunkT_0,\;T_{\text{chunk}}
fixed prompt tokens (rules, question, history) and tokens per chunk
rprefillr_{\text{prefill}}
prompt tokens processed per second before the first output token
cin, coutc_{\text{in}},\,c_{\text{out}}
price per input and output token; input tokens dominate when k is large
StageTypical choiceMain knobSee
Chunkingsplit on headings or paragraphs, title prependedchunk size, overlapDocument chunking
Embeddingbi-encoder, unit-normalised vectorsmodel, dimensionEmbedding models
IndexHNSW or IVF vectors + inverted index + metadatarecall vs latencyVector databases
RetrievalBM25 + dense, fused by RRFcandidates per laneHybrid search (BM25 + dense)
Rerankingcross-encoder over the top nn, model sizeReranking
Promptingnumbered sources, answer-only-from-sources rulek, order, formatPrompt templates
Evaluationretrieval and answer scored separatelylabelled query setRAG evaluation
Components, their main knob, and where each is covered

Reference points

RAG (2020)
Lewis et al.: DPR retriever over 21M 100-word Wikipedia passages, BART-large generator, top 5 or 10 passages; query encoder and generator fine-tuned jointly, document index kept fixed
Lost in the middle
Liu et al. (TACL 2024): with 10, 20 or 30 retrieved documents, multi-document QA accuracy was U-shaped in the position of the gold document
Survey taxonomy
Gao et al. (2023): naive, advanced and modular RAG; advanced adds pre-retrieval (query rewriting) and post-retrieval (reranking, compression) steps
Metadata
access-control lists, source, date and language are stored per chunk so that retrieval can filter before ranking, not after generation
one gold chunk among 10,000: how many chunks k should reach the prompt?0.40.60.81.0151015202530k = chunks in the promptrecall@kreader, gold presentend-to-end accuracyfaint: same, no rerankerrecall@5 (gold in prompt)0.9737P(right | gold in prompt)0.896end-to-end accuracy0.8764best k for this setup3 (0.8831)prompt tokens2000time to first token260 ms$ per 1k queries9.75time to first token, by stage (axis 0 to 1200 ms)embed 15 msretrieve 20 msrerank 25 msprefill 200 ms+ decoding 250 output tokens at 50 tok/s = 5000 ms; total 5260 msassumed: 300-token chunks, 500 fixed prompt tokens, prefill 10,000 tok/s, rerank 0.5 ms/pair,$3 / $15 per M input / output tokens; reader 0.92 − 0.006(k − 1), closed-book 0.15first stage: gold score N(μ, 1) vs N(0, 1); μ = 4.6; reranker μ = 4 over the top 50
retriever
fig.An exact recall model (one gold chunk among 10,000, gold score N(μ, 1) against N(0, 1) distractors) and an assumed reader, latency and price model. Defaults (hybrid μ = 4.6, reranker μ = 4 over the top 50, k = 5): recall@5 = 0.9737, accuracy 0.8764, 2,000 prompt tokens, TTFT 260 ms, 9.75 dollars per 1,000 queries; the best k is 3 (0.8831). Turn the reranker off: recall@5 drops to 0.8983, the best k moves to 7 and accuracy peaks at 0.8222. With BM25 only and no reranker, recall@5 is 0.684. Push k to 30 with the reranker on: recall barely moves (0.9782), accuracy falls to 0.7330 and the prompt grows to 9,500 tokens (TTFT 1,010 ms, 32.25 dollars).

Worked example: sizing k

Default setup (hybrid, rerank top 50, k = 5)

  1. 1Hybrid retrieval puts the gold chunk in the top 50 with probability 0.97820.9782. Among those 50 the reranker puts it in its top 5 with probability 0.99530.9953, so R@5=0.9782×0.9953=0.9737R@5=0.9782\times0.9953=0.9737.
  2. 2Reader: pread(5)=0.92−0.006⋅4=0.896p_{\text{read}}(5)=0.92-0.006\cdot4=0.896. Accuracy =0.9737⋅0.896+0.0263⋅0.15=0.8724+0.0039=0.8764=0.9737\cdot0.896+0.0263\cdot0.15=0.8724+0.0039=0.8764.
  3. 3Prompt =500+5⋅300=2,000=500+5\cdot300=2{,}000 tokens. TTFT =15+20+50⋅0.5+2000/10,000 s=15+20+25+200=260=15+20+50\cdot0.5+2000/10{,}000\,\text{s}=15+20+25+200=260 ms.
  4. 4Cost per 1,000 queries, at the assumed 3 and 15 dollars per million input and output tokens: (2000⋅3+250⋅15)/1000=6.00+3.75=9.75(2000\cdot3+250\cdot15)/1000=6.00+3.75=9.75 dollars.
Setupkrecall@kAccuracyPrompt tokensTTFT (ms)Dollars per 1k queries
BM25, no reranker50.68400.66022,0002359.75
hybrid, no reranker50.89830.82012,0002359.75
hybrid, no reranker70.91580.82222,60029511.55
hybrid + rerank 5050.97370.87642,0002609.75
hybrid + rerank 5030.96720.88311,4002007.95
hybrid + rerank 50300.97820.73309,5001,01032.25
Configurations compared in the figure's model (prices and speeds assumed)

The reranker costs 25 ms and pays for itself: it lets kk drop from 7 to 3, which saves 1,200 prompt tokens per query, cuts TTFT by 95 ms and raises accuracy by 6 points. The exact numbers depend on the assumed reader curve; the shape, a peak at small kk once the ranking is good, is what to look for on your own data.

python
1SYSTEM = ("Answer only from the numbered sources. Cite them as [n]. "
2 "If the sources do not contain the answer, say so.")
3
4def answer(question: str, user, history=(), k=5, n=50):
5 q = rewrite(question, history) if history else question # stand-alone query
6 acl = {"acl": {"any_of": user.groups}} # filter BEFORE ranking
7 cands = index.hybrid_search(q, top_n=n, filter=acl) # BM25 + dense, RRF
8 top = reranker.rank(q, cands)[:k] # cross-encoder
9 sources = "\n\n".join(f"[{i+1}] {c.title} ({c.updated})\n{c.text}"
10 for i, c in enumerate(top))
11 out = llm.generate(system=SYSTEM,
12 user=f"Sources:\n{sources}\n\nQuestion: {question}",
13 max_tokens=250, stream=True)
14 log(query=q, chunk_ids=[c.id for c in top], answer=out) # for RAG evaluation
15 return out, [(i + 1, c.url) for i, c in enumerate(top)]
The online path as one function (sketch: index, reranker and llm stand for your search and model clients; not run here)

Where it's used

Naive RAG

Embed, top-k vector search, stuff the prompt. Good for prototypes; fails on exact identifiers, follow-up questions and long-tail queries.

Advanced RAG

Adds query rewriting, hybrid retrieval, reranking and citation checks. The default for production help desks and document assistants.

Agentic and graph RAG

The model decides when and what to retrieve and may retrieve repeatedly (Agentic RAG); or retrieval runs over an entity graph and community summaries for corpus-wide questions (GraphRAG).

Check yourself

A RAG assistant answers 62% of test questions correctly. Logs show a gold chunk was in the prompt for 64% of questions, and the answer was right in 93% of those cases. What should you fix first?

Check yourself

In the budget figure, why does accuracy fall when k rises from 5 to 30 with the reranker on?

Remember

  • ✓Two pipelines: offline parse, chunk, embed, index (with metadata and ACLs); online rewrite, retrieve, rerank, prompt, generate with citations.
  • ✓P(correct)≈R@k pread(k)+(1−R@k) pclosedP(\text{correct})\approx R@k\,p_{\text{read}}(k)+(1-R@k)\,p_{\text{closed}}: retrieval recall caps quality.
  • ✓More chunks raise recall but cost tokens and latency and can lower reading accuracy; a reranker lets k shrink.
  • ✓Latency budget: embed + retrieve + n tpairn\,t_{\text{pair}} + prefill of T0+kTchunkT_0+kT_{\text{chunk}} tokens.
  • ✓Filter by access before ranking, treat retrieved text as untrusted, re-index on change, and evaluate retrieval and answers separately.
← back to Generative AI roadmap