Agentic RAG

Let the LLM control retrieval: route each query, decompose it into hops, retrieve, grade what came back, correct misses, and stop when a judge says the evidence suffices. Multi-hop accuracy rises; every extra step costs an LLM call.

8 min read762 wordsadvanced0 viewsupdated 2026-10-21

Definition

Agentic RAG is retrieval-augmented generation in which the model decides whether, what and how often to retrieve. Instead of one fixed retrieve-then-read pass, it runs a loop: plan a sub-query, retrieve, grade the passages, reflect on whether the evidence is enough, and either answer or go round again. Key papers: IRCoT (Trivedi et al., ACL 2023), FLARE (Jiang et al., EMNLP 2023), Self-RAG (Asai, Wu, Wang, Sil and Hajishirzi, 2023; ICLR 2024), Corrective RAG (CRAG, Yan et al., 2024) and Adaptive-RAG (Jeong et al., NAACL 2024).

Classic RAG (see RAG architecture) retrieves once with the user's question. That fails in two common cases. First, multi-hop questions: "Which university did the director of Oppenheimer (2023) attend?" The answer passage mentions Christopher Nolan, not Oppenheimer, so the original query rarely finds it. Second, bad retrieval: when the top-kk chunks are irrelevant, a one-pass system still answers from them.

The fix is to put retrieval inside the agent loop (see The agent loop and ReAct prompting). Retrieval becomes a tool the model calls, with the user's question, a rewritten query or a sub-question. Its results are checked before they are used. The cost is latency and tokens: every plan, grade and judge step is another model call.

Q: Which university did thedirector of Oppenheimer(2023) attend?router(small classifier)no retrieval → LLM answersone retrieval → answermulti-hop loop (chosen)1 plan / decomposehop 1: who directedOppenheimer (2023)?2 retrieve top-krewritten sub-query,BM25 + dense index3 grade passagescorrect → keep stripsambiguous → keep + webincorrect → web/rewrite4 reflect / judgesupported? enough?no → plan next hopyes → write answerhop 2: where didNolan study?evidence scratchpad[1] Oppenheimer (2023)is directed byChristopher Nolan.[2] Nolan studied Englishliterature at UCL.stop whenjudge score ≥ τ, or Bhops used, or no new factA: University College London [1][2] · 2 hops, 2 retrievals, every hop citedSelf-RAG (Asai et al. 2023): one model emits reflection tokens itself[Retrieve] yes / no / continue · [IsRel] relevant / irrelevant[IsSup] fully / partially / no support · [IsUse] 5 … 1
fig.One two-hop question through an agentic RAG loop. A router picks the multi-hop path. Hop 1 retrieves "Oppenheimer is directed by Christopher Nolan"; the judge says the evidence is incomplete, so hop 2 asks where Nolan studied. The grader keeps, supplements or replaces each retrieval. The loop stops when the judge is satisfied, the budget runs out or nothing new is found, and the answer cites both hops.

Four control decisions

DecisionMechanismExample method
Route: retrieve at all? single or multi-step?classifier predicts query complexityAdaptive-RAG: a small T5 classifier picks no retrieval, single-step or multi-step
Decompose: what to retrieve next?generate a follow-up sub-questionSelf-Ask ("Are follow up questions needed here:"); IRCoT uses each new chain-of-thought sentence as the next query
Grade: is this passage usable?relevance evaluator on each passageCRAG: a fine-tuned T5-large scores passages as Correct, Ambiguous or Incorrect
Stop: is the evidence enough?support/sufficiency check or confidenceSelf-RAG reflection tokens; FLARE retrieves only when an upcoming token is low-probability
What the agent decides, and the method that introduced or popularised each decision

CRAG's three actions are concrete. Correct: keep the passages, but split them into short strips and drop irrelevant strips ("decompose-then-recompose"). Incorrect: discard them and run a web search with a rewritten query. Ambiguous: combine both. Query rewriting itself is covered in Query rewriting and HyDE.

Self-RAG: reflection tokens

Self-RAG trains one LM (Llama 2 7B and 13B in the paper) to emit special tokens alongside its text. Retrieve (yes / no / continue) decides whether to fetch passages before the next segment. IsRel marks a passage relevant or irrelevant. IsSup says whether the segment is fully, partially or not supported by it. IsUse rates overall usefulness from 5 to 1. The training data was labelled offline by a critic model, so no separate grader runs at inference.

S(critique)=∑G∈{IsRel, IsSup, IsUse}wG sG,sG=p(r^G)∑i=1NGp(riG)\mathcal{S}(\text{critique})=\sum_{G\in\{\text{IsRel},\,\text{IsSup},\,\text{IsUse}\}} w^{G}\,s^{G},\qquad s^{G}=\frac{p\big(\hat r^{G}\big)}{\sum_{i=1}^{N^G}p\big(r_i^{G}\big)}
r^G\hat r^{G}
the desirable token of group G (relevant, fully supported, usefulness 5)
NGN^G
number of tokens in the group; s is the normalised probability of the good one
wGw^{G}
weights set at inference: raising the IsSup weight trades fluency for groundedness without retraining

At inference Self-RAG retrieves KK passages, generates one candidate segment per passage in parallel, and keeps the candidate with the best combination of segment likelihood and S\mathcal{S} (a segment-level beam search). Retrieval itself fires when the normalised probability of Retrieve = yes exceeds a threshold, so retrieval frequency is a tunable knob.

Why loops win on multi-hop questions

A chain of hh hops succeeds only if every hop's fact is found. With per-hop recall pip_i, one pass gives ∏ipi\prod_i p_i, and the later pip_i are small because the original query does not name the bridge entity. A loop re-queries with the bridge entity and can retry a missed hop. With per-try success pp, a perfect judge and a budget of BB retrievals:

P(all h facts within B)=∑j=hB(j−1h−1) ph (1−p) j−hP(\text{all } h \text{ facts within } B)=\sum_{j=h}^{B}\binom{j-1}{h-1}\,p^{h}\,(1-p)^{\,j-h}
jj
the retrieval on which the hh-th fact is found (negative binomial)
pp
probability one rewritten sub-query finds the missing fact
BB
retrieval budget; the cap that bounds cost and latency

For p=0.75p=0.75 and h=2h=2: B=2B=2 gives 0.56250.5625, B=3B=3 gives 0.84380.8438, B=4B=4 gives 0.94920.9492. A single pass where the second hop has recall 0.20.2 gives 0.75×0.2=0.150.75\times0.2=0.15. Real judges are noisy, so the loop also stops too early or runs past the point of need. The figure simulates exactly that.

Reference points

IRCoT
Trivedi et al., ACL 2023. Interleaves retrieval with chain-of-thought: each new reasoning sentence becomes the next query. Evaluated on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC
FLARE
Jiang et al., EMNLP 2023. Drafts the next sentence; if any token falls below a probability threshold, uses the draft as a query, retrieves and regenerates the sentence
Self-RAG
Asai et al., arXiv October 2023, ICLR 2024. Four reflection-token types; retrieval on demand; critique-weighted segment beam search
CRAG
Yan et al., 2024. T5-large retrieval evaluator, three actions (Correct, Incorrect, Ambiguous), web search as fallback; plugs into an existing RAG pipeline
Adaptive-RAG
Jeong et al., NAACL 2024. Routes by predicted complexity: no retrieval, single-step, or multi-step iterative retrieval
200 questions · hops 0/1/2/3: 36/79/57/28accuracy vs LLM calls per question (B = 1…5)0.000.250.500.751.0001234B=1B=4single shotaccuracy by hops: single shot vs current92920 hops67851 hop33792 hops7433 hopscurrent: accuracy 0.785 · LLM calls/q 2.92 · retrievals/q 1.92 (single shot 0.535 at 1.00)why each question stoppedrouted 0single shot 0stop, complete 167stop, too early 23hit budget 10trace of question 2 (needs 2 hops; τ = 0.55): retrieve → grade → judge “enough evidence?”step 1fact foundevidence 1/2judge 0.53 gostep 2missevidence 1/2judge 0.27 gostep 3missevidence 1/2judge 0.65 stopanswerwrong4 LLM calls
loop
fig.200 synthetic questions needing 0–3 hops (36/79/57/28). A rewritten sub-query finds a missing fact with p = 0.75; the original query finds a later hop with p = 0.2. Default (iterative, B = 4, τ = 0.55): accuracy 0.785 at 2.92 LLM calls per question, against 0.535 at 1.00 for single shot; 23 questions stop too early. Question 2's trace shows one: the judge scores 0.65 with only 1 of 2 facts. Turn on corrective: 0.830 at 3.20 calls. Raise τ to 0.80 (corrective off): 0.860 at 4.20 calls, and 88 questions hit the budget. Router on: 0.725 at 2.60 calls, because 13 multi-hop questions are wrongly sent to no retrieval (35 of the 36 zero-hop ones are routed correctly). With corrective on, question 2's third step misses, the grader catches it, and web search supplies the second fact.

Worked example: scoring two Self-RAG candidates

Weights $w^{\text{IsRel}}=1,\;w^{\text{IsSup}}=1,\;w^{\text{IsUse}}=0.5$

  1. 1Candidate A (from passage d1d_1): normalised sIsRel=0.9s^{\text{IsRel}}=0.9, sIsSup=0.7s^{\text{IsSup}}=0.7 (fully supported), sIsUse=0.6s^{\text{IsUse}}=0.6 (usefulness 5). SA=0.9+0.7+0.5×0.6=1.90\mathcal{S}_A=0.9+0.7+0.5\times0.6=1.90.
  2. 2Candidate B (from d2d_2): 0.60.6, 0.30.3, 0.50.5. SB=0.6+0.3+0.5×0.5=1.15\mathcal{S}_B=0.6+0.3+0.5\times0.5=1.15.
  3. 3If the two segments have similar likelihood, A wins by 0.750.75. Set wIsSup=3w^{\text{IsSup}}=3: SA=0.9+2.1+0.3=3.30\mathcal{S}_A=0.9+2.1+0.3=3.30 and SB=0.6+0.9+0.25=1.75\mathcal{S}_B=0.6+0.9+0.25=1.75, so the gap widens to 1.551.55 and support dominates the choice.
python
1def agentic_rag(question, B=4, tau=0.55):
2 if classify(question) == "no_retrieval": # router, e.g. a small classifier
3 return llm(f"Answer: {question}")
4 notes, query = [], question
5 for hop in range(B):
6 passages = search(query, k=5)
7 grade = llm(f"Rate relevance of these passages to '{query}' as "
8 f"correct/ambiguous/incorrect:\n{passages}")
9 if grade != "correct": # CRAG-style correction
10 passages += web_search(llm(f"Rewrite as a search query: {query}"))
11 notes += passages
12 verdict = llm(f"Question: {question}\nEvidence: {notes}\n"
13 "Return JSON {\"enough\": p, \"next_subquestion\": str}")
14 if verdict["enough"] >= tau or not verdict["next_subquestion"]:
15 break # judge stop or nothing left to ask
16 query = verdict["next_subquestion"] # e.g. "Where did Christopher Nolan study?"
17 return llm(f"Answer from the evidence only, cite [n].\n{notes}\nQ: {question}")
An agentic RAG loop with routing, grading, corrective fallback and a stopping rule (sketch: llm, classify, search and web_search stand for your model, router and retrievers; not run here)

Use when

  • +Answers multi-hop and comparison questions that one retrieval cannot reach
  • +Detects and recovers from bad retrieval (grading, rewriting, web fallback) instead of answering from noise
  • +Retrieval frequency, support weight and budget are inference-time knobs

Avoid when

  • −Roughly 2–5× the LLM calls (and latency) of single-shot RAG in the figure's setting
  • −A noisy sufficiency judge causes premature stops or wasted hops
  • −Harder to debug and evaluate: per-step traces and stop reasons must be logged

Check yourself

Per-try retrieval success is p = 0.75 and the judge is perfect. What is the probability of collecting both facts of a 2-hop question within B = 3 retrievals?

Check yourself

In the figure, lowering the stop threshold τ from 0.55 to 0.30 (B = 4, corrective off) mainly does what?

Remember

  • ✓Agentic RAG puts retrieval inside a loop the LLM controls: route, decompose, retrieve, grade, reflect, stop.
  • ✓Multi-hop success is a product over hops; re-querying with the bridge entity and retrying misses is what lifts it.
  • ✓Self-RAG: Retrieve, IsRel, IsSup, IsUse tokens; candidates ranked by ∑GwGsG\sum_G w^G s^G plus likelihood.
  • ✓CRAG grades retrieval as Correct / Ambiguous / Incorrect and falls back to web search; Adaptive-RAG routes by query complexity.
  • ✓Stopping rules (judge score ≥τ\ge\tau, hop budget BB, no new evidence) set the accuracy–cost trade-off.
  • ✓Log stop reasons and tune thresholds on a labelled eval set; cap hops and treat retrieved text as untrusted.
← back to Generative AI roadmap