Definition
Agentic RAG is retrieval-augmented generation in which the model decides whether, what and how often to retrieve. Instead of one fixed retrieve-then-read pass, it runs a loop: plan a sub-query, retrieve, grade the passages, reflect on whether the evidence is enough, and either answer or go round again. Key papers: IRCoT (Trivedi et al., ACL 2023), FLARE (Jiang et al., EMNLP 2023), Self-RAG (Asai, Wu, Wang, Sil and Hajishirzi, 2023; ICLR 2024), Corrective RAG (CRAG, Yan et al., 2024) and Adaptive-RAG (Jeong et al., NAACL 2024).
Classic RAG (see RAG architecture) retrieves once with the user's question. That fails in two common cases. First, multi-hop questions: "Which university did the director of Oppenheimer (2023) attend?" The answer passage mentions Christopher Nolan, not Oppenheimer, so the original query rarely finds it. Second, bad retrieval: when the top- chunks are irrelevant, a one-pass system still answers from them.
The fix is to put retrieval inside the agent loop (see The agent loop and ReAct prompting). Retrieval becomes a tool the model calls, with the user's question, a rewritten query or a sub-question. Its results are checked before they are used. The cost is latency and tokens: every plan, grade and judge step is another model call.
Four control decisions
| Decision | Mechanism | Example method |
|---|---|---|
| Route: retrieve at all? single or multi-step? | classifier predicts query complexity | Adaptive-RAG: a small T5 classifier picks no retrieval, single-step or multi-step |
| Decompose: what to retrieve next? | generate a follow-up sub-question | Self-Ask ("Are follow up questions needed here:"); IRCoT uses each new chain-of-thought sentence as the next query |
| Grade: is this passage usable? | relevance evaluator on each passage | CRAG: a fine-tuned T5-large scores passages as Correct, Ambiguous or Incorrect |
| Stop: is the evidence enough? | support/sufficiency check or confidence | Self-RAG reflection tokens; FLARE retrieves only when an upcoming token is low-probability |
CRAG's three actions are concrete. Correct: keep the passages, but split them into short strips and drop irrelevant strips ("decompose-then-recompose"). Incorrect: discard them and run a web search with a rewritten query. Ambiguous: combine both. Query rewriting itself is covered in Query rewriting and HyDE.
Self-RAG: reflection tokens
Self-RAG trains one LM (Llama 2 7B and 13B in the paper) to emit special tokens alongside its text. Retrieve (yes / no / continue) decides whether to fetch passages before the next segment. IsRel marks a passage relevant or irrelevant. IsSup says whether the segment is fully, partially or not supported by it. IsUse rates overall usefulness from 5 to 1. The training data was labelled offline by a critic model, so no separate grader runs at inference.
- the desirable token of group G (relevant, fully supported, usefulness 5)
- number of tokens in the group; s is the normalised probability of the good one
- weights set at inference: raising the IsSup weight trades fluency for groundedness without retraining
At inference Self-RAG retrieves passages, generates one candidate segment per passage in parallel, and keeps the candidate with the best combination of segment likelihood and (a segment-level beam search). Retrieval itself fires when the normalised probability of Retrieve = yes exceeds a threshold, so retrieval frequency is a tunable knob.
Why loops win on multi-hop questions
A chain of hops succeeds only if every hop's fact is found. With per-hop recall , one pass gives , and the later are small because the original query does not name the bridge entity. A loop re-queries with the bridge entity and can retry a missed hop. With per-try success , a perfect judge and a budget of retrievals:
- the retrieval on which the -th fact is found (negative binomial)
- probability one rewritten sub-query finds the missing fact
- retrieval budget; the cap that bounds cost and latency
For and : gives , gives , gives . A single pass where the second hop has recall gives . Real judges are noisy, so the loop also stops too early or runs past the point of need. The figure simulates exactly that.
Reference points
- IRCoT
- Trivedi et al., ACL 2023. Interleaves retrieval with chain-of-thought: each new reasoning sentence becomes the next query. Evaluated on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC
- FLARE
- Jiang et al., EMNLP 2023. Drafts the next sentence; if any token falls below a probability threshold, uses the draft as a query, retrieves and regenerates the sentence
- Self-RAG
- Asai et al., arXiv October 2023, ICLR 2024. Four reflection-token types; retrieval on demand; critique-weighted segment beam search
- CRAG
- Yan et al., 2024. T5-large retrieval evaluator, three actions (Correct, Incorrect, Ambiguous), web search as fallback; plugs into an existing RAG pipeline
- Adaptive-RAG
- Jeong et al., NAACL 2024. Routes by predicted complexity: no retrieval, single-step, or multi-step iterative retrieval
Worked example: scoring two Self-RAG candidates
Weights $w^{\text{IsRel}}=1,\;w^{\text{IsSup}}=1,\;w^{\text{IsUse}}=0.5$
- 1Candidate A (from passage ): normalised , (fully supported), (usefulness 5). .
- 2Candidate B (from ): , , . .
- 3If the two segments have similar likelihood, A wins by . Set : and , so the gap widens to and support dominates the choice.
1def agentic_rag(question, B=4, tau=0.55):2 if classify(question) == "no_retrieval": # router, e.g. a small classifier3 return llm(f"Answer: {question}")4 notes, query = [], question5 for hop in range(B):6 passages = search(query, k=5)7 grade = llm(f"Rate relevance of these passages to '{query}' as "8 f"correct/ambiguous/incorrect:\n{passages}")9 if grade != "correct": # CRAG-style correction10 passages += web_search(llm(f"Rewrite as a search query: {query}"))11 notes += passages12 verdict = llm(f"Question: {question}\nEvidence: {notes}\n"13 "Return JSON {\"enough\": p, \"next_subquestion\": str}")14 if verdict["enough"] >= tau or not verdict["next_subquestion"]:15 break # judge stop or nothing left to ask16 query = verdict["next_subquestion"] # e.g. "Where did Christopher Nolan study?"17 return llm(f"Answer from the evidence only, cite [n].\n{notes}\nQ: {question}")Use when
- +Answers multi-hop and comparison questions that one retrieval cannot reach
- +Detects and recovers from bad retrieval (grading, rewriting, web fallback) instead of answering from noise
- +Retrieval frequency, support weight and budget are inference-time knobs
Avoid when
- −Roughly 2–5× the LLM calls (and latency) of single-shot RAG in the figure's setting
- −A noisy sufficiency judge causes premature stops or wasted hops
- −Harder to debug and evaluate: per-step traces and stop reasons must be logged
Check yourself
Per-try retrieval success is p = 0.75 and the judge is perfect. What is the probability of collecting both facts of a 2-hop question within B = 3 retrievals?
Check yourself
In the figure, lowering the stop threshold τ from 0.55 to 0.30 (B = 4, corrective off) mainly does what?
Remember
- ✓Agentic RAG puts retrieval inside a loop the LLM controls: route, decompose, retrieve, grade, reflect, stop.
- ✓Multi-hop success is a product over hops; re-querying with the bridge entity and retrying misses is what lifts it.
- ✓Self-RAG: Retrieve, IsRel, IsSup, IsUse tokens; candidates ranked by plus likelihood.
- ✓CRAG grades retrieval as Correct / Ambiguous / Incorrect and falls back to web search; Adaptive-RAG routes by query complexity.
- ✓Stopping rules (judge score , hop budget , no new evidence) set the accuracy–cost trade-off.
- ✓Log stop reasons and tune thresholds on a labelled eval set; cap hops and treat retrieved text as untrusted.