# Agentic RAG

> Let the LLM control retrieval: route each query, decompose it into hops, retrieve, grade what came back, correct misses, and stop when a judge says the evidence suffices. Multi-hop accuracy rises; every extra step costs an LLM call.

Source: https://www.mlfoundry.in/learn/agentic-rag  
Section: Generative AI › Retrieval-Augmented Generation (RAG)  
Level: advanced · 8 min read · updated 2026-10-21

**Agentic RAG:** is retrieval-augmented generation in which the model decides *whether*, *what* and *how often* to retrieve. Instead of one fixed retrieve-then-read pass, it runs a loop: plan a sub-query, retrieve, grade the passages, reflect on whether the evidence is enough, and either answer or go round again. Key papers: IRCoT (Trivedi et al., ACL 2023), FLARE (Jiang et al., EMNLP 2023), **Self-RAG** (Asai, Wu, Wang, Sil and Hajishirzi, 2023; ICLR 2024), Corrective RAG (**CRAG**, Yan et al., 2024) and Adaptive-RAG (Jeong et al., NAACL 2024).

Classic RAG (see **RAG architecture**) retrieves once with the user's question. That fails in two common cases. First, multi-hop questions: "Which university did the director of *Oppenheimer* (2023) attend?" The answer passage mentions Christopher Nolan, not *Oppenheimer*, so the original query rarely finds it. Second, bad retrieval: when the top-$k$ chunks are irrelevant, a one-pass system still answers from them.

The fix is to put retrieval inside the agent loop (see **The agent loop** and **ReAct prompting**). Retrieval becomes a tool the model calls, with the user's question, a rewritten query or a sub-question. Its results are checked before they are used. The cost is latency and tokens: every plan, grade and judge step is another model call.

> Figure: One two-hop question through an agentic RAG loop. A router picks the multi-hop path. Hop 1 retrieves "Oppenheimer is directed by Christopher Nolan"; the judge says the evidence is incomplete, so hop 2 asks where Nolan studied. The grader keeps, supplements or replaces each retrieval. The loop stops when the judge is satisfied, the budget runs out or nothing new is found, and the answer cites both hops.

## Four control decisions

What the agent decides, and the method that introduced or popularised each decision

| Decision | Mechanism | Example method |
| --- | --- | --- |
| Route: retrieve at all? single or multi-step? | classifier predicts query complexity | Adaptive-RAG: a small T5 classifier picks no retrieval, single-step or multi-step |
| Decompose: what to retrieve next? | generate a follow-up sub-question | Self-Ask ("Are follow up questions needed here:"); IRCoT uses each new chain-of-thought sentence as the next query |
| Grade: is this passage usable? | relevance evaluator on each passage | CRAG: a fine-tuned T5-large scores passages as Correct, Ambiguous or Incorrect |
| Stop: is the evidence enough? | support/sufficiency check or confidence | Self-RAG reflection tokens; FLARE retrieves only when an upcoming token is low-probability |

CRAG's three actions are concrete. **Correct**: keep the passages, but split them into short strips and drop irrelevant strips ("decompose-then-recompose"). **Incorrect**: discard them and run a web search with a rewritten query. **Ambiguous**: combine both. Query rewriting itself is covered in **Query rewriting and HyDE**.

## Self-RAG: reflection tokens

Self-RAG trains one LM (Llama 2 7B and 13B in the paper) to emit special tokens alongside its text. **Retrieve** (yes / no / continue) decides whether to fetch passages before the next segment. **IsRel** marks a passage relevant or irrelevant. **IsSup** says whether the segment is fully, partially or not supported by it. **IsUse** rates overall usefulness from 5 to 1. The training data was labelled offline by a critic model, so no separate grader runs at inference.

$$
\mathcal{S}(\text{critique})=\sum_{G\in\{\text{IsRel},\,\text{IsSup},\,\text{IsUse}\}} w^{G}\,s^{G},\qquad s^{G}=\frac{p\big(\hat r^{G}\big)}{\sum_{i=1}^{N^G}p\big(r_i^{G}\big)}
$$

- $\hat r^{G}$: the desirable token of group G (relevant, fully supported, usefulness 5)
- $N^G$: number of tokens in the group; s is the normalised probability of the good one
- $w^{G}$: weights set at inference: raising the IsSup weight trades fluency for groundedness without retraining

At inference Self-RAG retrieves $K$ passages, generates one candidate segment per passage in parallel, and keeps the candidate with the best combination of segment likelihood and $\mathcal{S}$ (a segment-level beam search). Retrieval itself fires when the normalised probability of Retrieve = yes exceeds a threshold, so retrieval frequency is a tunable knob.

## Why loops win on multi-hop questions

A chain of $h$ hops succeeds only if every hop's fact is found. With per-hop recall $p_i$, one pass gives $\prod_i p_i$, and the later $p_i$ are small because the original query does not name the bridge entity. A loop re-queries with the bridge entity and can retry a missed hop. With per-try success $p$, a perfect judge and a budget of $B$ retrievals:

$$
P(\text{all } h \text{ facts within } B)=\sum_{j=h}^{B}\binom{j-1}{h-1}\,p^{h}\,(1-p)^{\,j-h}
$$

- $j$: the retrieval on which the $h$-th fact is found (negative binomial)
- $p$: probability one rewritten sub-query finds the missing fact
- $B$: retrieval budget; the cap that bounds cost and latency

For $p=0.75$ and $h=2$: $B=2$ gives $0.5625$, $B=3$ gives $0.8438$, $B=4$ gives $0.9492$. A single pass where the second hop has recall $0.2$ gives $0.75\times0.2=0.15$. Real judges are noisy, so the loop also stops too early or runs past the point of need. The figure simulates exactly that.

**Reference points**

- **IRCoT:** Trivedi et al., ACL 2023. Interleaves retrieval with chain-of-thought: each new reasoning sentence becomes the next query. Evaluated on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC
- **FLARE:** Jiang et al., EMNLP 2023. Drafts the next sentence; if any token falls below a probability threshold, uses the draft as a query, retrieves and regenerates the sentence
- **Self-RAG:** Asai et al., arXiv October 2023, ICLR 2024. Four reflection-token types; retrieval on demand; critique-weighted segment beam search
- **CRAG:** Yan et al., 2024. T5-large retrieval evaluator, three actions (Correct, Incorrect, Ambiguous), web search as fallback; plugs into an existing RAG pipeline
- **Adaptive-RAG:** Jeong et al., NAACL 2024. Routes by predicted complexity: no retrieval, single-step, or multi-step iterative retrieval

> Figure: 200 synthetic questions needing 0–3 hops (36/79/57/28). A rewritten sub-query finds a missing fact with p = 0.75; the original query finds a later hop with p = 0.2. Default (iterative, B = 4, τ = 0.55): accuracy 0.785 at 2.92 LLM calls per question, against 0.535 at 1.00 for single shot; 23 questions stop too early. Question 2's trace shows one: the judge scores 0.65 with only 1 of 2 facts. Turn on corrective: 0.830 at 3.20 calls. Raise τ to 0.80 (corrective off): 0.860 at 4.20 calls, and 88 questions hit the budget. Router on: 0.725 at 2.60 calls, because 13 multi-hop questions are wrongly sent to no retrieval (35 of the 36 zero-hop ones are routed correctly). With corrective on, question 2's third step misses, the grader catches it, and web search supplies the second fact.

## Worked example: scoring two Self-RAG candidates

**Weights $w^{\text{IsRel}}=1,\;w^{\text{IsSup}}=1,\;w^{\text{IsUse}}=0.5$**

1. Candidate A (from passage $d_1$): normalised $s^{\text{IsRel}}=0.9$, $s^{\text{IsSup}}=0.7$ (fully supported), $s^{\text{IsUse}}=0.6$ (usefulness 5). $\mathcal{S}_A=0.9+0.7+0.5\times0.6=1.90$.
2. Candidate B (from $d_2$): $0.6$, $0.3$, $0.5$. $\mathcal{S}_B=0.6+0.3+0.5\times0.5=1.15$.
3. If the two segments have similar likelihood, A wins by $0.75$. Set $w^{\text{IsSup}}=3$: $\mathcal{S}_A=0.9+2.1+0.3=3.30$ and $\mathcal{S}_B=0.6+0.9+0.25=1.75$, so the gap widens to $1.55$ and support dominates the choice.

```python
def agentic_rag(question, B=4, tau=0.55):
    if classify(question) == "no_retrieval":          # router, e.g. a small classifier
        return llm(f"Answer: {question}")
    notes, query = [], question
    for hop in range(B):
        passages = search(query, k=5)
        grade = llm(f"Rate relevance of these passages to '{query}' as "
                    f"correct/ambiguous/incorrect:\n{passages}")
        if grade != "correct":                         # CRAG-style correction
            passages += web_search(llm(f"Rewrite as a search query: {query}"))
        notes += passages
        verdict = llm(f"Question: {question}\nEvidence: {notes}\n"
                      "Return JSON {\"enough\": p, \"next_subquestion\": str}")
        if verdict["enough"] >= tau or not verdict["next_subquestion"]:
            break                                      # judge stop or nothing left to ask
        query = verdict["next_subquestion"]            # e.g. "Where did Christopher Nolan study?"
    return llm(f"Answer from the evidence only, cite [n].\n{notes}\nQ: {question}")
```

An agentic RAG loop with routing, grading, corrective fallback and a stopping rule (sketch: llm, classify, search and web_search stand for your model, router and retrievers; not run here)

> **Early stops and runaway loops:** The judge is the weak point. A lenient judge stops after one hop with half the evidence, as question 2 does in the figure. A strict one spends the whole budget on every query. Always cap hops and tokens, log the stop reason for each query, and tune $\tau$ on a labelled set with **RAG evaluation** metrics, not by eye. Every retrieved web page is also untrusted input to later steps (see **Prompt injection**).

> **Route first, loop second:** Most production traffic is single-hop. Routing simple queries to one retrieval (or none) keeps average cost near classic RAG and spends loops only where they help. The router's mistakes are asymmetric: sending a multi-hop question down the cheap path costs accuracy, the reverse only costs money.

**Strengths**

- Answers multi-hop and comparison questions that one retrieval cannot reach
- Detects and recovers from bad retrieval (grading, rewriting, web fallback) instead of answering from noise
- Retrieval frequency, support weight and budget are inference-time knobs

**Limitations**

- Roughly 2–5× the LLM calls (and latency) of single-shot RAG in the figure's setting
- A noisy sufficiency judge causes premature stops or wasted hops
- Harder to debug and evaluate: per-step traces and stop reasons must be logged

**Question:** Per-try retrieval success is p = 0.75 and the judge is perfect. What is the probability of collecting both facts of a 2-hop question within B = 3 retrievals?

- A. 0.5625
- B. 0.75
- C. 0.8438
- D. 0.9492

**Answer:** C. 0.8438 — Success on try 2: $p^2=0.5625$. Success on try 3 (one miss among the first two): $\binom{2}{1}p^2(1-p)=2\times0.5625\times0.25=0.28125$. Total $0.84375$. With $B=4$ it is $0.9492$.

**Question:** In the figure, lowering the stop threshold τ from 0.55 to 0.30 (B = 4, corrective off) mainly does what?

- A. Raises accuracy, because the agent reads more passages
- B. Cuts LLM calls but lets many questions stop with incomplete evidence, so accuracy falls
- C. Has no effect, since the budget is the binding constraint
- D. Makes the router skip retrieval

**Answer:** B. Cuts LLM calls but lets many questions stop with incomplete evidence, so accuracy falls — At τ = 0.30 the judge accepts almost any evidence: 82 of 200 questions stop too early, calls fall from 2.92 to 2.21 per question, and accuracy drops from 0.785 to 0.570, close to single shot.

**Key points**

- Agentic RAG puts retrieval inside a loop the LLM controls: route, decompose, retrieve, grade, reflect, stop.
- Multi-hop success is a product over hops; re-querying with the bridge entity and retrying misses is what lifts it.
- Self-RAG: Retrieve, IsRel, IsSup, IsUse tokens; candidates ranked by $\sum_G w^G s^G$ plus likelihood.
- CRAG grades retrieval as Correct / Ambiguous / Incorrect and falls back to web search; Adaptive-RAG routes by query complexity.
- Stopping rules (judge score $\ge\tau$, hop budget $B$, no new evidence) set the accuracy–cost trade-off.
- Log stop reasons and tune thresholds on a labelled eval set; cap hops and treat retrieved text as untrusted.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
