# GRPO

> GRPO (Shao et al., DeepSeekMath 2024) is PPO without a critic. It samples G answers per prompt, gives each the advantage (r − mean)/std of its group, and maximises a clipped ratio objective minus β times a k3 KL estimate against a frozen reference. DeepSeek-R1 trained it on verifiable rewards.

Source: https://www.mlfoundry.in/learn/grpo  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**GRPO:** (Group Relative Policy Optimization) is the policy-gradient method introduced by Shao, Wang, Zhu, Xu, Song, Zhang, Li, Wu and Guo in "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (DeepSeek-AI, 2024). For each prompt $q$ it samples a **group** of $G$ outputs from the current policy, scores them, and uses each output's reward relative to the group, $\hat A_i=(r_i-\mathrm{mean}(r))/\mathrm{std}(r)$, as the advantage of every one of its tokens. It keeps PPO's clipped probability ratio and adds a per-token KL penalty to a frozen reference model. It has **no value network**.

**RLHF** in the InstructGPT style fine-tunes the language model with PPO. PPO needs an advantage for every token, and it gets one from a learned **value model** $V_\psi$ and generalised advantage estimation (GAE). For an LLM the value model is usually another transformer about the size of the policy. Training it costs memory and compute, and it is hard to train well: the reward arrives only at the last token, yet the value model must predict it at every earlier token.

The fix is an old one from REINFORCE: use other samples as the baseline. Kool, van Hoof and Welling ("Buy 4 REINFORCE Samples, Get a Baseline for Free!", 2019) used the mean reward of the other samples for the same input. Ahmadian et al. (2024) brought this leave-one-out baseline (RLOO) back to RLHF. GRPO puts the same idea inside PPO's clipped objective and also divides by the group's standard deviation. The DeepSeekMath authors point out that this suits reward models, which are usually trained on *comparisons* between outputs for the same question.

## The objective

$$
\mathcal J_{GRPO}(\theta)=\mathbb E_{q,\{o_i\}_{i=1}^G\sim\pi_{\theta_{old}}(\cdot\mid q)}\frac1G\sum_{i=1}^G\frac1{|o_i|}\sum_{t=1}^{|o_i|}\Big\{\min\big[\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_{i,t}\big]-\beta\,\mathbb D_{KL}\big[\pi_\theta\|\pi_{ref}\big]\Big\}
$$

- $\rho_{i,t}$: $\pi_\theta(o_{i,t}\mid q,o_{i,<t})/\pi_{\theta_{old}}(o_{i,t}\mid q,o_{i,<t})$, the probability ratio against the policy that sampled the group
- $\hat A_{i,t}$: advantage of token $t$ of output $i$; with outcome rewards it is the same for all tokens of $o_i$
- $\mathrm{clip}(\cdot,1-\varepsilon,1+\varepsilon)$: PPO's trust region: once the ratio leaves $[1-\varepsilon,1+\varepsilon]$ in the direction $\hat A$ favours, the token's gradient is zero
- $\tfrac1{|o_i|}$: each output's tokens are averaged over its own length, then outputs are averaged over the group
- $\beta$: KL weight; 0.04 in DeepSeekMath
- $\pi_{ref}$: frozen reference policy, the SFT model the run started from

$$
\hat A_{i,t}=\tilde r_i=\frac{r_i-\mathrm{mean}(r_1,\dots,r_G)}{\mathrm{std}(r_1,\dots,r_G)}\quad\text{(outcome)},\qquad \hat A_{i,t}=\sum_{\text{steps }j:\ \mathrm{index}(j)\ge t}\tilde r_i^{\,\mathrm{index}(j)}\quad\text{(process)}
$$

- $r_i$: scalar reward of output $i$: a reward model's score in DeepSeekMath, a rule-based check in DeepSeek-R1
- $\tilde r_i$: reward standardised within the group: the group mean is the baseline, the group std sets the scale
- $\tilde r_i^{\,\mathrm{index}(j)}$: with process supervision, each reasoning step gets a reward, all step rewards in the group are normalised the same way, and a token's advantage is the sum of normalised rewards of the steps from its own onward

Because the advantage is standardised, the reward's scale and offset do not matter. A verifier that returns 0/1 and one that returns 0/10 give the same update. What matters is the ranking of the $G$ outputs and how far apart they are.

> Figure: One GRPO step. The old policy samples G outputs for a prompt. A rule-based verifier (DeepSeek-R1) or a reward model (DeepSeekMath) scores each one. Rewards are standardised inside the group, and every token of an output shares its advantage. The loss combines PPO's clipped ratio with a per-token k3 KL estimate against the frozen reference. The value model, which PPO would train alongside the policy, is gone.

## The KL term: the k3 estimator

PPO-based RLHF subtracts a per-token KL penalty from the reward, so the KL passes through the value model and GAE. GRPO puts the KL straight into the loss and estimates it per sampled token with an estimator from John Schulman's note "Approximating KL Divergence" (2020), often called **k3**:

$$
\mathbb D_{KL}\big[\pi_\theta\|\pi_{ref}\big]\approx k_3=\frac{\pi_{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)}-\log\frac{\pi_{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)}-1,\qquad \mathbb E_{o\sim\pi_\theta}\Big[\tfrac{\pi_{ref}}{\pi_\theta}\Big]=1
$$

- $x=\pi_{ref}/\pi_\theta$: likelihood ratio of the sampled token under the reference and the current policy
- $x-\log x-1\ge0$: always non-negative, since $\log x\le x-1$; zero only when $x=1$
- $\mathbb E[x]=1$: so $\mathbb E[k_3]=\mathbb E[-\log x]=\mathrm{KL}(\pi_\theta\|\pi_{ref})$: unbiased, with lower variance than the plain $-\log x$ estimator, which can be negative per sample

The DeepSeekMath paper writes every method's gradient in one form, $\mathbb E\big[\tfrac1{|o|}\sum_t GC(q,o,t)\,\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})\big]$, and compares the **gradient coefficient** $GC$. For GRPO (on-policy, ignoring the clip), differentiating $-\beta k_3$ gives $\beta(x-1)\nabla\log\pi_\theta$, so $GC=\hat A_{i,t}+\beta\big(\pi_{ref}/\pi_\theta-1\big)$. The KL term pushes a token up when the reference likes it more than the policy does ($x>1$) and down otherwise. For SFT the coefficient is 1 on every token. For rejection-sampling fine-tuning it is 1 on correct outputs and 0 on wrong ones. GRPO, like PPO, also gives *negative* weight to wrong outputs, scaled by how far below the group they fall.

**DeepSeekMath 7B and the R1 runs**

- **Base:** DeepSeekMath-Base 7B: DeepSeek-Coder-Base-v1.5 7B trained for another 500B tokens, including the 120B-token DeepSeekMath Corpus mined from Common Crawl
- **RL data:** about 144K chain-of-thought questions related to GSM8K and MATH, taken from the SFT data; RL starts from DeepSeekMath-Instruct 7B
- **GRPO settings:** $G=64$ outputs per question, $\beta=0.04$, policy learning rate $10^{-6}$, max length 1024, batch size 1024, one policy update per sampling round
- **Reward model:** trained from DeepSeekMath-Base 7B with learning rate $2\cdot10^{-5}$; iterative GRPO retrains it on fresh policy samples with 10% replay of old data
- **Results (no tools):** MATH: 46.8% Instruct → 51.7% RL; GSM8K: 82.9% → 88.2%; self-consistency over 64 samples reaches 60.9% on MATH
- **DeepSeek-R1-Zero (2025):** GRPO applied directly to DeepSeek-V3-Base with rule-based accuracy and format rewards and no SFT; AIME 2024 pass@1 rose from 15.6% to 71.0% (86.7% with majority voting)
- **DeepSeek-R1:** cold-start SFT, reasoning GRPO with an added language-consistency reward, about 800K rejection-sampled SFT examples, then a final RL stage; AIME 2024 pass@1 79.8%, MATH-500 97.3%

## Verifiable rewards and DeepSeek-R1

DeepSeek-R1 ("DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", DeepSeek-AI, 2025) made GRPO the default RL algorithm for reasoning models. R1-Zero used no neural reward model at all. Its **accuracy reward** checks the final answer of a math problem, given in a required format such as a box, or runs code against test cases. Its **format reward** checks that the reasoning sits between `<think>` and `</think>` tags. The authors avoided neural reward models here because the policy can learn to exploit them (reward hacking) over a long run.

Under this RL, R1-Zero's responses grew steadily longer, and it learned to re-check and revise its own work, including the "aha moment" the paper shows. Its outputs were hard to read and mixed languages, so DeepSeek-R1 added a cold-start SFT stage and a language-consistency reward. Binary verifiable rewards fit GRPO well. With 0/1 rewards and $k$ correct out of $G$, every correct answer gets $\sqrt{(G-k)/k}$ and every wrong one gets $-\sqrt{k/(G-k)}$: rare successes are pushed up hard.

DeepSeekMath also reports a caution. RL improved Maj@K but not Pass@K. The authors read this as RL making the output distribution more robust, raising the correct answers already among the model's top-K samples, rather than adding new basic capabilities.

> Figure: One prompt and a group of G sampled answers, each correct (reward 1) with probability p. In this toy, wrong answers are drawn longer than right ones. At the default (G = 8, p = 0.4, sample 5), 3 of 8 are correct: mean reward 0.375, std 0.484, so each correct answer gets Â = +1.291 and each wrong one −0.775. Dividing by length makes the per-token push uneven: the 312-token wrong answer gets −2.48·10⁻³ per token, the 836-token one only −0.93·10⁻³, which is 2.68× less. Long wrong answers are penalised least. Switch to Dr. GRPO (no ÷std, no ÷|oᵢ|): advantages become +0.625 and −0.375, every wrong token gets the same −0.37·10⁻³, and answer-level pushes grow with length (wrong −0.31 … −0.11). The right panel plots the clipped surrogate for the selected answer: with Â < 0 the gradient is zero once ρ < 0.8. The lower panel shows the total |advantage| a prompt contributes against the number correct k. With ÷std a prompt with k = 1 gets 5.29 against 8.00 at k = 4 (0.66×). Without it, 1.75 against 4.00 (0.44×). This is the difficulty bias Dr. GRPO removes. At p = 0.1 sample 5 is all wrong, so every advantage is 0; with G = 8 that happens for 43.0% of prompts (0.017 at p = 0.4 and G = 8, under 0.001 at p = 0.4 and G = 16).

## Worked example: a group of four

One math question, $G=4$ sampled answers, a 0/1 verifier. Only the first is correct. Advantages use the population std, as in the formula. TRL's implementation uses the Bessel-corrected `torch.std`, which would give $(1.5,-0.5,-0.5,-0.5)$ instead.

Group statistics: mean 0.25, std 0.433 (computed in Python)

| output | reward rᵢ | length \|oᵢ\| | Âᵢ | GRPO per-token Âᵢ/\|oᵢ\| | Dr. GRPO per-token (rᵢ − r̄)/1024 |
| --- | --- | --- | --- | --- | --- |
| o₁ | 1 | 120 | +1.732 | +0.01443 | +0.000732 |
| o₂ | 0 | 300 | −0.577 | −0.00192 | −0.000244 |
| o₃ | 0 | 80 | −0.577 | −0.00722 | −0.000244 |
| o₄ | 0 | 500 | −0.577 | −0.00115 | −0.000244 |

**From rewards to one token's loss**

1. Group statistics: $\bar r=0.25$ and $\mathrm{std}=\sqrt{0.25\cdot0.75}=0.433$. So $\hat A_1=0.75/0.433=+1.732$ and $\hat A_{2,3,4}=-0.25/0.433=-0.577$. The advantages sum to zero, as they always do.
2. Length normalisation: each wrong answer's tokens together receive $-0.577$. The 500-token $o_4$ spreads it to $-0.00115$ per token, the 80-token $o_3$ concentrates it to $-0.00722$, which is 6.25 times more per token. Under GRPO, rambling is the cheaper way to be wrong.
3. Clipping a good token: a token of $o_1$ with $\rho=1.3$ gives $\min(1.3\cdot1.732,\ 1.2\cdot1.732)=\min(2.252,2.078)=2.078$. The clipped branch is active, so this token gives no more gradient this round.
4. Clipping a bad token: a token of $o_2$ with $\rho=0.7$ gives $\min(0.7\cdot(-0.577),\ 0.8\cdot(-0.577))=\min(-0.404,-0.462)=-0.462$. That is clipped too: its probability has already fallen 30%, past the $1-\varepsilon$ bound.
5. KL for a token with $\pi_\theta=0.5$ and $\pi_{ref}=0.4$: $x=0.8$ and $k_3=0.8-\ln0.8-1=0.0231$, weighted by $\beta=0.04$. Check of unbiasedness on a 3-token vocabulary with $\pi_\theta=(0.5,0.3,0.2)$ and $\pi_{ref}=(0.4,0.4,0.2)$: $\mathbb E_{\pi_\theta}[k_3]=0.02527=\mathrm{KL}(\pi_\theta\|\pi_{ref})$.
6. Dr. GRPO on the same group: advantages $(0.75,-0.25,-0.25,-0.25)$ divided by a constant 1024. Every wrong token now gets $-0.000244$, so the 500-token wrong answer receives $500/80=6.25$ times the total push of the 80-token one.

## Known issues and fixes

**Length bias.** Liu et al. ("Understanding R1-Zero-Like Training: A Critical Perspective", 2025) note that the $1/|o_i|$ factor is not neutral. For a correct answer ($\hat A>0$) shorter outputs get a larger per-token reward. For a wrong answer ($\hat A<0$) longer outputs get a smaller per-token penalty. The optimiser is therefore nudged toward longer wrong answers, which can show up as steadily growing response length that is not all useful reasoning.

**Difficulty bias.** Dividing by $\mathrm{std}(r)$ up-weights prompts whose rewards barely vary, those that are almost always or almost never solved. With 0/1 rewards a prompt with $k=1$ of 8 correct contributes $\sum|\hat A_i|=5.29$ against $8$ at $k=4$. Without the std it would be 1.75 against 4. Their **Dr. GRPO** ("GRPO done right") removes both factors: advantages are $r_i-\mathrm{mean}(r)$, and token losses are summed and divided by a constant such as the generation budget. This makes the objective an unbiased policy gradient again.

**Dead groups.** If all $G$ rewards are equal, every advantage is zero and the prompt contributes nothing but KL. With pass rate $p$ that happens with probability $p^G+(1-p)^G$, which is large for very easy and very hard prompts. DAPO (Yu et al., ByteDance Seed, 2025) over-samples and filters such groups ("dynamic sampling"). It also raises only the upper clip bound ($\varepsilon_{low}=0.2$, $\varepsilon_{high}=0.28$) so that unlikely tokens can grow, averages the loss over all tokens in the batch instead of per output, shapes the reward for over-long outputs, and drops the KL term. With these changes it reports 50 points on AIME 2024 from Qwen2.5-32B base.

```python
import re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
policy = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)
ref = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16).eval().requires_grad_(False)
opt = torch.optim.AdamW(policy.parameters(), lr=1e-6)
G, eps_clip, beta = 8, 0.2, 0.04

def reward(text, gold):                     # verifiable 0/1 reward: boxed answer must match
    m = re.search(r"\\boxed\{([^}]*)\}", text)
    return float(m is not None and m.group(1).strip() == gold)

def token_logprobs(model, seq, P):          # log π(o_t | q, o_<t) for the completion tokens
    logits = model(seq).logits[:, P - 1:-1].float()
    return torch.log_softmax(logits, -1).gather(-1, seq[:, P:, None]).squeeze(-1)

def group_advantages(r, scale_std=True, eps=1e-4):
    a = r - r.mean()
    return a / (r.std() + eps) if scale_std else a   # torch.std is Bessel-corrected

prompt, gold = "What is 7 * 8? Put the final answer in \\boxed{}.", "56"
enc = tok(tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
          add_generation_prompt=True), return_tensors="pt")
P = enc.input_ids.shape[1]

with torch.no_grad():                       # 1. sample a group from π_θold
    seq = policy.generate(**enc, do_sample=True, temperature=1.0, top_p=1.0,
                          max_new_tokens=512, num_return_sequences=G, pad_token_id=tok.eos_token_id)
    comp = seq[:, P:]
    is_eos = comp == tok.eos_token_id       # keep tokens up to and including the first EOS
    mask = ((is_eos.cumsum(1) - is_eos.long()) == 0).float()
    r = torch.tensor([reward(tok.decode(c, skip_special_tokens=True), gold) for c in comp])
    adv = group_advantages(r)               # r = [1,0,0,0] would give ≈ [1.5, -0.5, -0.5, -0.5]
    logp_old = token_logprobs(policy, seq, P)
    logp_ref = token_logprobs(ref, seq, P)

# 2. one policy update (DeepSeekMath does a single update per sampling round)
logp = token_logprobs(policy, seq, P)
ratio = torch.exp(logp - logp_old)
a = adv[:, None]
surr = torch.min(ratio * a, ratio.clamp(1 - eps_clip, 1 + eps_clip) * a)
log_x = logp_ref - logp                     # log(π_ref / π_θ)
k3 = torch.exp(log_x) - log_x - 1           # ≥ 0, unbiased for KL(π_θ ‖ π_ref)
per_token = -(surr - beta * k3) * mask
loss = (per_token.sum(1) / mask.sum(1)).mean()          # GRPO: 1/|o_i| per output, then 1/G
# Dr. GRPO instead: loss = per_token.sum() / (G * 512), with adv = group_advantages(r, False)
opt.zero_grad(); loss.backward(); opt.step()
```

One GRPO step for a single prompt with Hugging Face transformers: sample G outputs, verify, standardise within the group, clipped loss with the k3 KL (not run here; PyTorch)

> **Most prompts may teach nothing:** A group whose rewards are all equal has zero advantages. At pass rate 0.1 and $G=8$, 43% of groups are all wrong, so the effective batch is far smaller than the nominal one, and gradient noise rises. Monitor the fraction of zero-std groups. Fix it with a larger $G$, curriculum filtering, or DAPO-style resampling. Do not try to fix it with a bigger learning rate. Add an $\epsilon$ to the std, or the all-equal case divides zero by zero.

> **The group mean is a Monte Carlo value estimate:** PPO's critic estimates $V(q)$, the expected reward from the prompt. GRPO estimates the same quantity by averaging $G$ samples, so $r_i-\bar r$ is a sampled advantage at the sequence level. It is noisier per token than GAE and has no credit assignment inside the output, but it costs no extra parameters. Without the std division it is an unbiased policy gradient up to a factor $(G-1)/G$, because $r_i$ sits in its own baseline. RLOO's leave-one-out mean removes even that.

> **No critic does not mean no extra models:** GRPO still keeps a frozen reference $\pi_{ref}$ for the KL term, a reward model unless the reward is a rule, and $\pi_{\theta_{old}}$'s log-probabilities from sampling time. What it removes is the trained value network, roughly a second copy of the policy with its own optimiser state. Many verifiable-reward recipes (DAPO, Dr. GRPO) set $\beta=0$ and drop the reference too.

**Strengths**

- No value model: saves about a policy-sized network, its optimiser state and its training instabilities
- Advantages are invariant to reward scale and offset, and fit comparative reward models and 0/1 verifiers
- Simple to implement on top of a PPO codebase; the KL sits directly in the loss
- Powered DeepSeekMath-RL and the DeepSeek-R1 reasoning models

**Limitations**

- Needs $G$ samples per prompt (64 in DeepSeekMath), so generation dominates the cost
- No credit assignment within an output under outcome rewards: every token gets the same advantage
- $1/|o_i|$ and $\div\mathrm{std}$ add length and difficulty biases (Dr. GRPO)
- Prompts the model always or never solves give no signal

- **Reasoning RL:** math and code with verifiable rewards: DeepSeek-R1-Zero, DeepSeek-R1 and many open reproductions
- **RL with a reward model:** DeepSeekMath-RL used a learned reward model with outcome or process supervision
- **Library support:** TRL's `GRPOTrainer`, OpenRLHF and verl implement GRPO and its variants

**Question:** A group of G = 8 answers has rewards all equal to 1. What does GRPO's policy-gradient term do on this prompt?

- A. Pushes up all eight answers equally
- B. Nothing: every advantage is (1 − 1)/std = 0, so only the KL term acts
- C. Pushes up the shortest answer most
- D. Divides by zero and has to be skipped by the optimiser

**Answer:** B. Nothing: every advantage is (1 − 1)/std = 0, so only the KL term acts — The group mean equals every reward, so all advantages are zero (implementations add $\epsilon$ to the std to avoid $0/0$). A prompt that is always solved, or never solved, gives no learning signal. That is why DAPO filters such groups and why pass rate matters when choosing training prompts.

**Question:** Under the original GRPO loss, why can wrong answers drift longer?

- A. The k3 KL estimator is smaller for long sequences
- B. The clip range widens with length
- C. Each output's loss is averaged over its own length, so a long wrong answer's fixed negative advantage is spread thinner per token
- D. The reward model prefers long answers

**Answer:** C. Each output's loss is averaged over its own length, so a long wrong answer's fixed negative advantage is spread thinner per token — All tokens of $o_i$ share $\hat A_i<0$ and the loss averages over $|o_i|$, so each token is penalised $\hat A_i/|o_i|$. Longer wrong outputs are penalised less per token. Dr. GRPO replaces $1/|o_i|$ with a constant normaliser.

**Key points**

- GRPO replaces PPO's value model with a group baseline: sample $G$ outputs per prompt and set every token's advantage to $(r_i-\mathrm{mean}\,r)/\mathrm{std}\,r$.
- Loss: $\frac1G\sum_i\frac1{|o_i|}\sum_t\big[\min(\rho\hat A,\mathrm{clip}(\rho,1\pm\varepsilon)\hat A)-\beta k_3\big]$ with $k_3=x-\log x-1$, $x=\pi_{ref}/\pi_\theta$, an unbiased non-negative KL estimate.
- DeepSeekMath-RL 7B used $G=64$, $\beta=0.04$ and lr $10^{-6}$, and reached 51.7% on MATH. DeepSeek-R1-Zero ran GRPO on rule-based rewards and raised AIME 2024 pass@1 from 15.6% to 71.0%.
- Known biases: $1/|o_i|$ penalises long wrong answers less, and $\div\mathrm{std}$ up-weights near-trivial and near-impossible prompts. Dr. GRPO removes both.
- All-equal groups give zero gradient, with probability $p^G+(1-p)^G$; DAPO resamples them, decouples the clip range and drops the KL.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
