Definition
GRPO (Group Relative Policy Optimization) is the policy-gradient method introduced by Shao, Wang, Zhu, Xu, Song, Zhang, Li, Wu and Guo in "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (DeepSeek-AI, 2024). For each prompt it samples a group of outputs from the current policy, scores them, and uses each output's reward relative to the group, , as the advantage of every one of its tokens. It keeps PPO's clipped probability ratio and adds a per-token KL penalty to a frozen reference model. It has no value network.
RLHF in the InstructGPT style fine-tunes the language model with PPO. PPO needs an advantage for every token, and it gets one from a learned value model and generalised advantage estimation (GAE). For an LLM the value model is usually another transformer about the size of the policy. Training it costs memory and compute, and it is hard to train well: the reward arrives only at the last token, yet the value model must predict it at every earlier token.
The fix is an old one from REINFORCE: use other samples as the baseline. Kool, van Hoof and Welling ("Buy 4 REINFORCE Samples, Get a Baseline for Free!", 2019) used the mean reward of the other samples for the same input. Ahmadian et al. (2024) brought this leave-one-out baseline (RLOO) back to RLHF. GRPO puts the same idea inside PPO's clipped objective and also divides by the group's standard deviation. The DeepSeekMath authors point out that this suits reward models, which are usually trained on comparisons between outputs for the same question.
The objective
- , the probability ratio against the policy that sampled the group
- advantage of token of output ; with outcome rewards it is the same for all tokens of
- PPO's trust region: once the ratio leaves in the direction favours, the token's gradient is zero
- each output's tokens are averaged over its own length, then outputs are averaged over the group
- KL weight; 0.04 in DeepSeekMath
- frozen reference policy, the SFT model the run started from
- scalar reward of output : a reward model's score in DeepSeekMath, a rule-based check in DeepSeek-R1
- reward standardised within the group: the group mean is the baseline, the group std sets the scale
- with process supervision, each reasoning step gets a reward, all step rewards in the group are normalised the same way, and a token's advantage is the sum of normalised rewards of the steps from its own onward
Because the advantage is standardised, the reward's scale and offset do not matter. A verifier that returns 0/1 and one that returns 0/10 give the same update. What matters is the ranking of the outputs and how far apart they are.
The KL term: the k3 estimator
PPO-based RLHF subtracts a per-token KL penalty from the reward, so the KL passes through the value model and GAE. GRPO puts the KL straight into the loss and estimates it per sampled token with an estimator from John Schulman's note "Approximating KL Divergence" (2020), often called k3:
- likelihood ratio of the sampled token under the reference and the current policy
- always non-negative, since ; zero only when
- so : unbiased, with lower variance than the plain estimator, which can be negative per sample
The DeepSeekMath paper writes every method's gradient in one form, , and compares the gradient coefficient . For GRPO (on-policy, ignoring the clip), differentiating gives , so . The KL term pushes a token up when the reference likes it more than the policy does () and down otherwise. For SFT the coefficient is 1 on every token. For rejection-sampling fine-tuning it is 1 on correct outputs and 0 on wrong ones. GRPO, like PPO, also gives negative weight to wrong outputs, scaled by how far below the group they fall.
DeepSeekMath 7B and the R1 runs
- Base
- DeepSeekMath-Base 7B: DeepSeek-Coder-Base-v1.5 7B trained for another 500B tokens, including the 120B-token DeepSeekMath Corpus mined from Common Crawl
- RL data
- about 144K chain-of-thought questions related to GSM8K and MATH, taken from the SFT data; RL starts from DeepSeekMath-Instruct 7B
- GRPO settings
- outputs per question, , policy learning rate , max length 1024, batch size 1024, one policy update per sampling round
- Reward model
- trained from DeepSeekMath-Base 7B with learning rate ; iterative GRPO retrains it on fresh policy samples with 10% replay of old data
- Results (no tools)
- MATH: 46.8% Instruct → 51.7% RL; GSM8K: 82.9% → 88.2%; self-consistency over 64 samples reaches 60.9% on MATH
- DeepSeek-R1-Zero (2025)
- GRPO applied directly to DeepSeek-V3-Base with rule-based accuracy and format rewards and no SFT; AIME 2024 pass@1 rose from 15.6% to 71.0% (86.7% with majority voting)
- DeepSeek-R1
- cold-start SFT, reasoning GRPO with an added language-consistency reward, about 800K rejection-sampled SFT examples, then a final RL stage; AIME 2024 pass@1 79.8%, MATH-500 97.3%
Verifiable rewards and DeepSeek-R1
DeepSeek-R1 ("DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", DeepSeek-AI, 2025) made GRPO the default RL algorithm for reasoning models. R1-Zero used no neural reward model at all. Its accuracy reward checks the final answer of a math problem, given in a required format such as a box, or runs code against test cases. Its format reward checks that the reasoning sits between <think> and </think> tags. The authors avoided neural reward models here because the policy can learn to exploit them (reward hacking) over a long run.
Under this RL, R1-Zero's responses grew steadily longer, and it learned to re-check and revise its own work, including the "aha moment" the paper shows. Its outputs were hard to read and mixed languages, so DeepSeek-R1 added a cold-start SFT stage and a language-consistency reward. Binary verifiable rewards fit GRPO well. With 0/1 rewards and correct out of , every correct answer gets and every wrong one gets : rare successes are pushed up hard.
DeepSeekMath also reports a caution. RL improved Maj@K but not Pass@K. The authors read this as RL making the output distribution more robust, raising the correct answers already among the model's top-K samples, rather than adding new basic capabilities.
Worked example: a group of four
One math question, sampled answers, a 0/1 verifier. Only the first is correct. Advantages use the population std, as in the formula. TRL's implementation uses the Bessel-corrected torch.std, which would give instead.
| output | reward rᵢ | length |oᵢ| | Âᵢ | GRPO per-token Âᵢ/|oᵢ| | Dr. GRPO per-token (rᵢ − r̄)/1024 |
|---|---|---|---|---|---|
| o₁ | 1 | 120 | +1.732 | +0.01443 | +0.000732 |
| o₂ | 0 | 300 | −0.577 | −0.00192 | −0.000244 |
| o₃ | 0 | 80 | −0.577 | −0.00722 | −0.000244 |
| o₄ | 0 | 500 | −0.577 | −0.00115 | −0.000244 |
From rewards to one token's loss
- 1Group statistics: and . So and . The advantages sum to zero, as they always do.
- 2Length normalisation: each wrong answer's tokens together receive . The 500-token spreads it to per token, the 80-token concentrates it to , which is 6.25 times more per token. Under GRPO, rambling is the cheaper way to be wrong.
- 3Clipping a good token: a token of with gives . The clipped branch is active, so this token gives no more gradient this round.
- 4Clipping a bad token: a token of with gives . That is clipped too: its probability has already fallen 30%, past the bound.
- 5KL for a token with and : and , weighted by . Check of unbiasedness on a 3-token vocabulary with and : .
- 6Dr. GRPO on the same group: advantages divided by a constant 1024. Every wrong token now gets , so the 500-token wrong answer receives times the total push of the 80-token one.
Known issues and fixes
Length bias. Liu et al. ("Understanding R1-Zero-Like Training: A Critical Perspective", 2025) note that the factor is not neutral. For a correct answer () shorter outputs get a larger per-token reward. For a wrong answer () longer outputs get a smaller per-token penalty. The optimiser is therefore nudged toward longer wrong answers, which can show up as steadily growing response length that is not all useful reasoning.
Difficulty bias. Dividing by up-weights prompts whose rewards barely vary, those that are almost always or almost never solved. With 0/1 rewards a prompt with of 8 correct contributes against at . Without the std it would be 1.75 against 4. Their Dr. GRPO ("GRPO done right") removes both factors: advantages are , and token losses are summed and divided by a constant such as the generation budget. This makes the objective an unbiased policy gradient again.
Dead groups. If all rewards are equal, every advantage is zero and the prompt contributes nothing but KL. With pass rate that happens with probability , which is large for very easy and very hard prompts. DAPO (Yu et al., ByteDance Seed, 2025) over-samples and filters such groups ("dynamic sampling"). It also raises only the upper clip bound (, ) so that unlikely tokens can grow, averages the loss over all tokens in the batch instead of per output, shapes the reward for over-long outputs, and drops the KL term. With these changes it reports 50 points on AIME 2024 from Qwen2.5-32B base.
1import re, torch2from transformers import AutoModelForCausalLM, AutoTokenizer34name = "Qwen/Qwen2.5-0.5B-Instruct"5tok = AutoTokenizer.from_pretrained(name)6policy = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)7ref = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16).eval().requires_grad_(False)8opt = torch.optim.AdamW(policy.parameters(), lr=1e-6)9G, eps_clip, beta = 8, 0.2, 0.041011def reward(text, gold): # verifiable 0/1 reward: boxed answer must match12 m = re.search(r"\\boxed\{([^}]*)\}", text)13 return float(m is not None and m.group(1).strip() == gold)1415def token_logprobs(model, seq, P): # log π(o_t | q, o_<t) for the completion tokens16 logits = model(seq).logits[:, P - 1:-1].float()17 return torch.log_softmax(logits, -1).gather(-1, seq[:, P:, None]).squeeze(-1)1819def group_advantages(r, scale_std=True, eps=1e-4):20 a = r - r.mean()21 return a / (r.std() + eps) if scale_std else a # torch.std is Bessel-corrected2223prompt, gold = "What is 7 * 8? Put the final answer in \\boxed{}.", "56"24enc = tok(tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,25 add_generation_prompt=True), return_tensors="pt")26P = enc.input_ids.shape[1]2728with torch.no_grad(): # 1. sample a group from π_θold29 seq = policy.generate(**enc, do_sample=True, temperature=1.0, top_p=1.0,30 max_new_tokens=512, num_return_sequences=G, pad_token_id=tok.eos_token_id)31 comp = seq[:, P:]32 is_eos = comp == tok.eos_token_id # keep tokens up to and including the first EOS33 mask = ((is_eos.cumsum(1) - is_eos.long()) == 0).float()34 r = torch.tensor([reward(tok.decode(c, skip_special_tokens=True), gold) for c in comp])35 adv = group_advantages(r) # r = [1,0,0,0] would give ≈ [1.5, -0.5, -0.5, -0.5]36 logp_old = token_logprobs(policy, seq, P)37 logp_ref = token_logprobs(ref, seq, P)3839# 2. one policy update (DeepSeekMath does a single update per sampling round)40logp = token_logprobs(policy, seq, P)41ratio = torch.exp(logp - logp_old)42a = adv[:, None]43surr = torch.min(ratio * a, ratio.clamp(1 - eps_clip, 1 + eps_clip) * a)44log_x = logp_ref - logp # log(π_ref / π_θ)45k3 = torch.exp(log_x) - log_x - 1 # ≥ 0, unbiased for KL(π_θ ‖ π_ref)46per_token = -(surr - beta * k3) * mask47loss = (per_token.sum(1) / mask.sum(1)).mean() # GRPO: 1/|o_i| per output, then 1/G48# Dr. GRPO instead: loss = per_token.sum() / (G * 512), with adv = group_advantages(r, False)49opt.zero_grad(); loss.backward(); opt.step()Use when
- +No value model: saves about a policy-sized network, its optimiser state and its training instabilities
- +Advantages are invariant to reward scale and offset, and fit comparative reward models and 0/1 verifiers
- +Simple to implement on top of a PPO codebase; the KL sits directly in the loss
- +Powered DeepSeekMath-RL and the DeepSeek-R1 reasoning models
Avoid when
- −Needs samples per prompt (64 in DeepSeekMath), so generation dominates the cost
- −No credit assignment within an output under outcome rewards: every token gets the same advantage
- − and add length and difficulty biases (Dr. GRPO)
- −Prompts the model always or never solves give no signal
Where it's used
Reasoning RL
math and code with verifiable rewards: DeepSeek-R1-Zero, DeepSeek-R1 and many open reproductions
RL with a reward model
DeepSeekMath-RL used a learned reward model with outcome or process supervision
Library support
TRL's GRPOTrainer, OpenRLHF and verl implement GRPO and its variants
Check yourself
A group of G = 8 answers has rewards all equal to 1. What does GRPO's policy-gradient term do on this prompt?
Check yourself
Under the original GRPO loss, why can wrong answers drift longer?
Remember
- ✓GRPO replaces PPO's value model with a group baseline: sample outputs per prompt and set every token's advantage to .
- ✓Loss: with , , an unbiased non-negative KL estimate.
- ✓DeepSeekMath-RL 7B used , and lr , and reached 51.7% on MATH. DeepSeek-R1-Zero ran GRPO on rule-based rewards and raised AIME 2024 pass@1 from 15.6% to 71.0%.
- ✓Known biases: penalises long wrong answers less, and up-weights near-trivial and near-impossible prompts. Dr. GRPO removes both.
- ✓All-equal groups give zero gradient, with probability ; DAPO resamples them, decouples the clip range and drops the KL.