GRPO

GRPO (Shao et al., DeepSeekMath 2024) is PPO without a critic. It samples G answers per prompt, gives each the advantage (r − mean)/std of its group, and maximises a clipped ratio objective minus β times a k3 KL estimate against a frozen reference. DeepSeek-R1 trained it on verifiable rewards.

15 min read1451 wordsadvanced0 viewsupdated 2026-10-13

Definition

GRPO (Group Relative Policy Optimization) is the policy-gradient method introduced by Shao, Wang, Zhu, Xu, Song, Zhang, Li, Wu and Guo in "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (DeepSeek-AI, 2024). For each prompt qq it samples a group of GG outputs from the current policy, scores them, and uses each output's reward relative to the group, A^i=(ri−mean(r))/std(r)\hat A_i=(r_i-\mathrm{mean}(r))/\mathrm{std}(r), as the advantage of every one of its tokens. It keeps PPO's clipped probability ratio and adds a per-token KL penalty to a frozen reference model. It has no value network.

RLHF in the InstructGPT style fine-tunes the language model with PPO. PPO needs an advantage for every token, and it gets one from a learned value model VψV_\psi and generalised advantage estimation (GAE). For an LLM the value model is usually another transformer about the size of the policy. Training it costs memory and compute, and it is hard to train well: the reward arrives only at the last token, yet the value model must predict it at every earlier token.

The fix is an old one from REINFORCE: use other samples as the baseline. Kool, van Hoof and Welling ("Buy 4 REINFORCE Samples, Get a Baseline for Free!", 2019) used the mean reward of the other samples for the same input. Ahmadian et al. (2024) brought this leave-one-out baseline (RLOO) back to RLHF. GRPO puts the same idea inside PPO's clipped objective and also divides by the group's standard deviation. The DeepSeekMath authors point out that this suits reward models, which are usually trained on comparisons between outputs for the same question.

The objective

JGRPO(θ)=Eq,{oi}i=1G∼πθold(⋅∣q)1G∑i=1G1∣oi∣∑t=1∣oi∣{min⁡[ρi,tA^i,t, clip(ρi,t,1−ε,1+ε)A^i,t]−β DKL[πθ∥πref]}\mathcal J_{GRPO}(\theta)=\mathbb E_{q,\{o_i\}_{i=1}^G\sim\pi_{\theta_{old}}(\cdot\mid q)}\frac1G\sum_{i=1}^G\frac1{|o_i|}\sum_{t=1}^{|o_i|}\Big\{\min\big[\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_{i,t}\big]-\beta\,\mathbb D_{KL}\big[\pi_\theta\|\pi_{ref}\big]\Big\}
ρi,t\rho_{i,t}
πθ(oi,t∣q,oi,<t)/πθold(oi,t∣q,oi,<t)\pi_\theta(o_{i,t}\mid q,o_{i,<t})/\pi_{\theta_{old}}(o_{i,t}\mid q,o_{i,<t}), the probability ratio against the policy that sampled the group
A^i,t\hat A_{i,t}
advantage of token tt of output ii; with outcome rewards it is the same for all tokens of oio_i
clip(⋅,1−ε,1+ε)\mathrm{clip}(\cdot,1-\varepsilon,1+\varepsilon)
PPO's trust region: once the ratio leaves [1−ε,1+ε][1-\varepsilon,1+\varepsilon] in the direction A^\hat A favours, the token's gradient is zero
1∣oi∣\tfrac1{|o_i|}
each output's tokens are averaged over its own length, then outputs are averaged over the group
β\beta
KL weight; 0.04 in DeepSeekMath
πref\pi_{ref}
frozen reference policy, the SFT model the run started from
A^i,t=r~i=ri−mean(r1,…,rG)std(r1,…,rG)(outcome),A^i,t=∑steps j: index(j)≥tr~i index(j)(process)\hat A_{i,t}=\tilde r_i=\frac{r_i-\mathrm{mean}(r_1,\dots,r_G)}{\mathrm{std}(r_1,\dots,r_G)}\quad\text{(outcome)},\qquad \hat A_{i,t}=\sum_{\text{steps }j:\ \mathrm{index}(j)\ge t}\tilde r_i^{\,\mathrm{index}(j)}\quad\text{(process)}
rir_i
scalar reward of output ii: a reward model's score in DeepSeekMath, a rule-based check in DeepSeek-R1
r~i\tilde r_i
reward standardised within the group: the group mean is the baseline, the group std sets the scale
r~i index(j)\tilde r_i^{\,\mathrm{index}(j)}
with process supervision, each reasoning step gets a reward, all step rewards in the group are normalised the same way, and a token's advantage is the sum of normalised rewards of the steps from its own onward

Because the advantage is standardised, the reward's scale and offset do not matter. A verifier that returns 0/1 and one that returns 0/10 give the same update. What matters is the ranking of the GG outputs and how far apart they are.

GRPO step for one prompt: sample a group, score it, normalise inside the group, updateprompt qquestion+ formatpolicy π_θoldsamples Goutputs per qo₁ r=1o₂ r=0o₃ r=0⋮o_G r=1reward rᵢrule verifier (R1)or reward modelgroup-relative Âᵢ(rᵢ − mean r) / std rsame Âᵢ for every tokenreference π_reffrozen SFTmodelper-token KL, k3x − log x − 1 ≥ 0x = π_ref(oᵢ,ₜ)/π_θ(oᵢ,ₜ)objective (maximise)1/G Σᵢ 1/|oᵢ| Σₜ [ min(ρÂᵢ, clip(ρ, 1±ε)Âᵢ) − β·k3 ]ρ = π_θ(oᵢ,ₜ)/π_θold(oᵢ,ₜ)value model V_ψ (PPO)not used: baseline = group meanPPO: policy + value + reward + reference; per-token KL inside the rewardGRPO: policy + reward + reference; KL added to the loss directlya group whose rewards are all equal gives Âᵢ = 0: no gradientDeepSeekMath-RL 7B: G = 64 samples/question, β = 0.04, policy lr 1e-6, max length 1024, batch 1024~144K GSM8K/MATH-style questions · one policy update per sampling round · ε is the PPO clip range
fig.One GRPO step. The old policy samples G outputs for a prompt. A rule-based verifier (DeepSeek-R1) or a reward model (DeepSeekMath) scores each one. Rewards are standardised inside the group, and every token of an output shares its advantage. The loss combines PPO's clipped ratio with a per-token k3 KL estimate against the frozen reference. The value model, which PPO would train alongside the policy, is gone.

The KL term: the k3 estimator

PPO-based RLHF subtracts a per-token KL penalty from the reward, so the KL passes through the value model and GAE. GRPO puts the KL straight into the loss and estimates it per sampled token with an estimator from John Schulman's note "Approximating KL Divergence" (2020), often called k3:

DKL[πθ∥πref]≈k3=πref(oi,t∣⋅)πθ(oi,t∣⋅)−log⁡πref(oi,t∣⋅)πθ(oi,t∣⋅)−1,Eo∼πθ[πrefπθ]=1\mathbb D_{KL}\big[\pi_\theta\|\pi_{ref}\big]\approx k_3=\frac{\pi_{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)}-\log\frac{\pi_{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)}-1,\qquad \mathbb E_{o\sim\pi_\theta}\Big[\tfrac{\pi_{ref}}{\pi_\theta}\Big]=1
x=πref/πθx=\pi_{ref}/\pi_\theta
likelihood ratio of the sampled token under the reference and the current policy
x−log⁡x−1≥0x-\log x-1\ge0
always non-negative, since log⁡x≤x−1\log x\le x-1; zero only when x=1x=1
E[x]=1\mathbb E[x]=1
so E[k3]=E[−log⁡x]=KL(πθ∥πref)\mathbb E[k_3]=\mathbb E[-\log x]=\mathrm{KL}(\pi_\theta\|\pi_{ref}): unbiased, with lower variance than the plain −log⁡x-\log x estimator, which can be negative per sample

The DeepSeekMath paper writes every method's gradient in one form, E[1∣o∣∑tGC(q,o,t) ∇θlog⁡πθ(ot∣q,o<t)]\mathbb E\big[\tfrac1{|o|}\sum_t GC(q,o,t)\,\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})\big], and compares the gradient coefficient GCGC. For GRPO (on-policy, ignoring the clip), differentiating −βk3-\beta k_3 gives β(x−1)∇log⁡πθ\beta(x-1)\nabla\log\pi_\theta, so GC=A^i,t+β(πref/πθ−1)GC=\hat A_{i,t}+\beta\big(\pi_{ref}/\pi_\theta-1\big). The KL term pushes a token up when the reference likes it more than the policy does (x>1x>1) and down otherwise. For SFT the coefficient is 1 on every token. For rejection-sampling fine-tuning it is 1 on correct outputs and 0 on wrong ones. GRPO, like PPO, also gives negative weight to wrong outputs, scaled by how far below the group they fall.

DeepSeekMath 7B and the R1 runs

Base
DeepSeekMath-Base 7B: DeepSeek-Coder-Base-v1.5 7B trained for another 500B tokens, including the 120B-token DeepSeekMath Corpus mined from Common Crawl
RL data
about 144K chain-of-thought questions related to GSM8K and MATH, taken from the SFT data; RL starts from DeepSeekMath-Instruct 7B
GRPO settings
G=64G=64 outputs per question, β=0.04\beta=0.04, policy learning rate 10−610^{-6}, max length 1024, batch size 1024, one policy update per sampling round
Reward model
trained from DeepSeekMath-Base 7B with learning rate 2⋅10−52\cdot10^{-5}; iterative GRPO retrains it on fresh policy samples with 10% replay of old data
Results (no tools)
MATH: 46.8% Instruct → 51.7% RL; GSM8K: 82.9% → 88.2%; self-consistency over 64 samples reaches 60.9% on MATH
DeepSeek-R1-Zero (2025)
GRPO applied directly to DeepSeek-V3-Base with rule-based accuracy and format rewards and no SFT; AIME 2024 pass@1 rose from 15.6% to 71.0% (86.7% with majority voting)
DeepSeek-R1
cold-start SFT, reasoning GRPO with an added language-consistency reward, about 800K rejection-sampled SFT examples, then a final RL stage; AIME 2024 pass@1 79.8%, MATH-500 97.3%

Verifiable rewards and DeepSeek-R1

DeepSeek-R1 ("DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", DeepSeek-AI, 2025) made GRPO the default RL algorithm for reasoning models. R1-Zero used no neural reward model at all. Its accuracy reward checks the final answer of a math problem, given in a required format such as a box, or runs code against test cases. Its format reward checks that the reasoning sits between <think> and </think> tags. The authors avoided neural reward models here because the policy can learn to exploit them (reward hacking) over a long run.

Under this RL, R1-Zero's responses grew steadily longer, and it learned to re-check and revise its own work, including the "aha moment" the paper shows. Its outputs were hard to read and mixed languages, so DeepSeek-R1 added a cold-start SFT stage and a language-consistency reward. Binary verifiable rewards fit GRPO well. With 0/1 rewards and kk correct out of GG, every correct answer gets (G−k)/k\sqrt{(G-k)/k} and every wrong one gets −k/(G−k)-\sqrt{k/(G-k)}: rare successes are pushed up hard.

DeepSeekMath also reports a caution. RL improved Maj@K but not Pass@K. The authors read this as RL making the output distribution more robust, raising the correct answers already among the model's top-K samples, rather than adding new basic capabilities.

GRPO: Âᵢ = (rᵢ − mean r) / std r · token t of oᵢ pushed with Âᵢ / |oᵢ| · no criticone prompt, G = 8 samples, each correct with p = 0.40 (reward 1) or wrong (reward 0)oᵢr|oᵢ| tokensadvantage Âᵢper token (×10⁻³)o1✗836−0.775−0.93o2✓298+1.291+4.33o3✗738−0.775−1.05o4✓294+1.291+4.39o5✗344−0.775−2.25o6✗312−0.775−2.48o7✓397+1.291+3.25o8✗765−0.775−1.01o1: clipped surrogate vs ratio ρ0.81.01.2Â = −0.775; shaded: gradient 0lower π(o) until ρ = 0.8Σ|Â| vs # correct k (bars: P(k))048k = 1: 5.29 · k = 4: 8.00E[Σ|Â|] at p = 0.40: 7.23k = 3/8 correct · mean r = 0.375 · std r = 0.484 · P(all equal) = 0.40^8 + 0.60^8 = 0.017answer-level push Σₜ Âᵢ/|oᵢ| = Âᵢ: each correct answer +1.29, each wrong one −0.77, whatever its length÷|oᵢ|: the 836-token wrong answer is penalised 2.68× less per token than the 312-token one÷std: a prompt with k = 1 gets 0.66× the signal of k = 4 (without ÷std: 0.44×)
group size
fig.One prompt and a group of G sampled answers, each correct (reward 1) with probability p. In this toy, wrong answers are drawn longer than right ones. At the default (G = 8, p = 0.4, sample 5), 3 of 8 are correct: mean reward 0.375, std 0.484, so each correct answer gets  = +1.291 and each wrong one −0.775. Dividing by length makes the per-token push uneven: the 312-token wrong answer gets −2.48·10⁻³ per token, the 836-token one only −0.93·10⁻³, which is 2.68× less. Long wrong answers are penalised least. Switch to Dr. GRPO (no ÷std, no ÷|oᵢ|): advantages become +0.625 and −0.375, every wrong token gets the same −0.37·10⁻³, and answer-level pushes grow with length (wrong −0.31 … −0.11). The right panel plots the clipped surrogate for the selected answer: with  < 0 the gradient is zero once ρ < 0.8. The lower panel shows the total |advantage| a prompt contributes against the number correct k. With ÷std a prompt with k = 1 gets 5.29 against 8.00 at k = 4 (0.66×). Without it, 1.75 against 4.00 (0.44×). This is the difficulty bias Dr. GRPO removes. At p = 0.1 sample 5 is all wrong, so every advantage is 0; with G = 8 that happens for 43.0% of prompts (0.017 at p = 0.4 and G = 8, under 0.001 at p = 0.4 and G = 16).

Worked example: a group of four

One math question, G=4G=4 sampled answers, a 0/1 verifier. Only the first is correct. Advantages use the population std, as in the formula. TRL's implementation uses the Bessel-corrected torch.std, which would give (1.5,−0.5,−0.5,−0.5)(1.5,-0.5,-0.5,-0.5) instead.

outputreward rᵢlength |oᵢ|ÂᵢGRPO per-token Âᵢ/|oᵢ|Dr. GRPO per-token (rᵢ − r̄)/1024
o₁1120+1.732+0.01443+0.000732
o₂0300−0.577−0.00192−0.000244
o₃080−0.577−0.00722−0.000244
o₄0500−0.577−0.00115−0.000244
Group statistics: mean 0.25, std 0.433 (computed in Python)

From rewards to one token's loss

  1. 1Group statistics: rˉ=0.25\bar r=0.25 and std=0.25⋅0.75=0.433\mathrm{std}=\sqrt{0.25\cdot0.75}=0.433. So A^1=0.75/0.433=+1.732\hat A_1=0.75/0.433=+1.732 and A^2,3,4=−0.25/0.433=−0.577\hat A_{2,3,4}=-0.25/0.433=-0.577. The advantages sum to zero, as they always do.
  2. 2Length normalisation: each wrong answer's tokens together receive −0.577-0.577. The 500-token o4o_4 spreads it to −0.00115-0.00115 per token, the 80-token o3o_3 concentrates it to −0.00722-0.00722, which is 6.25 times more per token. Under GRPO, rambling is the cheaper way to be wrong.
  3. 3Clipping a good token: a token of o1o_1 with ρ=1.3\rho=1.3 gives min⁡(1.3⋅1.732, 1.2⋅1.732)=min⁡(2.252,2.078)=2.078\min(1.3\cdot1.732,\ 1.2\cdot1.732)=\min(2.252,2.078)=2.078. The clipped branch is active, so this token gives no more gradient this round.
  4. 4Clipping a bad token: a token of o2o_2 with ρ=0.7\rho=0.7 gives min⁡(0.7⋅(−0.577), 0.8⋅(−0.577))=min⁡(−0.404,−0.462)=−0.462\min(0.7\cdot(-0.577),\ 0.8\cdot(-0.577))=\min(-0.404,-0.462)=-0.462. That is clipped too: its probability has already fallen 30%, past the 1−ε1-\varepsilon bound.
  5. 5KL for a token with πθ=0.5\pi_\theta=0.5 and πref=0.4\pi_{ref}=0.4: x=0.8x=0.8 and k3=0.8−ln⁡0.8−1=0.0231k_3=0.8-\ln0.8-1=0.0231, weighted by β=0.04\beta=0.04. Check of unbiasedness on a 3-token vocabulary with πθ=(0.5,0.3,0.2)\pi_\theta=(0.5,0.3,0.2) and πref=(0.4,0.4,0.2)\pi_{ref}=(0.4,0.4,0.2): Eπθ[k3]=0.02527=KL(πθ∥πref)\mathbb E_{\pi_\theta}[k_3]=0.02527=\mathrm{KL}(\pi_\theta\|\pi_{ref}).
  6. 6Dr. GRPO on the same group: advantages (0.75,−0.25,−0.25,−0.25)(0.75,-0.25,-0.25,-0.25) divided by a constant 1024. Every wrong token now gets −0.000244-0.000244, so the 500-token wrong answer receives 500/80=6.25500/80=6.25 times the total push of the 80-token one.

Known issues and fixes

Length bias. Liu et al. ("Understanding R1-Zero-Like Training: A Critical Perspective", 2025) note that the 1/∣oi∣1/|o_i| factor is not neutral. For a correct answer (A^>0\hat A>0) shorter outputs get a larger per-token reward. For a wrong answer (A^<0\hat A<0) longer outputs get a smaller per-token penalty. The optimiser is therefore nudged toward longer wrong answers, which can show up as steadily growing response length that is not all useful reasoning.

Difficulty bias. Dividing by std(r)\mathrm{std}(r) up-weights prompts whose rewards barely vary, those that are almost always or almost never solved. With 0/1 rewards a prompt with k=1k=1 of 8 correct contributes ∑∣A^i∣=5.29\sum|\hat A_i|=5.29 against 88 at k=4k=4. Without the std it would be 1.75 against 4. Their Dr. GRPO ("GRPO done right") removes both factors: advantages are ri−mean(r)r_i-\mathrm{mean}(r), and token losses are summed and divided by a constant such as the generation budget. This makes the objective an unbiased policy gradient again.

Dead groups. If all GG rewards are equal, every advantage is zero and the prompt contributes nothing but KL. With pass rate pp that happens with probability pG+(1−p)Gp^G+(1-p)^G, which is large for very easy and very hard prompts. DAPO (Yu et al., ByteDance Seed, 2025) over-samples and filters such groups ("dynamic sampling"). It also raises only the upper clip bound (εlow=0.2\varepsilon_{low}=0.2, εhigh=0.28\varepsilon_{high}=0.28) so that unlikely tokens can grow, averages the loss over all tokens in the batch instead of per output, shapes the reward for over-long outputs, and drops the KL term. With these changes it reports 50 points on AIME 2024 from Qwen2.5-32B base.

python
1import re, torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4name = "Qwen/Qwen2.5-0.5B-Instruct"
5tok = AutoTokenizer.from_pretrained(name)
6policy = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)
7ref = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16).eval().requires_grad_(False)
8opt = torch.optim.AdamW(policy.parameters(), lr=1e-6)
9G, eps_clip, beta = 8, 0.2, 0.04
10
11def reward(text, gold): # verifiable 0/1 reward: boxed answer must match
12 m = re.search(r"\\boxed\{([^}]*)\}", text)
13 return float(m is not None and m.group(1).strip() == gold)
14
15def token_logprobs(model, seq, P): # log π(o_t | q, o_<t) for the completion tokens
16 logits = model(seq).logits[:, P - 1:-1].float()
17 return torch.log_softmax(logits, -1).gather(-1, seq[:, P:, None]).squeeze(-1)
18
19def group_advantages(r, scale_std=True, eps=1e-4):
20 a = r - r.mean()
21 return a / (r.std() + eps) if scale_std else a # torch.std is Bessel-corrected
22
23prompt, gold = "What is 7 * 8? Put the final answer in \\boxed{}.", "56"
24enc = tok(tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
25 add_generation_prompt=True), return_tensors="pt")
26P = enc.input_ids.shape[1]
27
28with torch.no_grad(): # 1. sample a group from π_θold
29 seq = policy.generate(**enc, do_sample=True, temperature=1.0, top_p=1.0,
30 max_new_tokens=512, num_return_sequences=G, pad_token_id=tok.eos_token_id)
31 comp = seq[:, P:]
32 is_eos = comp == tok.eos_token_id # keep tokens up to and including the first EOS
33 mask = ((is_eos.cumsum(1) - is_eos.long()) == 0).float()
34 r = torch.tensor([reward(tok.decode(c, skip_special_tokens=True), gold) for c in comp])
35 adv = group_advantages(r) # r = [1,0,0,0] would give ≈ [1.5, -0.5, -0.5, -0.5]
36 logp_old = token_logprobs(policy, seq, P)
37 logp_ref = token_logprobs(ref, seq, P)
38
39# 2. one policy update (DeepSeekMath does a single update per sampling round)
40logp = token_logprobs(policy, seq, P)
41ratio = torch.exp(logp - logp_old)
42a = adv[:, None]
43surr = torch.min(ratio * a, ratio.clamp(1 - eps_clip, 1 + eps_clip) * a)
44log_x = logp_ref - logp # log(π_ref / π_θ)
45k3 = torch.exp(log_x) - log_x - 1 # ≥ 0, unbiased for KL(π_θ ‖ π_ref)
46per_token = -(surr - beta * k3) * mask
47loss = (per_token.sum(1) / mask.sum(1)).mean() # GRPO: 1/|o_i| per output, then 1/G
48# Dr. GRPO instead: loss = per_token.sum() / (G * 512), with adv = group_advantages(r, False)
49opt.zero_grad(); loss.backward(); opt.step()
One GRPO step for a single prompt with Hugging Face transformers: sample G outputs, verify, standardise within the group, clipped loss with the k3 KL (not run here; PyTorch)

Use when

  • +No value model: saves about a policy-sized network, its optimiser state and its training instabilities
  • +Advantages are invariant to reward scale and offset, and fit comparative reward models and 0/1 verifiers
  • +Simple to implement on top of a PPO codebase; the KL sits directly in the loss
  • +Powered DeepSeekMath-RL and the DeepSeek-R1 reasoning models

Avoid when

  • −Needs GG samples per prompt (64 in DeepSeekMath), so generation dominates the cost
  • −No credit assignment within an output under outcome rewards: every token gets the same advantage
  • −1/∣oi∣1/|o_i| and ÷std\div\mathrm{std} add length and difficulty biases (Dr. GRPO)
  • −Prompts the model always or never solves give no signal

Where it's used

Reasoning RL

math and code with verifiable rewards: DeepSeek-R1-Zero, DeepSeek-R1 and many open reproductions

RL with a reward model

DeepSeekMath-RL used a learned reward model with outcome or process supervision

Library support

TRL's GRPOTrainer, OpenRLHF and verl implement GRPO and its variants

Check yourself

A group of G = 8 answers has rewards all equal to 1. What does GRPO's policy-gradient term do on this prompt?

Check yourself

Under the original GRPO loss, why can wrong answers drift longer?

Remember

  • ✓GRPO replaces PPO's value model with a group baseline: sample GG outputs per prompt and set every token's advantage to (ri−mean r)/std r(r_i-\mathrm{mean}\,r)/\mathrm{std}\,r.
  • ✓Loss: 1G∑i1∣oi∣∑t[min⁡(ρA^,clip(ρ,1±ε)A^)−βk3]\frac1G\sum_i\frac1{|o_i|}\sum_t\big[\min(\rho\hat A,\mathrm{clip}(\rho,1\pm\varepsilon)\hat A)-\beta k_3\big] with k3=x−log⁡x−1k_3=x-\log x-1, x=πref/πθx=\pi_{ref}/\pi_\theta, an unbiased non-negative KL estimate.
  • ✓DeepSeekMath-RL 7B used G=64G=64, β=0.04\beta=0.04 and lr 10−610^{-6}, and reached 51.7% on MATH. DeepSeek-R1-Zero ran GRPO on rule-based rewards and raised AIME 2024 pass@1 from 15.6% to 71.0%.
  • ✓Known biases: 1/∣oi∣1/|o_i| penalises long wrong answers less, and ÷std\div\mathrm{std} up-weights near-trivial and near-impossible prompts. Dr. GRPO removes both.
  • ✓All-equal groups give zero gradient, with probability pG+(1−p)Gp^G+(1-p)^G; DAPO resamples them, decouples the clip range and drops the KL.
← back to Deep Learning roadmap