DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Overview

Shao et al. (2024) is two papers in one. The first half builds the DeepSeekMath Corpus, 120B tokens of math web pages selected from Common Crawl, and uses it to continue pretraining DeepSeek-Coder-Base-v1.5 7B. The resulting DeepSeekMath-Base 7B scores 36.2% on the competition-level MATH benchmark with few-shot chain-of-thought prompting, above Minerva 540B at 33.6% (Table 2). The second half introduces Group Relative Policy Optimization (GRPO), the reinforcement-learning algorithm DeepSeek later used to train DeepSeek-R1.

GRPO is PPO with the value network removed. For each question the policy samples a group of $G$ outputs, a reward model scores them, and each output's advantage is its reward standardized within the group: $\hat A_i = (r_i - \operatorname{mean}(\mathbf r)) / \operatorname{std}(\mathbf r)$. The clipped importance ratio from PPO stays. The KL anchor to the reference model moves out of the reward and into the loss, as a per-token estimator. The critic, typically a second network about the size of the policy, is gone. The price is $G$ samples per question, and the paper uses $G = 64$.

Starting from DeepSeekMath-Instruct 7B and training only on chain-of-thought questions tied to GSM8K and MATH, GRPO raises MATH top-1 accuracy from 46.8% to 51.7% and GSM8K from 82.9% to 88.2%, both without tools (Table 5). The abstract reports 60.9% on MATH with self-consistency over 64 samples. For scale, Table 5 lists GPT-4 at 52.9% and Gemini Ultra at 53.2% on MATH.

The RL half also contains a conceptual contribution. Section 5.2 writes the parameter gradient of SFT, rejection-sampling fine-tuning (RFT), DPO, PPO and GRPO in one form, a per-token score function times a method-specific gradient coefficient. That form separates three design choices: where the training outputs come from, what supplies the reward, and how the reward becomes a coefficient. The same section reports that RL raised Maj@K accuracy but not Pass@K, a result later work re-tested at much larger $K$.

Mathematical Contributions

The DeepSeekMath Corpus

Section 2.1 describes an iterative recall loop over Common Crawl.

1. Train a fastText classifier with 500,000 OpenWebMath pages as positives and 500,000 random Common Crawl pages as negatives (vector dimension 256, learning rate 0.1, word n-grams up to length 3, minimum word count 3, 3 epochs).
2. Score a URL-deduplicated and near-deduplicated Common Crawl of 40B HTML pages, rank pages by classifier score, and keep the top of the ranking. Pretraining runs on the top 40B, 80B, 120B and 160B tokens set the cutoff; the first iteration keeps 40B tokens.
3. Split Common Crawl into domains (pages sharing a base URL). A domain with more than 10% of its pages already collected is labeled math-related, for example mathoverflow.net.
4. Annotators mark the math URL paths inside those domains, for example mathoverflow.net/questions. Uncollected pages under those paths join the seed set, and the classifier is retrained.

After four iterations the corpus holds 35.5M pages and 120B tokens. The loop stopped because nearly 98% of the fourth iteration's data had already been collected in the third. Decontamination removes any text segment containing a 10-gram that matches a substring of benchmarks such as GSM8K, MATH, CMATH and AGIEval. Benchmark texts shorter than 10 grams but at least 3 grams long are filtered by exact matching.

Table 1 isolates corpus quality by training the same 1.3B DeepSeek-LLM for 150B tokens on each math corpus and evaluating with few-shot chain-of-thought prompting:

The corpus is mostly English and Chinese, which the paper uses to explain the CMATH gap. Section 5.1 adds two pretraining findings. Code training before math training raised math accuracy both with and without tool use (Table 6). Math training on arXiv-only corpora (MathPile and ArXiv-RedPajama) brought no notable improvement on the paper's benchmarks (Tables 8 and 9), a conclusion the authors restrict to the tasks, data mixtures and model scales they tested.

DeepSeekMath-Base 7B trains for 500B tokens: 56% DeepSeekMath Corpus, 4% AlgebraicStack, 10% arXiv, 20% GitHub code and 10% English and Chinese natural-language text (Section 2.3). Instruction tuning on 776K examples in chain-of-thought, program-of-thought and tool-integrated formats produces DeepSeekMath-Instruct 7B (Section 3), the starting point for RL.

PPO as the paper writes it

For a question $q \sim P(Q)$ and an output $o$ sampled from the old policy, PPO maximizes (Equation 1)

$\mathcal{J}_{\mathrm{PPO}}(\theta) = \mathbb{E}_{q,\, o \sim \pi_{\theta_{\mathrm{old}}}(O \mid q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|} \min\Big(\rho_t A_t,\; \mathrm{clip}\big(\rho_t, 1-\varepsilon, 1+\varepsilon\big) A_t\Big)\right], \qquad \rho_t = \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{\theta_{\mathrm{old}}}(o_t \mid q, o_{<t})}.$

The advantage $A_t$ comes from Generalized Advantage Estimation (GAE) applied to the rewards $r_{\ge t}$ and a learned value function $V_\psi$. To limit over-optimization of the reward model $r_\varphi$, the KL penalty to the reference model $\pi_{\mathrm{ref}}$ (usually the SFT model) is folded into the per-token reward (Equation 2), following InstructGPT:

$r_t = r_\varphi(q, o_{\le t}) - \beta \log \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{\mathrm{ref}}(o_t \mid q, o_{<t})}.$

Section 4.1.1 raises two objections to the value function here. It is usually a network of size comparable to the policy, which costs memory and compute. And the reward model usually scores only the last token, which makes it hard to train a value function that is accurate at every token.

The GRPO objective

For each question GRPO samples $o_1, \ldots, o_G$ from $\pi_{\theta_{\mathrm{old}}}$ and maximizes (Equation 3)

$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G} \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \Big\{ \min\Big(\rho_{i,t} \hat A_{i,t},\; \mathrm{clip}\big(\rho_{i,t}, 1-\varepsilon, 1+\varepsilon\big) \hat A_{i,t}\Big) - \beta\, \mathbb{D}_{\mathrm{KL}}\big[\pi_\theta \,\|\, \pi_{\mathrm{ref}}\big] \Big\}\right],$

with the expectation over $q \sim P(Q)$ and the group $o_1, \ldots, o_G \sim \pi_{\theta_{\mathrm{old}}}(O \mid q)$, and $\rho_{i,t} = \pi_\theta(o_{i,t} \mid q, o_{i,<t}) / \pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})$. Three things differ from Equations 1 and 2: the average over $G$ outputs of the same question, an advantage $\hat A_{i,t}$ computed only from rewards inside the group, and a KL term placed in the objective instead of in the reward. The paper motivates the group-relative advantage by how reward models are trained, on comparisons between outputs for the same question. It moves the KL term to avoid complicating the computation of $\hat A_{i,t}$.

In the reported run (Section 4.2) the reward model starts from DeepSeekMath-Base 7B and trains with learning rate $2 \times 10^{-5}$ on data built following Math-Shepherd. The policy uses learning rate $10^{-6}$, KL coefficient $\beta = 0.04$, $G = 64$ samples per question, maximum length 1024 and batch size 1024, with a single policy update after each sampling stage. The RL questions are about 144K chain-of-thought-format questions related to GSM8K and MATH, taken from the SFT data.

The single update matters for the analysis below. Appendix A.1.5 uses it to drop the min and clip operations. With one step per sampling stage, $\pi_{\theta_{\mathrm{old}}} = \pi_\theta$ where the gradient is taken, so $\rho = 1$ sits inside $[1-\varepsilon, 1+\varepsilon]$ and the clipped objective has the same gradient as $\rho \hat A$. The calculations on this page work in that regime.

Group-normalized advantages

With outcome supervision (Section 4.1.2) the reward model returns one score per output, $\mathbf r = (r_1, \ldots, r_G)$, and every token of $o_i$ receives the same advantage:

$\hat A_{i,t} = \tilde r_i = \frac{r_i - \operatorname{mean}(\mathbf r)}{\operatorname{std}(\mathbf r)}, \qquad t = 1, \ldots, |o_i|.$

The advantages in a group sum to zero, so within each question GRPO raises the probability of above-average outputs and lowers the probability of below-average ones. The paper does not analyze two further consequences of this formula.

Ties. If all $G$ rewards are equal, the numerator and $\operatorname{std}(\mathbf r)$ are both zero and the formula is undefined. Section 4.1.2 does not say how this case is handled. Under any convention that keeps the denominator positive, the advantage is zero and the question contributes only through the KL term. The paper's run uses real-valued reward-model scores. With a binary correctness reward, every all-correct or all-wrong group gives zero advantage. DAPO (Yu et al., 2025), whose rule-based reward is $+1$ when the predicted answer matches the ground truth and $-1$ otherwise, reports the number of prompts with accuracy 1 rising during training, and it oversamples so that every prompt in a batch has accuracy strictly between 0 and 1.

Per-question weight. Take 0/1 rewards, let $\hat p$ be the fraction of correct outputs in the group, and read $\operatorname{std}$ as the population standard deviation over the $G$ rewards. Then $\operatorname{std}(\mathbf r) = \sqrt{\hat p(1-\hat p)}$ and

$\hat A_{\text{correct}} = \sqrt{\frac{1-\hat p}{\hat p}}, \qquad \hat A_{\text{wrong}} = -\sqrt{\frac{\hat p}{1-\hat p}}.$

At $\hat p = 1/2$ these are $\pm 1$. At $G = 64$ with one correct output, that output gets $\sqrt{63} \approx 7.94$ and each wrong output gets $-1/\sqrt{63} \approx -0.126$. Relative to plain mean-centering, the division multiplies the whole question's contribution by $1/\sqrt{\hat p(1-\hat p)}$. That factor is at least 2 and grows without bound as $\hat p$ approaches 0 or 1, so questions the policy almost always or almost never solves get the largest weight. Liu et al. (2025) call this a question-level difficulty bias.

What the group mean estimates

This subsection is a derivation added on this page, not a result stated in the paper. Fix a question $q$, drop the KL term ($\beta = 0$) and the standard-deviation division, and work in the single-update regime, so each token contributes $\hat A_{i,t}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t})$. The token gradients of output $i$ sum to $\nabla_\theta \log \pi_\theta(o_i \mid q)$, so the gradient estimate for $q$ is

$\hat g = \frac{1}{G} \sum_{i=1}^{G} \frac{r_i - \bar r}{|o_i|}\, \nabla_\theta \log \pi_\theta(o_i \mid q), \qquad \bar r = \frac{1}{G}\sum_{j=1}^{G} r_j.$

The group mean contains $r_i$ itself. Writing $\bar r_{-i}$ for the mean of the other $G - 1$ rewards, one line of algebra gives

$r_i - \bar r = \frac{G-1}{G}\,\big(r_i - \bar r_{-i}\big).$

Mean-centering is therefore the REINFORCE leave-one-out (RLOO) baseline, which Ahmadian et al. (2024) credit to Kool et al. (2019), scaled by $(G-1)/G$. Let $R(q, o)$ be the reward-model score and $\mu(q) = \mathbb{E}_{o \sim \pi_\theta(\cdot \mid q)}[R(q, o)]$. The $G$ outputs are sampled independently, so $\bar r_{-i}$ is independent of $o_i$ and has mean $\mu(q)$. Using that and $\mathbb{E}[f(o)\, \nabla_\theta \log \pi_\theta(o \mid q)] = \nabla_\theta \mathbb{E}[f(o)]$ for any $f$ that does not depend on $\theta$ (the score-function identity),

$\mathbb{E}[\hat g] = \frac{G-1}{G}\, \nabla_\theta\, \mathbb{E}_{o \sim \pi_\theta(\cdot \mid q)}\!\left[\frac{R(q, o) - c}{|o|}\right] \quad \text{evaluated at } c = \mu(q),$

with $c$ held fixed during differentiation. Two readings follow.

• If every output had the same length $L$, then $\mathbb{E}[\hat g] = \frac{G-1}{G L} \nabla_\theta \mu(q)$, an unbiased policy gradient up to a positive constant.
• With variable lengths, the expected update increases $\mathbb{E}[(R - c)/|o|]$. For outputs scored above the current mean, shorter is better. For outputs scored below it, longer shrinks the penalty. Liu et al. (2025) identify this response-level length bias and remove both the $1/|o_i|$ and the standard-deviation normalizations in their Dr. GRPO variant.

Restoring the division by $\operatorname{std}(\mathbf r)$ breaks the factorization, since the standard deviation depends on every output in the group, including $o_i$.

The KL term

GRPO estimates the per-token KL with an estimator from Schulman (2020), given as Equation 4. Write $\lambda_{i,t} = \pi_{\mathrm{ref}}(o_{i,t} \mid q, o_{i,<t}) / \pi_\theta(o_{i,t} \mid q, o_{i,<t})$. Then

$\mathbb{D}_{\mathrm{KL}}\big[\pi_\theta \,\|\, \pi_{\mathrm{ref}}\big] = \lambda_{i,t} - \log \lambda_{i,t} - 1.$

Fix a prefix $h = (q, o_{i,<t})$ and draw a token $a \sim \pi_\theta(\cdot \mid h)$, writing $\lambda$ for the ratio at $a$. Two properties hold.

• Nonnegative. Since $\log x \le x - 1$ for $x > 0$, each term is at least 0, with equality only when the two policies give the sampled token the same probability.
• Unbiased for $\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})$. $\mathbb{E}[\lambda] = \sum_a \pi_{\mathrm{ref}}(a \mid h) = 1$ whenever $\pi_\theta(\cdot \mid h)$ puts positive mass on every token, which a softmax policy does, and $\mathbb{E}[-\log \lambda] = \mathrm{KL}\big(\pi_\theta(\cdot \mid h) \,\|\, \pi_{\mathrm{ref}}(\cdot \mid h)\big)$.

The gradient is a different object. Differentiating $-\beta(\lambda - \log\lambda - 1)$ through $\pi_\theta$ gives $\beta(\lambda - 1)\, \nabla_\theta \log \pi_\theta(a \mid h)$, the second term of the paper's GRPO gradient coefficient (Equation 21). Averaging over $a \sim \pi_\theta(\cdot \mid h)$ and using $\sum_a \pi_\theta(a \mid h)\, \nabla_\theta \log \pi_\theta(a \mid h) = 0$:

$\mathbb{E}_{a}\big[\beta(\lambda - 1)\, \nabla_\theta \log \pi_\theta(a \mid h)\big] = \beta \sum_a \pi_{\mathrm{ref}}(a \mid h)\, \nabla_\theta \log \pi_\theta(a \mid h) = -\beta\, \nabla_\theta\, \mathrm{KL}\big(\pi_{\mathrm{ref}}(\cdot \mid h) \,\|\, \pi_\theta(\cdot \mid h)\big).$

At each visited prefix the term's expected gradient therefore reduces $\mathrm{KL}(\pi_{\mathrm{ref}} \,\|\, \pi_\theta)$, the KL divergence with its arguments swapped, while its value estimates $\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})$. This calculation, which is not in the paper, holds the prefix fixed and ignores how $\theta$ changes which prefixes get sampled.

Process supervision and iterative RL

Section 4.1.3 replaces the outcome score with step scores from a process reward model, following Math-Shepherd (Wang et al., 2023). Output $i$ has $K_i$ steps, step $j$ ends at token $\mathrm{index}(j)$, and the process reward model gives it the score $r_i^{\mathrm{index}(j)}$. All step scores in the group form the set $\mathbf R$. Each is standardized with $\operatorname{mean}(\mathbf R)$ and $\operatorname{std}(\mathbf R)$, and a token's advantage is the sum of the standardized scores of the steps that end at or after it:

$\hat A_{i,t} = \sum_{j:\ \mathrm{index}(j) \ge t} \tilde r_i^{\,\mathrm{index}(j)}.$

Section 4.1.4 and Algorithm 1 add an outer loop. Each iteration starts by setting the reference model to the current policy, then trains the policy with GRPO. After that, the reward model is trained further on data built from the policy's samples, with a replay mechanism that mixes in 10% historical data. The paper runs two iterations with the 7B model (Figure 6) and reports the largest gain in the first.

PPO and GRPO at a glance

One gradient for six methods

Section 5.2.1 writes the gradient of a training method $\mathcal A$ as (Equation 5)

$\nabla_\theta \mathcal{J}_{\mathcal{A}}(\theta) = \mathbb{E}_{(q, o) \sim \mathcal{D}}\left[\frac{1}{|o|} \sum_{t=1}^{|o|} GC_{\mathcal{A}}(q, o, t, \pi_{rf})\, \nabla_\theta \log \pi_\theta(o_t \mid q, o_{<t})\right].$

A method is fixed by three components: the data source $\mathcal D$, the reward function $\pi_{rf}$, and the algorithm that turns data and reward into the gradient coefficient $GC_{\mathcal A}$. Appendix A.1 derives each row of Table 10, reproduced here with the coefficients written out ($P_{sft}$ is the SFT data distribution and $\pi_{sft}$ the SFT model):

For DPO with a preferred output $o^+$ and a rejected output $o^-$, Equation 14 gives

$GC_{\mathrm{DPO}} = \sigma\!\left(\beta \log \frac{\pi_\theta(o^-_t \mid q, o^-_{<t})}{\pi_{\mathrm{ref}}(o^-_t \mid q, o^-_{<t})} - \beta \log \frac{\pi_\theta(o^+_t \mid q, o^+_{<t})}{\pi_{\mathrm{ref}}(o^+_t \mid q, o^+_{<t})}\right),$

which multiplies $+\nabla_\theta \log \pi_\theta$ on tokens of $o^+$ and $-\nabla_\theta \log \pi_\theta$ on tokens of $o^-$ (Equation 13). The appendix writes the DPO objective with log-ratios averaged over tokens (Equation 12). The DPO paper uses the sequence log-ratio, which is the sum over tokens. For PPO and GRPO the appendix assumes $\pi_{\theta_{\mathrm{old}}} = \pi_\theta$, the single-update regime above, so each coefficient is an advantage plus, for GRPO, the KL term.

Read row by row, the table separates methods on two axes. SFT reinforces every token of a human-chosen output equally, RFT and Online RFT reinforce correct outputs and ignore incorrect ones, and PPO and GRPO assign signed, reward-dependent coefficients that push incorrect outputs down. RFT and DPO are offline (outputs sampled from the SFT model); Online RFT, PPO and GRPO are online (outputs sampled from the current policy).

What the ablations show

Figure 5 trains DeepSeekMath-Instruct 1.3B with RFT, Online RFT, GRPO with outcome supervision and GRPO with process supervision, and tracks GSM8K and MATH accuracy over training steps.

• Online RFT matches RFT early and pulls ahead later. The paper's explanation is that the policy's samples drift away from the SFT model's as training proceeds, so offline data goes stale.
• GRPO beats Online RFT. The paper attributes this to the coefficient. GRPO's (Equation 21) depends on the reward value and is negative for below-average outputs, while Online RFT's (Equation 10) is 1 for every correct output and 0 for every incorrect one.
• GRPO with process supervision beats GRPO with outcome supervision.

These are single training curves on a 1.3B model and two benchmarks, drawn without error bands, and the curves are visibly noisy.

Maj@K rises, Pass@K does not

Pass@K counts a problem as solved if any of $K$ sampled answers is correct. Maj@K counts it as solved if the most frequent final answer among $K$ samples is correct. Figure 7 compares DeepSeekMath-Instruct 7B and DeepSeekMath-RL 7B at temperature 0.7 for $K$ from 1 to 64 on GSM8K and MATH. RL raises Maj@K at every $K$. The RL gain in Pass@K disappears as $K$ grows. The two Pass@K curves start apart at $K = 1$, where Pass@1 is single-sample accuracy, and meet by $K = 16$ on MATH; on GSM8K the instruction-tuned model's curve crosses the RL model's and finishes above it at $K = 64$. Section 5.2.2 reads this as follows: "it seems that the improvement is attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities."

The mechanism is simple. Pass@K depends on which answers the model can produce within $K$ draws; Maj@K depends on which answer it produces most often. An update that moves probability onto correct answers the model already generates raises Maj@K and top-1 accuracy without enlarging the set of answers it can reach. The authors tie this to their setup, since RL used only questions from the SFT stage and plain nucleus sampling. Section 5.2.3 lists out-of-distribution prompts, tree-search decoding, reward models that generalize, and algorithms that tolerate noisy rewards as next steps. Yue et al. (2025) later ran the comparison at much larger $K$ across several RLVR-trained model families and found that base models reach higher pass@k than their RL-trained versions once $k$ is large.

What It Gets Right

The critic removal fits the reward structure. With one score per output, a per-token value function has to predict a terminal score from every prefix, which is the difficulty Section 4.1.1 names. For an outcome score with no per-token penalty, the value a critic must learn at the first token is $\mu(q)$, the expected reward for the question, and the group mean is a Monte Carlo estimate of it. GRPO gives up the per-token baselines a critic would supply beyond that point. In exchange it needs no second network, and the derivation above shows its baseline is the leave-one-out baseline up to a constant factor.

The data claim is measured where it can be isolated. Table 1 trains the same 1.3B model for the same 150B tokens on four corpora and compares them on eight benchmarks. Table 2 is weaker evidence for data quality. Its 7B model scores 36.2% on MATH against 33.6% for Minerva 540B, a model 77 times larger, but it starts from a code model and trains on a 500B-token mixture that is 56% the new corpus, so the gap does not isolate the data. The negative arXiv result comes with its limits stated.

The unified gradient turns vague comparisons into specific ones. RFT and Online RFT differ in one component, the source of the sampled outputs. Online RFT and GRPO both sample from the current policy but differ in the reward source (a correctness rule or a trained reward model) and in the coefficient (a 0/1 indicator or a signed advantage). Figure 5 runs these methods on one 1.3B model, so it isolates the sampling choice and, as noted below, bundles the other two.

The paper reports a result that limits its own method. Figure 7 shows RL sharpening the output distribution rather than widening what the model can solve, and Section 5.2.2 says so directly.

Common Misconceptions

GRPO removes the reward model. It removes the value model. The paper's GRPO scores outputs with a trained reward model (Algorithm 1, Section 4.2), and iterative GRPO keeps retraining it. DeepSeek-R1-Zero later paired GRPO with rule-based rewards and no neural reward model (DeepSeek-R1, Section 2.2.2 of the first arXiv version).

GRPO is PPO minus the critic and nothing else. Three other changes come with it: advantages are divided by the group standard deviation, the KL anchor moves from the reward to the loss with a different estimator, and with outcome rewards every token of an output gets the same advantage. The standard-deviation division, together with the $1/|o_i|$ average, produces the difficulty and length weightings derived above. See the comparison page for how the group baseline changes gradient dynamics.

The KL term's gradient is the gradient of $\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})$. Its value is an unbiased estimate of that quantity. Treated as the loss $\beta(\lambda - \log\lambda - 1)$, its expected gradient at each visited prefix is $\beta\, \nabla_\theta \mathrm{KL}(\pi_{\mathrm{ref}} \| \pi_\theta)$. DAPO drops the term altogether for long chain-of-thought training, on the grounds that the policy is expected to move far from the initial model.

RL taught the model to solve new problems. The paper's own evidence points the other way. The Pass@K gain vanished at large $K$ while Maj@K rose at every $K$, and Section 5.2.2 attributes the gain to moving probability onto correct answers already among the model's top samples.

Figure 5 isolates the effect of negative coefficients. Going from Online RFT to GRPO changes the coefficient and the reward source at once. Online RFT uses a correctness rule, while GRPO uses a reward model trained on rule judgments (Section 5.2.1). The figure does not separate the two.

Connections to TheoremPath Topics

• Policy gradient theorem: the score function $\nabla_\theta \log \pi_\theta$ that every row of the unified gradient multiplies, and the identity behind the online rows (Online RFT, PPO and GRPO).
• Policy optimization: PPO and TRPO: the clipped surrogate GRPO keeps.
• Actor-critic methods: the learned baseline GRPO replaces with a group mean.
• KL divergence: the two argument orders separated in the KL-term analysis.
• DPO vs GRPO vs RL for reasoning: side-by-side comparison, including the zero-signal case for all-correct or all-wrong groups.
• Direct Preference Optimization: the offline method that is one row of Table 10.
• InstructGPT: the PPO pipeline whose per-token KL penalty Equation 2 cites.
• Reinforcement learning from human feedback (deep dive) and RLHF and alignment: the pipeline GRPO slots into.
• Reward models and verifiers and Verifier design and process reward: outcome and process scoring, as in Sections 4.1.2 and 4.1.3.
• Test-time compute and search: majority voting at inference, the quantity behind Maj@K.
• DeepSeek models: the model family where GRPO went next.
• Reasoning data curation and Data contamination and evaluation: the corpus loop and the n-gram decontamination rule.
• Post-training overview: where SFT, rejection sampling and RL sit in the post-training sequence.

Further Reading

• Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., et al. (2025). "Understanding R1-Zero-Like Training: A Critical Perspective." COLM 2025. arXiv:2503.20783. Names GRPO's response-length and question-difficulty biases and proposes Dr. GRPO, which removes both normalizations.
• Yu, Q., Zhang, Z., Zhu, R., et al. (2025). "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." arXiv:2503.14476. A GRPO-derived recipe with decoupled clip ranges, dynamic sampling, a token-level loss, overlong reward shaping and no KL term.
• Zheng, C., Liu, S., Li, M., et al. (2025). "Group Sequence Policy Optimization." arXiv:2507.18071. Replaces GRPO's token-level importance ratios with a sequence-level ratio and sequence-level clipping.
• Yue, Y., Chen, Z., Lu, R., et al. (2025). "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" NeurIPS 2025. arXiv:2504.13837. Re-tests the Pass@K observation at large $k$ across model families, RL algorithms and benchmarks.
• Ahmadian, A., Cremer, C., Gallé, M., et al. (2024). "Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs." arXiv:2402.14740. The leave-one-out baseline that mean-centering matches up to a factor $(G-1)/G$.
• DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948. Trains DeepSeek-R1-Zero from DeepSeek-V3-Base with GRPO and rule-based rewards, then reuses the same RL process for DeepSeek-R1 after a cold-start fine-tuning stage.

References

Canonical:

• Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300. Section, table and equation numbers on this page follow v3 (27 April 2024). Code and models: github.com/deepseek-ai/DeepSeek-Math.

Direct precursors:

• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). "Proximal Policy Optimization Algorithms." arXiv:1707.06347. The clipped surrogate in Equations 1 and 3.
• Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). "High-Dimensional Continuous Control Using Generalized Advantage Estimation." arXiv:1506.02438. GAE, the advantage estimator GRPO replaces.
• Ouyang, L. et al. (2022). "Training language models to follow instructions with human feedback." NeurIPS 2022. arXiv:2203.02155. The per-token KL penalty in Equation 2.
• Wang, P., Li, L., Shao, Z., et al. (2023). "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations." arXiv:2312.08935. The recipe the paper follows to build its reward-model training set, and the source of its process supervision setup.
• Yuan, Z., Yuan, H., Li, C., et al. (2023). "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models." arXiv:2308.01825. Rejection-sampling fine-tuning (RFT).
• Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." NeurIPS 2023. arXiv:2305.18290.
• Schulman, J. (2020). "Approximating KL Divergence." Blog post, joschu.net/blog/kl-approx.html. The estimator in Equation 4.

Data and evaluation:

• Paster, K., Dos Santos, M., Azerbayev, Z., & Ba, J. (2023). "OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text." arXiv:2310.06786. The seed corpus.
• Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016). "FastText.zip: Compressing text classification models." arXiv:1612.03651. The classifier library cited for the recall step.
• Azerbayev, Z. et al. (2023). "Llemma: An Open Language Model For Mathematics." arXiv:2310.10631. Proof-Pile-2 and the Llemma baselines.
• Lewkowycz, A. et al. (2022). "Solving Quantitative Reasoning Problems with Language Models." NeurIPS 2022. arXiv:2206.14858. Minerva.
• Hendrycks, D. et al. (2021). "Measuring Mathematical Problem Solving With the MATH Dataset." arXiv:2103.03874.
• Cobbe, K. et al. (2021). "Training Verifiers to Solve Math Word Problems." arXiv:2110.14168. GSM8K.
• Wang, X., Wei, J., Schuurmans, D., et al. (2022). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." arXiv:2203.11171. Majority voting over sampled answers, the procedure behind Maj@K.

Follow-up work:

• DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948. A revised version appeared in Nature 645, 633-638 (2025).
• Liu, Z. et al. (2025). "Understanding R1-Zero-Like Training: A Critical Perspective." COLM 2025. arXiv:2503.20783.
• Yu, Q. et al. (2025). "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." arXiv:2503.14476.
• Zheng, C. et al. (2025). "Group Sequence Policy Optimization." arXiv:2507.18071.
• Yue, Y. et al. (2025). "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" NeurIPS 2025. arXiv:2504.13837.
• Ahmadian, A. et al. (2024). "Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs." arXiv:2402.14740.

Textbook:

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Chapter 13, Sections 13.3 (REINFORCE), 13.4 (REINFORCE with Baseline) and 13.5 (Actor-Critic Methods).