Let's Verify Step by Step

Overview

Lightman et al. (2023) ask which training signal gives the better reward model for multi-step math solutions: a label on the final answer only, or a label on every step. The first is outcome supervision and the second is process supervision, following Uesato et al. (2022), who compared the two on the grade-school benchmark GSM8K and found similar final-answer error rates. This paper reruns the comparison with a GPT-4 base model, far more human feedback, and the harder MATH dataset.

The experiment holds the solution generator fixed and changes only the reward model. Each reward model is scored by best-of-N reranking: sample $N$ solutions to a test problem, keep the one the reward model ranks highest, and check its final answer. On a 500-problem subset of the MATH test set with $N = 1860$, the process-supervised reward model (PRM) selects a correct solution 78.2% of the time, the outcome-supervised reward model (ORM) 72.4%, and majority voting over final answers 69.6% (Figure 3). The PRM is trained on PRM800K, about 800,000 human step-level labels that the paper releases.

The large-scale ORM and PRM are trained on different data, so that 5.8-point gap is not a controlled comparison. The authors say so and add a small-scale experiment in which one large PRM labels the training data for smaller reward models, making the form of supervision the only variable. Process supervision wins there at every data size tested. In the same synthetic setup, a rule that picks convincing wrong-answer solutions for labeling is estimated to be 2.6 times more data efficient than uniform selection (Section 4.2).

The paper does not train the generator against either reward model (Section 2.1). Every accuracy on this page is a reranking result, not the accuracy of a policy trained with the reward.

Mathematical Contributions

Setup

Fix a problem $x$ with a known final answer. The generator $G$ is the base model fine-tuned for one epoch on few-shot-generated solutions to MATH training problems that reach the correct final answer. The fine-tuning is meant only to teach it to write one step per line (Section 2.3). A solution is therefore a sequence of steps $y = (s_1, \ldots, s_T)$. An automatic grader returns $g(x, y) \in \{0, 1\}$ by comparing the final answer with the reference answer. It never reads the steps, so a solution that reaches the right answer through an invalid step gets $g = 1$. The paper calls these false positives (Section 2.5).

Large-scale models start from the GPT-4 base model, which was pretrained only on next-token prediction with no RLHF. Small-scale models have a similar design with roughly 200 times less pretraining compute. Both are first fine-tuned on MathMix, a set of about 1.5B math-relevant tokens, with a 1B-token variant for the small models (Section 2.2, Appendix A).

Outcome supervision

The ORM follows the token-level verifiers of Cobbe et al. (2021). Its training solutions are sampled uniformly from the generator, 100 per problem at large scale (Section 3), and each is labeled with $g(x, y)$. The model makes a prediction at every token, and every token of a solution carries the same target (Appendix E). The solution score is the prediction at the final token:

$$r_{\mathrm{ORM}}(x, y) = \hat p_\phi\big(g(x, y) = 1 \mid x, y\big).$$

The label is one bit per solution. On a wrong solution the model has to infer which step caused the failure. Section 6.1 calls this the credit-assignment problem and offers it as the explanation for the gap. On hard problems most sampled solutions contain an error somewhere, so a negative outcome label says little.

Process supervision

Labelers rate each step $+1$ (positive), $0$ (neutral), or $-1$ (negative). Under the instructions (Appendix D), a step is neutral if it is appropriate in context, reasonable, correct, and contains only computations that are easy to verify. It is positive if it meets those conditions and also moves toward the solution, and negative otherwise. Labelers see the reference final answer but not a reference solution.

The PRM is the base model fine-tuned to predict that rating as a single token placed after the last token of each step, trained by maximum likelihood on those target tokens (Section 2.6, Appendix F.1). One forward pass over a solution returns all step predictions. Write

$$q_t(x, y) = \hat p_\theta\big(\ell_t \geq 0 \mid x, s_1, \ldots, s_t\big)$$

for the predicted probability that step $t$ is not negative. Counting neutral steps as positive is the paper's default. The solution score is the product

$$r_{\mathrm{PRM}}(x, y) = \prod_{t=1}^{T} q_t(x, y),$$

described in the paper as the probability, under the PRM, that every step is correct.

Process labels stop at the first incorrect step (Section 2.6). The authors chose this to make the comparison more direct, and it keeps human labeling cost comparable, because without an easy-to-check final answer, deciding whether a solution is correct amounts to finding its first mistake. On a correct solution, both kinds of label say that every step is fine. On an incorrect one, both say that a mistake exists, and the process label also says where the first one is. In the synthetic experiments of Section 4, where one model supplies both label types (Appendix H), the outcome label is a function of the process label, namely the event that no step is marked negative. The process label then carries at least as much information, and the only extra information is the location of the first error.

A reading of the product rule (ours, not the paper's). Because supervision stops at the first incorrect step, a step is supervised only when no earlier step on its trajectory was marked incorrect. Each $q_t$ therefore estimates a conditional probability, and the chain rule gives

$$\Pr(\ell_1 \geq 0, \ldots, \ell_T \geq 0) = \prod_{t=1}^{T} \Pr\big(\ell_t \geq 0 \mid \ell_1 \geq 0, \ldots, \ell_{t-1} \geq 0\big).$$

With the PRM outputs in place of the factors on the right, the product is the model's estimate of the left side.

Appendix F.2 varies two choices, the treatment of neutral steps and the reduction over steps, and reports all four combinations (Table 4, best-of-1860):

The spread is 0.8 points. Each factor lies in $[0, 1]$, so adding steps can only lower the product; the paper notes the resulting bias against long solutions. For the same reason, $\prod_t q_t \leq \min_t q_t$.

Best-of-N as the evaluation

For a test problem, draw $y_1, \ldots, y_N$ from $G(\cdot \mid x)$, select $\hat y = \arg\max_i r(x, y_i)$, and record $g(x, \hat y)$. The reported number is the mean of $g(x, \hat y)$ over test problems (Section 2.1). The large-scale evaluation generates 1,860 solutions per problem, and the curves for $N \leq 1000$ are computed over many subsamples of that pool (Figure 3). The released scored samples omit solutions that did not reach an answer within 1,024 tokens, so some problems have fewer than 1,860 (PRM800K repository). Majority voting ignores the reward model and returns the most common final answer among the $N$ samples, the self-consistency rule of Wang et al. (2022).

Two facts bound every curve in Figure 3. Let $p_x = \Pr(g(x, y) = 1)$ for one draw $y \sim G(\cdot \mid x)$.

1. At $N = 1$ there is nothing to choose between, so every selection rule has accuracy $p_x$.
2. The selected solution is correct only if some sample is correct, so accuracy is at most $1 - (1 - p_x)^N$. A perfect verifier attains this ceiling (see verifier design and process reward).

A learned reward model falls between the two. The next result locates it.

Proposition (best-of-N accuracy is a posterior at the top score). Fix a problem. Let $(C_1, S_1), \ldots, (C_N, S_N)$ be independent copies of a pair $(C, S)$, where $C \in \{0, 1\}$ records whether a sampled solution is correct and $S$ is its reward-model score. Assume the distribution function $H$ of $S$ is continuous, so ties have probability zero. Let $\pi(s) = \Pr(C = 1 \mid S = s)$, let $S_{(N)} = \max_i S_i$, and let $i^\star$ be the index attaining it. Then

$$\Pr(C_{i^\star} = 1) = \mathbb{E}\big[\pi(S_{(N)})\big].$$

If also $\pi(s) \to \pi^\star$ as $s$ increases to $s^\star = \sup\{s : H(s) < 1\}$, then $\Pr(C_{i^\star} = 1) \to \pi^\star$ as $N \to \infty$.

Proof. The samples are exchangeable and ties have probability zero. Conditioning on $S_1$ and using independence,

$$\Pr(C_{i^\star} = 1) = N \Pr\big(C_1 = 1 \text{ and } S_j < S_1 \text{ for all } j \geq 2\big) = N\, \mathbb{E}\big[\pi(S_1)\, H(S_1)^{N-1}\big].$$

The same argument with $\pi$ replaced by any bounded measurable $\varphi$ gives $\mathbb{E}[\varphi(S_{(N)})] = N\, \mathbb{E}[\varphi(S_1) H(S_1)^{N-1}]$, which proves the identity. For the limit, $S_{(N)}$ increases to $s^\star$ almost surely, and $S_{(N)} < s^\star$ almost surely because a continuous $H$ puts no mass at $s^\star$. Since $0 \leq \pi \leq 1$, bounded convergence gives $\mathbb{E}[\pi(S_{(N)})] \to \pi^\star$. $\square$

At $N = 1$ the formula returns $\mathbb{E}[\pi(S)] = p_x$ for every reward model. At large $N$ it depends only on how often the highest scores go to correct solutions. A reward model can classify most solutions correctly and still lose at large $N$ if a set of wrong solutions with positive probability scores above every correct one. As $N$ grows, a member of that set becomes likely to appear in the pool, and it gets selected. Which reward model wins is a property of the upper tail of its scores, so a comparison has to name $N$. The paper's curves subsample a fixed pool rather than drawing fresh samples, which the independence assumption only approximates. The proposition is consistent with two observations in the paper, though it does not show that it explains them. In Figure 3 the ORM curve flattens at large $N$ while the PRM curve keeps rising, and on the easiest fifth of problems the ORM's accuracy falls slightly as $N$ grows while the PRM's does not (Appendix G). See order statistics for the distribution of $S_{(N)}$.

Headline results

Large-scale MATH evaluation, 500 held-out problems (Figure 3):

The PRM is above both baselines at every $N$ plotted, and Section 3 states that "the performance gap widens as N increases." Combining the PRM with majority voting through RM-weighted voting did not noticeably improve performance. Appendix G splits the test set into quintiles by the generator's pass rate. The PRM's advantage appears in all five. The authors also write that more samples have the largest positive effect on the hardest quintile. That holds in relative terms, but in Figure 6 the absolute gains in PRM accuracy across the plotted range of $N$ are larger in quintile 3 (about 0.44 to 0.93) and quintile 4 (about 0.18 to 0.75) than in quintile 5 (about 0.02 to 0.24).

Out-of-distribution check, best-of-100 on recent exam problems released after the pretraining data was compiled (Section 5, Table 1):

The PRM leads on every exam. The ORM is below majority voting on all three AP exams, so its aggregate lead over majority voting comes from AMC10/12. The text of Section 5 gives the held-out set as 224 questions while the problem counts in Table 1 sum to 234. The arXiv v1 and ICLR 2024 versions both carry the two figures.

PRM800K

All labeled solutions come from the large-scale generator (Section 2.4). The filtered training set contains 800K step-level labels across 75K solutions to 12K problems. The raw release has 1,085,590 step-level labels over 101,599 solutions. Filtering drops quality-control labels and labels the labeler could not complete (Appendix B).

Collection ran in two phases (Appendix B, Appendix D):

• Phase 1, about 5% of the data (about 40,000 labels). Labelers rated several alternative completions at each step and could write their own step when every candidate was negative.
• Phase 2, the remainder, in 10 generations. Whole solutions were generated in advance, labeling stopped at the first negative step, the current best PRM ranked each generation's samples, and the highest-scoring wrong-answer solutions went to labelers. The PRM was retrained between generations.

The selection rule changes the class balance sharply (Table 3):

Labelers had to agree with gold labels on at least 75% of 30 screening questions before phase 2, and 10 to 20 quality-control problems per generation were served at random during the work.

The split is nonstandard. MATH has 12,500 problems (Hendrycks et al., 2021), 7,500 of them for training. To reduce overfitting as more solutions per problem were labeled, the authors moved 4,500 test problems into training and evaluate on the remaining 500, chosen uniformly at random. Figure 5 shows that their subject and difficulty mix is representative of the full test set (Appendix C). The repository publishes this split, the labeling instructions, the answer grader, and the scored test samples. Its README describes the grader as conservative. It sometimes rejects correct answers and may occasionally accept incorrect ones.

Active learning

The paper expects the most information from labels where the current PRM is wrong. Section 2.4 therefore selects convincing wrong-answer solutions: rated highly by the current best PRM, with a final answer that does not match the reference. The product score makes the argument exact. If $r_{\mathrm{PRM}}(x, y) \geq \tau$, then $q_t(x, y) \geq \tau$ for every step, because each factor is at most 1. So if a wrong final answer does come from an incorrect step, the PRM gave that step probability at least $\tau$ of being fine. The qualifier matters, since the grader can reject a correct answer (Section 2.4 notes occasional misgrading).

The human collection was not ablated at scale. The 2.6x estimate comes from the synthetic setup (Section 4.2):

1. Train a small reward model, $\mathrm{PRM}_{\text{selector}}$, on one sample per problem, and use it to score 1,000 samples per problem.
2. For a budget of $n$ samples per problem, take 80% as the most convincing wrong-answer samples and 20% as the most convincing of the rest, right or wrong.
3. Label the chosen samples with the large-scale PRM and train on those labels.

Comparing the slopes of fitted lines with and without this selection in Figure 4a gives the estimate of about 2.6 times the data efficiency of uniform labeling. The paper does not write out the calculation. The largest active-learning dataset, 200 samples per problem, falls slightly below the trend, which the authors attribute to 200 being a large fraction of the 1,000-sample pool. Retraining the selector between rounds was unstable, and the resulting models were no better.

Small-scale synthetic supervision

Human process labels are expensive, so Section 4 uses the large-scale PRM, $\mathrm{PRM}_{\text{large}}$, as the labeler for smaller models. A step counts as incorrect when $\mathrm{PRM}_{\text{large}}$ gives the negative rating probability above 0.2, a threshold chosen because the model is slightly miscalibrated toward positive ratings (Appendix H). From 1 to 200 sampled solutions per problem, each dataset receives three kinds of labels:

• process labels from $\mathrm{PRM}_{\text{large}}$, up to the first step it marks incorrect
• outcome labels from $\mathrm{PRM}_{\text{large}}$, correct if and only if it marks every step correct
• outcome labels from final-answer checking

Only the labels differ between the three series. Under best-of-500 evaluation, process supervision beats both outcome variants at every dataset size (Figure 4a, mean and standard deviation over three seeds). Figure 4b takes the model trained on 200 samples per problem from each series and varies $N$. From about $N = 50$ on, outcome labels from $\mathrm{PRM}_{\text{large}}$ beat final-answer labels, by about 4 points at $N = 1000$. At $N = 20$ and below the two outcome series are within about a point of each other, and final-answer labels are slightly ahead at $N = 5$ and $N = 10$. The authors call $\mathrm{PRM}_{\text{large}}$ outcome supervision "noticeably more effective" and attribute the difference to $\mathrm{PRM}_{\text{large}}$ rejecting right answers reached by wrong steps. They treat the $\mathrm{PRM}_{\text{large}}$ outcome series as the more relevant baseline and leave the choice to the reader.

Section 7.1 connects the data scaling in Figure 4a to Uesato et al. In that figure, process supervision on one sample per problem and final-answer outcome supervision on 200 samples per problem both reach about 48.5%. The authors add that the trend shows process supervision beating outcome supervision "when scaled up". At equal budgets, though, the gap in Figure 4a narrows as the budget grows: about 22 points at one sample per problem, and about 6 to 10 points at 200.

What It Gets Right

The design isolates the reward model. The generator is frozen, no RL is run, and the metric is the accuracy of the solution the reward model picks. Differences in the curves are differences in ranking, not in what the generator learned from a reward signal.

The paper names its own confound and runs the experiment that removes it. The large-scale result compares best-effort models on different training sets. The small-scale result holds the data fixed and varies only the label type, and it reports both candidate outcome baselines and names the stronger one as the more relevant.

The extra information in process labels is limited on purpose. On a correct solution both label types say the same thing. On an incorrect one, stopping process labels at the first error means the process label adds only the location of that error, so in the controlled comparison any advantage for process supervision comes from knowing where a solution first fails, not from labels on steps after the failure.

The artifacts are public. PRM800K, the 500-problem split, the grader, the labeling instructions, and the scored test samples are all released, so the best-of-N curves can be recomputed and the same split has been reused. Snell et al. (2024) evaluate on "the dataset split consisting of 12k train and 500 test questions, used in Lightman et al." (Section 4).

Limits of the Comparison

Different training sets at scale. The PRM trains on about 800K labels chosen by active learning and dominated by wrong-answer solutions. The ORM trains on 100 uniform samples per problem, a set with no overlap that the paper describes as an order of magnitude larger (Section 3). Mixing uniform samples into PRM800K for the ORM did not help, but the 78.2% versus 72.4% comparison remains between two different pipelines.

One teacher behind the synthetic labels. In Section 4, both the process labels and the preferred outcome labels come from $\mathrm{PRM}_{\text{large}}$, a model trained on human process labels. The experiment compares two ways of distilling one process-trained teacher. It does not compare human process feedback with human outcome feedback.

The metric grades answers, not steps. Best-of-N credit depends only on $g(x, \hat y)$, so a reranker that selects a right answer reached through a wrong step is scored as correct. Zhang et al. (2025) argue in their abstract that this biases best-of-N evaluation of PRMs, because generators produce correct answers with flawed processes and PRMs that tolerate them receive inflated best-of-N scores.

One domain, one generator family. All results are on math problems, with a GPT-4 generator at large scale and smaller models of similar design at small scale. The authors say it is unknown how far the results generalize beyond math (Section 6.2). Snell et al. (2024) report that a PRM trained on PRM800K was easy to exploit with best-of-N on samples from their PaLM 2 models and hypothesize a distribution shift from the GPT-4 samples in the dataset. They trained PRMs from Monte Carlo rollouts instead (Section 5.1).

A small, nonstandard test set. The evaluation has 500 problems. A binomial standard error for a 78% success rate on 500 problems is $\sqrt{0.782 \times 0.218 / 500} \approx 0.018$, about 1.8 points. The shaded bands in Figure 3 show variation across subsamples of solutions, not across draws of test problems, and the ORM and PRM numbers are paired on the same problems, so this figure is a scale for reading the curves rather than a test of the gap. The authors also cannot rule out MATH test problems in pretraining data (Section 6.3); they argue that contamination would affect all methods alike and point to the exam results in Section 5 as uncontaminated support.

The GSM8K reconciliation compares different things. Uesato et al. supervised the policy as well as the reward model, and their abstract reports that "pure outcome-based supervision produces similar final-answer error rates with less label supervision." They also found that reward models trained only on outcome labels agreed more closely with process labels than with outcome labels (Section 1). Section 7.1 of this paper summarizes that work as finding that process supervision reached similar results with less data. Anyone relying on the reconciliation should read Section 3 of Uesato et al. directly.

No evidence about RL. The claim of a negative alignment tax in Section 6.2 rests on reranking accuracy. Whether a PRM helps as an RL reward is a separate question. The first arXiv version of the DeepSeek-R1 report lists PRMs among its unsuccessful attempts and gives three limitations (Section 4.2): defining a fine-grained step is hard in general reasoning, judging an intermediate step is hard to automate and costly to do by hand, and, in the report's words, a model-based PRM "inevitably leads to reward hacking", while retraining it needs extra resources and complicates the training pipeline.

Common Misconceptions

"A PRM makes the model better at math." In this paper the generator never changes. The 78.2% figure is the accuracy of the solution a PRM selects from 1,860 samples of a fixed generator.

"Process supervision reaches 78% on MATH." The number is best-of-1860 on a 500-problem subset with a GPT-4 generator. It is not single-sample accuracy and not a score on the full 5,000-problem test set.

"The PRM beats the ORM by a fixed margin." At $N = 1$ every reranker has the generator's single-sample accuracy, so the gap is zero by construction. In Figure 3 it grows with $N$, and the proposition above explains why the large-$N$ ordering is governed by the top of each model's score distribution.

"PRM800K is 800,000 solutions." It is about 800,000 step-level labels on about 75,000 solutions to 12,000 problems after filtering. The raw release has 1,085,590 labels on 101,599 solutions.

"The paper scores a solution by its worst step." The default is the product of step probabilities with neutral counted as positive (Appendix F.2). The minimum is one of the four tested variants; with neutral counted as positive it scores 0.6 points lower. Snell et al. (2024) describe the minimum as the rule "done in Lightman et al." (Appendix E), which does not match Appendix F.2.

"The 2.6x data-efficiency figure comes from human labeling." It comes from the synthetic setup in which $\mathrm{PRM}_{\text{large}}$ supplies the labels. The human collection used a related selection rule but was not ablated at scale (Section 2.4).

Connections to TheoremPath Topics

• Verifier design and process reward: PRM and ORM design, verifier-guided search, and the best-of-N ceiling for a perfect verifier.
• Reward models and verifiers: learned reward models versus programmatic checks.
• Test-time compute and search: best-of-N as the simplest search over samples.
• Inference-time scaling laws: how accuracy moves with the sample budget $N$.
• Order statistics: the distribution of the maximum score $S_{(N)}$ used in the proposition.
• Reward hacking: selection against a learned score finds that score's errors as $N$ grows.
• Scalable oversight: step-level human judgment as a supervision signal.
• Reasoning data curation: choosing which solutions to label, as in the active-learning rule.
• Chain-of-thought and reasoning: the step-by-step solution format that process labels attach to.
• Data contamination and evaluation: the MATH contamination caveat in Section 6.3.
• Constitutional AI: another route to feedback with fewer human labels.

Further Reading

• Cobbe, K. et al. (2021). "Training Verifiers to Solve Math Word Problems." Introduces GSM8K and reranking with a trained verifier. The ORM here follows its token-level verifier.
• Uesato, J. et al. (2022). "Solving math word problems with process- and outcome-based feedback." The GSM8K comparison this paper extends. Its best system improves on the previous best published results, from 16.8% to 12.7% final-answer error and from 14.0% to 3.4% trace error, where trace error is measured only among solutions with a correct final answer. These are gains over prior work, not an outcome-versus-process comparison.
• Wang, P. et al. (2023). "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations." Builds process labels automatically and uses the resulting PRM both for reranking and for step-by-step PPO.
• Luo, L. et al. (2024). "Improve Mathematical Reasoning in Language Models by Automated Process Supervision." OmegaPRM: a tree search that locates the first error by binary search and collects over 1.5 million process labels with no human annotation.
• Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters." Search against process verifiers, allocated by problem difficulty. Uses the 500-problem split from this paper.
• Zheng, C. et al. (2024). "ProcessBench: Identifying Process Errors in Mathematical Reasoning." ACL 2025. 3,400 human-annotated cases for finding the earliest wrong step. Existing PRMs generalize poorly beyond GSM8K and MATH, and a PRM fine-tuned on PRM800K outperforms them.
• Zhang, Z. et al. (2025). "The Lessons of Developing Process Reward Models in Mathematical Reasoning." Compares Monte Carlo, LLM-as-a-judge, and human annotation for PRM data, and documents biases in best-of-N evaluation of PRMs.

References

Canonical:

• Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2023). "Let's Verify Step by Step." ICLR 2024. Sections 2 to 7 and Appendices A to H. Section and appendix numbering is the same in the arXiv v1 and ICLR proceedings versions.
• OpenAI. PRM800K repository. Data format, the 500-problem MATH split, the answer grader, and the scored best-of-N samples.

Direct precursors:

• Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). "Training Verifiers to Solve Math Word Problems."
• Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., & Higgins, I. (2022). "Solving math word problems with process- and outcome-based feedback." Section 3 for the final-answer and trace-error results.
• Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). "Measuring Mathematical Problem Solving With the MATH Dataset." NeurIPS 2021.
• Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR 2023. The majority-voting baseline.

Related methodology:

• Gao, L., Schulman, J., & Hilton, J. (2022). "Scaling Laws for Reward Model Overoptimization." Uses a large reward model in place of human labels, as Section 4 does, and measures overoptimization under both RL and best-of-n selection.

Follow-on work and critique:

• Wang, P. et al. (2023). "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations."
• Luo, L. et al. (2024). "Improve Mathematical Reasoning in Language Models by Automated Process Supervision."
• Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters." Section 4 for the dataset split, Section 5.1 for PRM training, Appendices E and F for aggregation and the ORM comparison.
• Zheng, C. et al. (2024). "ProcessBench: Identifying Process Errors in Mathematical Reasoning." ACL 2025.
• Zhang, Z. et al. (2025). "The Lessons of Developing Process Reward Models in Mathematical Reasoning."
• DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv v1, Section 4.2 for the PRM discussion. The arXiv record lists a journal version: Guo, D., Yang, D., Zhang, H., et al. (2025), "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning," Nature 645, 633 to 638.