Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Overview

Snell, Lee, Xu, and Kumar (UC Berkeley and Google DeepMind, 2024) ask a budget question. A language model may spend a fixed amount of extra computation on one prompt: how should it spend it, and when is that spend worth more than a larger model? All experiments use PaLM 2-S* on the 500-question MATH test split of Lightman et al. (2023). The arXiv preprint appeared on 6 August 2024, and the paper was accepted as an oral at ICLR 2025 under a slightly different title ("...than Scaling Parameters for Reasoning"). Every number reported for this paper comes from its arXiv v1 text and figures, and section and appendix numbers follow v1. The ICLR version renumbers the appendices.

The paper splits test-time methods into two families. One changes the proposal distribution: a model fine-tuned to revise its own earlier attempts. The other changes how a verifier is used: search guided by a process reward model (PRM) that scores each step of a partial solution. The authors compare the pair to MCMC, where a simple proposal is combined with a score function to sample from a harder target.

The central finding is that neither family wins everywhere. Which method helps depends on how hard the prompt is for the base model. On easy prompts, sequential revision beats parallel sampling, while beam search against the PRM loses accuracy as its budget grows. On harder prompts, a mix of parallel and sequential samples beats either extreme, and in the medium bins beam search beats best-of-N. On the hardest fifth of prompts, no method gains much. Tuning the setting separately for each difficulty bin, which the paper calls the compute-optimal strategy, reaches (for revisions) or nearly reaches (for search) the accuracy that best-of-N gets with up to 4x more generations. The two families are tuned separately. The policy never chooses between revision and search for a prompt, and the Discussion lists combining PRM tree search with revisions as future work.

The last section converts generations to FLOPs and asks whether the saved compute should instead go into pretraining a larger model. The answer is conditional. Figure 1 (right) groups questions as easy, medium and hard. In that summary the small model with test-time compute beats a model with about 14x more parameters on easy prompts at all three inference loads the paper tests. On medium and hard prompts the small model wins mainly with revisions and low inference volume. When inference tokens greatly outnumber pretraining tokens, the larger model wins every medium and hard comparison in that figure. The five-bin version (Figure 9) is less uniform. With PRM search the larger model also wins the easiest bin at the heaviest inference load, and it wins bin 3 at all three loads. The two kinds of compute do not trade one-for-one.

Two related papers reached arXiv in the six days before it. Brown et al.'s "Large Language Monkeys" appeared on 31 July 2024 and Wu et al.'s "Inference Scaling Laws" on 1 August 2024. The three are concurrent work.

Mathematical Contributions

The compute-optimal objective

Let $q$ be a prompt with correct answer $y^*(q)$, let $N$ be a compute budget, and let $\theta$ collect the test-time hyperparameters: the search algorithm and its beam width, or the split between parallel and sequential samples. Write $\mathrm{Target}(\theta, N, q)$ for the distribution over final outputs that these choices induce. Equation 1 of the paper defines the test-time compute-optimal strategy as

$\theta^*_{q, y^*(q)}(N) = \arg\max_{\theta}\; \mathbb{E}_{y \sim \mathrm{Target}(\theta, N, q)}\big[\mathbf{1}\{y = y^*(q)\}\big].$

Two properties of this definition shape the rest of the paper. The argmax is taken per prompt, so the best $\theta$ is a function of $q$ rather than one global setting. And the objective uses $y^*(q)$, which a deployed system never has. Equation 1 defines a target that any usable procedure can only approximate.

The approximation replaces dependence on $q$ with dependence on a discrete difficulty level $d(q) \in \{1, \dots, 5\}$. For each bin $b$ and budget $N$, pick the best setting from a finite grid $\Theta$ on one half of the bin's questions:

$\hat\theta_b(N) = \arg\max_{\theta \in \Theta}\; \widehat{\mathrm{Acc}}_{\text{fold 1}}\big(\theta, N \mid d(q) = b\big),$

then report the accuracy of $\hat\theta_b(N)$ on the other half, swap the halves, and average. This is the two-fold cross-validation inside each difficulty bin described in Section 3.2. The symbols $\Theta$ and $\widehat{\mathrm{Acc}}$ are this page's shorthand.

The budget unit is the generation: one answer sampled from the base model. Best-of-N costs $N$ generations; beam search with $N$ beams costs $N$; lookahead search with $k$ lookahead steps costs $N(k+1)$ (Section 5.3). The unit counts samples, and Section 7 converts it to FLOPs.

Difficulty bins from base-model pass rates

Following Lightman et al., difficulty is measured on the base model itself. For each test question, estimate the base model's pass@1 from 2048 samples,

$\hat p(q) = \frac{1}{2048}\sum_{j=1}^{2048} \mathbf{1}\{y_j = y^*(q)\}, \qquad y_j \sim \pi_{\text{base}}(\cdot \mid q),$

and cut the test set into five quantile bins of $\hat p$. Bin 1 holds the questions the base model solves most often, and bin 5 the ones it solves least often. The authors report that these model-specific bins predicted the effect of test-time compute better than MATH's hand-labeled difficulty levels.

That is oracle difficulty, since it needs $y^*(q)$. Predicted difficulty replaces the correctness indicator with the verifier's final-answer score, averaged over the same 2048 samples, and bins the same way (Appendix C). The predicted bins give trends similar to the oracle bins (Figures 11 and 12). Neither kind is free, since both spend 2048 samples per question, and the experiments do not charge that cost to the compute-optimal policy. Section 3.2 states this and points to cheaper difficulty estimation as future work (Section 8).

The verifier: a PRM trained on Monte Carlo values

The authors first tried the human step labels in PRM800K (Lightman et al.). A PRM trained on them was easy to exploit even with plain best-of-N on PaLM 2 samples. The authors attribute this to distribution shift between the GPT-4 solutions in PRM800K and PaLM 2 outputs. They switched to the label-free recipe of Math-Shepherd (Wang et al.). For a solution prefix $s_{1:t}$ sampled from the base policy, the step's target is the probability that the base policy, continued from $s_{1:t}$, ends at a correct answer. The paper estimates it with 16 Monte Carlo rollouts per step. With a 0/1 terminal reward, the estimate is the rollouts' success rate:

$v(s_{1:t}) = \Pr\big[\text{correct final answer} \mid s_{1:t}\big], \qquad \hat v(s_{1:t}) = \frac{1}{16}\sum_{i=1}^{16} \mathbf{1}\{\text{rollout } i \text{ is correct}\}.$

The PRM outputs $\hat y_\phi(s_{1:t}) \in (0, 1)$ and is trained with binary cross-entropy against these soft targets,

$\ell(\phi) = -\Big[\hat v \log \hat y_\phi + (1 - \hat v) \log\big(1 - \hat y_\phi\big)\Big],$

using AdamW (learning rate 3e-5, batch size 128) on 16 sampled solutions per training question, with unparsable solutions removed (Appendix D). The trained PRM is a value function for the base sampling policy under a sparse terminal reward. It does not judge whether a step is valid in isolation.

Two aggregation choices turn step scores into an answer. Step-wise, the solution's score is the PRM output at the last step. This beat the minimum and the product over steps (Appendix E, Figure 13), although earlier work with binary labels found the minimum best. Last-step scoring uses the PRM like an outcome reward model (ORM), yet this PRM still beat an ORM trained directly (Appendix F, Figure 14). In Appendix E the authors take this as a sign that per-step training may be useful mainly as representation learning rather than only as a scorer at inference time. Across answers, the paper uses best-of-N weighted selection from Li et al.: sum the scores of all samples that reach the same final answer and return the answer with the largest total,

$\hat a = \arg\max_{a} \sum_{j=1}^{N} \mathbf{1}\{\mathrm{ans}(y_j) = a\}\, \hat y_\phi(y_j).$

Setting every score to 1 recovers majority voting, which is the paper's verifier-free baseline.

Search against the PRM

The paper compares three search procedures, all sampling from the base model with a 4-shot prompt (Figure 2 illustrates them, and Appendix G gives the prompt).

Best-of-N weighted. Sample $N$ complete solutions independently and apply the selection rule above.

Beam search. Similar to BFS-V, with $N$ beams and beam width $M$:

1. Sample $N$ candidates for the first step.
2. Score each prefix with the PRM.
3. Keep the top $N/M$ prefixes.
4. From each survivor, sample $M$ proposals for the next step, which restores $N/M \times M = N$ prefixes. Return to step 2.

Search stops at the end of a solution or after 40 rounds, and best-of-N weighted selection picks among the $N$ finished candidates.

Lookahead search. At each beam step, roll every candidate forward up to $k$ further steps at temperature 0 and score the candidate with the PRM value at the end of that rollout. Setting $k = 0$ recovers beam search. The authors describe this as Monte Carlo tree search with the stochastic exploration terms removed, since the value function (the PRM) is already trained and frozen.

The sweep covers budgets up to 256 generations: beam width $M = \sqrt N$, beam width $M = 4$, lookahead $k = 3$ on both, and lookahead $k = 1$ with $M = \sqrt N$. Figure 3 (left) gives three results. Beam search beats best-of-N at small budgets. Its advantage shrinks as the budget grows, and at large budgets it often falls below best-of-N. Lookahead search, which spends $k$ extra samples per beam on rollouts, generally underperforms the others at equal budget. The authors trace the diminishing returns to exploitation of the PRM. Search finds solutions that end in repetitive, low-information steps, or solutions only one or two steps long, that the PRM scores highly.

Splitting by difficulty (beam search with $M = 4$ against best-of-N at budgets 4, 16, 64 and 256, in Figure 3, right) shows opposite trends that the aggregate curve hides:

Selecting the search setting per bin (Figure 4) nearly matches PRM best-of-N at 64 generations while using 16, a factor of 4, in the low-budget regime. At higher budgets the gain with predicted bins shrinks, while oracle bins keep improving.

Revising the proposal distribution

Prompting an off-the-shelf model to fix its own mistakes mostly fails on math problems (Huang et al., 2023), so the authors fine-tune PaLM 2-S* as a revision model, adapting the RISE recipe of Qu et al. (2024). Running multi-turn on-policy rollouts was too expensive in their setup, so they build trajectories after the fact (Section 6.1, Appendix H):

• Sample 64 solutions per training question in parallel at a higher temperature.
• For each correct solution, draw $K$ uniformly from $\{0, 1, 2, 3, 4\}$ and place $K$ incorrect solutions from the same pool in context before it. The last incorrect solution is the one closest to the correct one in character edit distance, and the rest are random.
• Fine-tune on the correct solution as the target.

The edit-distance pairing is meant to teach the model to fix a nearby wrong answer rather than ignore the context and start over. At inference the model generates a chain of revisions, and the context is truncated to the four most recent attempts. Pass@1 rises with each revision step, including steps past the four seen in training (Figure 6, left).

The chain has a failure mode. The model was trained only with wrong answers in context, but at test time a correct answer can appear there, and about 38% of correct answers get revised into incorrect ones under naive use. The fix is to never trust the last answer and to select across the whole chain with majority voting or a verifier. The base-model PRM transferred poorly to revision outputs, so the authors trained a separate ORM on the revision model's outputs, with earlier attempts in its context (Appendix J). With several chains, verifier selection is hierarchical: best-of-N weighted within each chain, then again across the chain winners. Majority voting instead pools every answer from every chain into one vote, because hierarchical voting behaved poorly when the chains were short or few (Appendix I). At equal budget, $N$ sequential revisions narrowly beat $N$ parallel samples from the same revision model under both selection rules (Figure 6, right).

The sequential-to-parallel ratio

A budget of $N$ generations can be split into $n_{\text{par}}$ independent chains of $n_{\text{seq}}$ revisions each:

$N = n_{\text{par}} \cdot n_{\text{seq}}, \qquad \rho = \frac{n_{\text{seq}}}{n_{\text{par}}}.$

A single chain ($n_{\text{par}} = 1$) gives $\rho = N$, pure sequential revision, and $N$ chains of length one give $\rho = 1/N$, which is best-of-N. The paper's example midpoint is $\sqrt N$ chains of length $\sqrt N$, so $\rho = 1$. The names $n_{\text{par}}$, $n_{\text{seq}}$ and $\rho$ are this page's. The paper plots the same quantity as the "sequential/parallel ratio".

The authors' intuition is that parallel samples search globally over different high-level approaches, while revisions refine locally an answer that is already close. The sweeps support a trade-off (Figure 7, verifier selection):

• At a fixed budget, more sequential compute tends to beat more parallel compute, but at higher budgets the best accuracy sits at an intermediate ratio.
• At $N = 128$, questions in the easier bins do best with fully sequential compute, while harder questions do best at an intermediate ratio.
• Under majority selection an ideal intermediate ratio still exists at a fixed budget. Across bins the pattern differs: easier questions are mostly insensitive to the ratio, and only harder questions show a best intermediate ratio (Appendix B, Figure 10).

Choosing $\rho$ per difficulty bin (Figure 8) beats parallel best-of-N while using 64 generations instead of 256. Parallel sampling plateaus at the larger budgets, while the compute-optimal curve keeps rising.

One negative result sits in Appendix K. Training the revision model further with ReST$^{\text{EM}}$, a simplified RL algorithm, made additional sequential revisions substantially hurt accuracy. The authors hypothesize that the online data exacerbates spurious correlations in the revision data. The revision gains are sensitive to how that data is collected.

Exchanging pretraining and inference FLOPs

Section 7 prices the two kinds of compute with the standard approximations. Write $P$ for parameter count (the paper writes $N$, which this page reserves for the generation budget), $D_{\text{pre}}$ for pretraining tokens and $D_{\text{inf}}$ for the total tokens generated at inference. Then

$X = 6\, P\, D_{\text{pre}}, \qquad Y = 2\, P\, D_{\text{inf}}$

are the pretraining and inference FLOPs. The paper takes the first from Hoffmann et al. and the second from Sardana et al. Scale the parameter count by a factor $\lambda$ (the paper's $M$) with pretraining data held fixed, which the authors match to how the LLaMA models were scaled and which leaves Chinchilla-style joint scaling to future work. Both terms grow by $\lambda$, because the larger model also costs $\lambda$ times more per generated token, so the total is $\lambda(X + Y)$.

The smaller model matches this total if its inference compute is multiplied by $K$:

$X + K\,Y = \lambda\,(X + Y) \quad\Longrightarrow\quad K = \lambda + (\lambda - 1)\,\frac{X}{Y} = \lambda + \frac{3(\lambda - 1)}{R}, \qquad R = \frac{D_{\text{inf}}}{D_{\text{pre}}}.$

The ratio $R$ carries the deployment assumption. With little inference relative to pretraining ($R \ll 1$), the pretraining FLOPs saved by not scaling up, $(\lambda - 1)X$, are large compared with $Y$ and buy many extra samples. With heavy inference ($R \gg 1$), $K$ falls toward $\lambda$.

The paper compares against a model with about 14x more parameters at three loads, $R = 0.16$, $0.79$ and $22$. Taking $\lambda = 14$ gives the following multipliers, computed on this page from the formula above:

If the larger model answers each question with a single greedy sample and all generations have similar length, $K$ is roughly the number of generations the small model may draw. The three values land near 256, 64 and 16, which lie on the paper's power-of-two budget grid.

The bars in Figure 1 (right) report, for easy, medium and hard questions, the "relative improvement in accuracy from test-time compute" in percent. A positive value means the small model with FLOPs-matched test-time compute beats the larger model's greedy accuracy. Read from the bar labels:

In this three-group view, easy questions favor test-time compute at every load for both methods. Hard questions favor the larger model everywhere except revisions at $R \ll 1$, where the gain is +11.8%, and PRM search at $R \ll 1$, where the two tie. Revisions win the comparison more often than PRM search does.

Figure 9 shows the same comparison per difficulty bin, with the larger model's greedy accuracy as a star at each load's FLOPs-matched budget ($2^4$, $2^6$ and $2^8$ generations). The five bins are less tidy than the three groups. In the revisions panel the small model beats the larger one on bin 1 at every load. In the PRM search panel the bin-1 star at $R \gg 1$ sits near 86%, above the compute-optimal curve near 79%, and the two roughly tie at the lower loads. The bin-3 stars in that panel, near 36%, sit above the curve at all three loads. Section 7 states the overall pattern in words. With mostly bin 4 and 5 questions, or with large $R$, pretraining is the better use of FLOPs; with mostly bin 1 to 3 questions (sometimes bin 4), or with low inference load, test-time compute is. The paper's own takeaway is that test-time and pretraining compute are not exchangeable one-for-one.

What It Gets Right

The paper turns "does test-time compute help?" into an allocation problem with an explicit objective (Equation 1), a budget unit, and a selection rule evaluated out of fold. Earlier work reported mixed results for self-revision and search on math. The difficulty split offers one explanation for that disagreement, since the same method helps in one bin and hurts in another. At high budgets, beam search and best-of-N look similar in aggregate, while the per-bin view (Figure 3, right) shows beam search losing ground on easy questions and winning on medium ones.

It reports its own failure modes: beam search degrading on easy prompts, lookahead losing to cheaper search, 38% of correct answers revised into wrong ones, the ReST$^{\text{EM}}$ revision model getting worse with more revisions, and negligible gains on the hardest bin. A reader can see where the method breaks as well as where it works.

The FLOPs comparison uses two standard approximations and one line of algebra, and its conclusion is stated as a function of $R$. A reader can substitute their own ratio of inference tokens to pretraining tokens instead of inheriting a slogan about which kind of compute is better.

It separates oracle from predicted difficulty and shows the per-bin policy keeps much of its benefit when ground truth is replaced by the verifier's own estimate, while stating that the cost of that estimate is not counted.

Common Misconceptions

"Test-time compute can replace a 14x larger model." The abstract conditions this on "problems where a smaller base model attains somewhat non-trivial success rates", and Section 7 adds the inference-load condition. In Figure 1, at $R \gg 1$ the larger model wins on medium and hard questions by 24% to 53% relative, and in Figure 9 it also wins the easiest bin under PRM search at that load. The result is a map of where the trade works and where it fails.

"Compute-optimal scaling is 4x more efficient." The abstract says more than 4x, Sections 5 and 6 say up to 4x at specific budget pairs (16 against 64 generations for search, 64 against 256 for revisions), and the Discussion says 2x to 4x. The factor is measured in generations, it is "nearly" reached rather than reached for search, and the difficulty estimate that selects the strategy used 2048 samples per question without being charged. A deployed policy must pay for its difficulty estimate.

"More search against a verifier is always better." Lookahead, the most expensive search, underperformed. Beam search fell below best-of-N on easy prompts at high budgets. Stronger optimization against an imperfect scorer finds the scorer's errors. Gao et al. (2022) measured the same pattern for reward models in RLHF, where past some point optimizing a proxy reward harder lowers the true reward.

"The PRM judges whether each step is correct." This PRM predicts the probability that the base policy reaches a correct answer from the current prefix. It is a value estimate tied to the policy that generated its training rollouts. It transferred poorly to the revision model's outputs (Appendix J), which the authors attribute to distribution shift and which is consistent with a policy-specific value function. Setlur et al. (2024) argue that a process reward should instead measure progress, the change in success probability caused by a step, under a prover policy distinct from the base policy.

"Difficulty is a sufficient statistic." The paper uses the phrase for a modeling choice: the strategy is allowed to depend on the prompt only through its bin. It is not sufficiency in the Fisher-Neyman sense; there is no likelihood and no factorization. The bins are quantiles of a noisy estimate of pass@1, and nothing in the setup rules out two prompts with equal pass rates needing different strategies.

"The results describe how current reasoning models use compute." The experiments cover one model family (PaLM 2-S*), one benchmark (MATH), a fine-tuned reviser and verifiers, and search that runs outside the model. Models trained with RL to reflect on and verify their own work within one response, such as DeepSeek-R1, spend test-time compute inside that response. The paper does not test that setting. Its Discussion lists distilling test-time outputs back into the base model as future work.

Connections to TheoremPath Topics

• Test-time compute and search: best-of-N, beam search and tree search as budgeted search procedures.
• Inference-time scaling laws: accuracy as a function of samples, and why verifiers cap the curve.
• Verifier design and process reward: outcome versus process supervision, and Monte Carlo step labels.
• Reward models and verifiers: how learned scorers are trained and where they fail.
• Reward hacking: the over-optimization that makes beam search degrade on easy prompts.
• Scaling compute-optimal training: the $6PD$ training-FLOP approximation used in Section 7.
• Scaling laws: the pretraining side of the exchange rate.
• Chain-of-thought and reasoning: the step-by-step solution format the PRM scores.
• The loop inside the model and latent reasoning: spending test-time compute inside the network rather than across samples.
• Training Compute-Optimal Large Language Models: the pretraining allocation this paper holds fixed.

Further Reading

• Brown, B. et al. (2024). "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling." arXiv:2407.21787. Concurrent work. Coverage, the fraction of problems solved by any sample, grows with the number of samples over four orders of magnitude, often log-linearly, but majority voting and reward models plateau beyond several hundred samples in domains without automatic verifiers.
• Wu, Y. et al. (2024). "Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models." arXiv:2408.00724. Concurrent work on the model-size versus inference-compute trade-off. Llemma-7B with the authors' tree search outperforms Llemma-34B on MATH across their tested inference strategies.
• Setlur, A. et al. (2024). "Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning." arXiv:2410.08146. Redefines the process reward as progress under a separate prover policy and reports test-time search against these verifiers as more than 8% more accurate and 1.5x to 5x more compute-efficient than against ORMs.
• Liu, R. et al. (2025). "Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling." arXiv:2502.06703. Extends the per-difficulty analysis across policy models and PRMs and finds the compute-optimal strategy depends on all three.
• Muennighoff, N. et al. (2025). "s1: Simple test-time scaling." arXiv:2501.19393. Controls test-time compute inside one sample by cutting the model's thinking short or extending it. Budget forcing lifts s1-32B from 50% to 57% on AIME24.

References

Canonical:

• Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters." arXiv:2408.03314. Published at ICLR 2025 (oral) under the title "Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning", OpenReview.

Methods the paper builds on:

• Lightman, H. et al. (2023). "Let's Verify Step by Step." arXiv:2305.20050. PRM800K, the MATH split, and pass-rate difficulty bins.
• Wang, P. et al. (2023). "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations." arXiv:2312.08935. Monte Carlo rollout labels for process reward models.
• Li, Y. et al. (2022). "Making Large Language Models Better Reasoners with Step-Aware Verifier." arXiv:2206.02336. Source of best-of-N weighted selection.
• Cobbe, K. et al. (2021). "Training Verifiers to Solve Math Word Problems." arXiv:2110.14168. Best-of-N selection with a learned verifier.
• Uesato, J. et al. (2022). "Solving math word problems with process- and outcome-based feedback." arXiv:2211.14275. Process versus outcome supervision.
• Qu, Y., Zhang, T., Garg, N., & Kumar, A. (2024). "Recursive Introspection: Teaching Language Model Agents How to Self-Improve." arXiv:2407.18219. The revision-model recipe the paper adapts.
• Huang, J. et al. (2023). "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR 2024. arXiv:2310.01798. Why the paper fine-tunes a reviser instead of prompting for self-correction.
• Hendrycks, D. et al. (2021). "Measuring Mathematical Problem Solving With the MATH Dataset." NeurIPS 2021. arXiv:2103.03874.
• Anil, R. et al. (2023). "PaLM 2 Technical Report." arXiv:2305.10403. The base model family.

Compute accounting:

• Hoffmann, J. et al. (2022). "Training Compute-Optimal Large Language Models." arXiv:2203.15556. Source the paper cites for $6PD$ training FLOPs.
• Sardana, N., Portes, J., Doubov, S., & Frankle, J. (2024). "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws." ICML 2024. arXiv:2401.00448. Source the paper cites for $2PD$ inference FLOPs.
• Kaplan, J. et al. (2020). "Scaling Laws for Neural Language Models." arXiv:2001.08361.
• Jones, A. L. (2021). "Scaling Scaling Laws with Board Games." arXiv:2104.03113. An earlier train-time versus test-time compute trade-off, for AlphaZero on Hex.

Critical refinements and follow-ups:

• Gao, L., Schulman, J., & Hilton, J. (2022). "Scaling Laws for Reward Model Overoptimization." arXiv:2210.10760.
• Setlur, A. et al. (2024). "Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning." arXiv:2410.08146.
• Brown, B. et al. (2024). "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling." arXiv:2407.21787.
• Liu, R. et al. (2025). "Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling." arXiv:2502.06703.
• DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." Nature 645, 633-638. arXiv:2501.12948.

Standard textbook:

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Section 8.10 (rollout algorithms) and Section 8.11 (Monte Carlo tree search), the background for lookahead search.