DPO vs GRPO vs RL for Reasoning

DPO reparameterizes the reward model through the policy itself. The implicit reward under DPO is:

$r_{\text{DPO}}(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$

The DPO loss directly optimizes on preference pairs $(y_w, y_l)$:

$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$

At convergence, this recovers the same optimal policy as RLHF with a Bradley-Terry reward model and KL penalty $\beta$.

DPO says: instead of training a reward model and then doing RL, observe that the optimal policy defines a reward function. So skip the reward model and optimize the policy directly on preference data. The log-ratio $\log(\pi_\theta / \pi_{\text{ref}})$ plays the role of the reward: increase the log-ratio for preferred outputs, decrease it for dispreferred ones.

DPO eliminates the reward model training, the PPO loop, and the associated hyperparameter tuning. This makes it simpler to implement, more stable to train, and cheaper in compute. It became the default preference optimization method for many research groups in 2023-2024.

DPO is an offline algorithm: it optimizes on a fixed dataset of preference pairs. It does not generate new outputs during training. This means it cannot explore. It only learns from the preferences in the training data. If the training data does not cover a failure mode, DPO cannot fix it. Additionally, the implicit reward can still be hacked: the model learns to increase $\log(\pi_\theta / \pi_{\text{ref}})$ for preferred outputs, which can be achieved by making the reference policy assign low probability (rather than by genuinely improving quality).

For a prompt $x$, GRPO generates a group of $K$ outputs $\{y_1, \ldots, y_K\}$ from the current policy $\pi_\theta$. Each output receives a reward $r_i$. The group-relative advantage is:

$\hat{A}_i = \frac{r_i - \text{mean}(\{r_j\})}{\text{std}(\{r_j\})}$

The GRPO objective is:

$\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}_x \left[\frac{1}{K} \sum_{i=1}^{K} \min\left(\rho_i \hat{A}_i, \, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon) \hat{A}_i\right)\right] + \beta \, \text{KL}(\pi_\theta \| \pi_{\text{ref}})$

where $\rho_i = \pi_\theta(y_i|x) / \pi_{\text{old}}(y_i|x)$ is the importance ratio (as in PPO). The key innovation: advantages are computed relative to the group, not against an external baseline or value function.

GRPO eliminates the need for a learned value function (critic). Instead of asking "how good is this output in absolute terms?" it asks "how good is this output compared to the other outputs the model just generated?" This group-relative comparison provides a natural baseline that adapts to the difficulty of each prompt.

For a hard prompt where all outputs are bad, the best-of-bad gets a positive advantage. For an easy prompt where all outputs are good, the worst-of-good gets a negative advantage. This automatic calibration stabilizes training.

GRPO was introduced by DeepSeek and used to train DeepSeek-R1 for mathematical and code reasoning. By eliminating the critic network, GRPO reduces memory requirements and removes a source of approximation error. The group-relative advantage provides richer signal than DPO's pairwise comparisons: from $K$ outputs you get $K$ advantage estimates, not just one pairwise comparison.

GRPO requires generating $K$ outputs per prompt during training, which increases compute cost. If $K$ is too small, the advantage estimates are noisy. If the reward model is unreliable, the group-relative advantage amplifies reward model errors. The model learns to distinguish between "slightly more hackable" and "slightly less hackable" outputs rather than between genuinely good and bad ones.

For reasoning tasks with a binary verifier $V(x, y) \in \{0, 1\}$ (e.g., code passes all tests, math answer is correct), the RL objective is:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, \, y \sim \pi_\theta(\cdot|x)}\left[V(x, y)\right] - \beta \, \text{KL}(\pi_\theta \| \pi_{\text{ref}})$

The gradient (via REINFORCE with baseline $b(x)$) is:

$\nabla_\theta J = \mathbb{E}\left[(V(x,y) - b(x)) \nabla_\theta \log \pi_\theta(y|x)\right]$

where $b(x)$ is typically the running average pass rate for prompt $x$.

This is the purest form of RL for reasoning: generate a solution, check if it is correct, reinforce correct solutions and suppress incorrect ones. The verifier provides ground truth rather than a learned proxy, eliminating reward model errors. The binary signal is sparse but honest.

Verifier-guided RL is how models learn to reason about math and code. Unlike preference-based methods (DPO, RLHF), the reward signal comes from objective verification, not human judgment. This eliminates reward hacking. You cannot "hack" a unit test suite or a mathematical proof checker. The limitation is that verifiers only exist for domains with checkable answers.

Binary rewards are sparse: most solutions to hard problems are wrong, giving reward 0 for most samples. This makes learning slow and high-variance. Process reward models (PRMs) address this by providing intermediate rewards for correct reasoning steps, but PRMs reintroduce the proxy reward problem. The fundamental tension: dense rewards from learned models are hackable; sparse rewards from verifiers are honest but noisy.