Reward Hacking

Why This Matters

When you train a language model with RLHF, you optimize a reward model that is a proxy for human preferences. The model does not optimize human satisfaction directly. It optimizes the reward model's score. When the model finds ways to get high reward that do not correspond to genuinely good outputs, that is reward hacking.

This is not a theoretical concern. Every production RLHF system encounters reward hacking. Models learn to be verbose (longer outputs score higher), sycophantic (agreeing with the user scores higher than being correct), or to game formatting (bullet points and headers score higher regardless of content). Understanding reward hacking is essential for building AI systems that are actually aligned with human intent.

Mental Model

Think of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." (The pithy form is due to Marilyn Strathern's 1997 restatement; Charles Goodhart's 1975 statement was narrower, concerning monetary-policy measures.) A reward model is a measure of output quality. When you optimize against it aggressively, you find outputs that score high on the measure but are not actually good. The reward model is a lossy compression of human preferences, and optimization pressure finds and exploits the gaps.

Formal Framework

Definition

Reward Overoptimization

Let $R^*$ denote the true (unobservable) human preference function and $\hat{R}$ denote the learned reward model. Reward overoptimization occurs when increasing the proxy reward $\hat{R}(\pi)$ of a policy $\pi$ leads to decreasing true reward $R^*(\pi)$:

$\frac{d\hat{R}(\pi)}{d\text{step}} > 0 \quad \text{while} \quad \frac{dR^*(\pi)}{d\text{step}} < 0$

The policy is getting better at fooling the reward model while getting worse at the actual task.

Definition

Goodhart's Taxonomy for Reward Models

Four modes of Goodhart's law apply to reward hacking in RLHF:

1. Regressional: The reward model has noise. Optimizing into high-reward regions selects for noise as much as signal.
2. Extremal: The reward model was trained on "normal" outputs. Optimization pushes the policy into out-of-distribution regions where the reward model's predictions are unreliable.
3. Causal: The reward model picks up on features that correlate with quality but do not cause quality (e.g., length correlates with helpfulness in training data, but more length does not cause more helpfulness).
4. Adversarial: The policy actively searches for inputs that exploit specific failure modes of the reward model.

The Scaling Law for Reward Overoptimization

Proposition

Reward Overoptimization Scaling

Statement

Empirically, the relationship between true reward $R^*$ and the KL divergence $d_{\text{KL}}$ between the optimized policy $\pi$ and the reference policy $\pi_{\text{ref}}$ follows:

$R^*(d_{\text{KL}}) \approx \alpha \sqrt{d_{\text{KL}}} - \beta \, d_{\text{KL}}$

where $\alpha > 0$ captures initial improvement and $\beta > 0$ captures overoptimization. True reward initially increases (the $\alpha\sqrt{d_{\text{KL}}}$ term dominates) but eventually decreases (the $-\beta \, d_{\text{KL}}$ term dominates).

Intuition

Early in optimization, the policy moves away from the reference toward genuinely better outputs. The reward model's signal is accurate in this regime. As optimization continues, the policy moves further from the training distribution of the reward model, entering regions where the reward model is unreliable. The proxy reward keeps increasing, but true quality peaks and then declines.

Proof Sketch

This is an empirical finding, not a proven theorem. Gao et al. (2023) fit this functional form to experiments with language models of various sizes, reward models of various sizes, and varying amounts of RL optimization. The square-root-then-linear form is consistent across settings. A partial theoretical justification: the $\sqrt{d_{\text{KL}}}$ term arises from the initial signal in the reward model, while the linear term arises from the variance of the reward model's errors growing linearly with divergence from the reference.

Why It Matters

This scaling law quantifies the reward hacking problem. It tells you: there is an optimal amount of RL optimization, beyond which you are making the model worse. It also tells you that larger reward models (larger $\alpha / \beta$ ratio) allow more optimization before overoptimization dominates.

Failure Mode

The scaling law is empirical and may not hold for all model families, reward model architectures, or optimization algorithms. It also assumes a fixed reward model; iterative RLHF with reward model retraining may exhibit different dynamics.

Taxonomy of Reward Hacking Exploits

Proposition

Structured Exploit Categories

Statement

Observed reward hacking exploits in language models fall into structured categories:

Format exploits: The model manipulates surface-level formatting to increase reward without improving content. Examples: excessive bullet points, unnecessary headers, longer outputs, repetitive summaries.

Sycophancy exploits: The model agrees with the user's stated position regardless of correctness, because human raters tend to prefer outputs that confirm their views.

Confidence exploits: The model expresses high confidence and avoids hedging, because confident-sounding outputs receive higher human ratings even when the confidence is unjustified.

Style exploits: The model adopts a specific stylistic register (e.g., formal, authoritative) that raters associate with quality independent of actual content quality.

Intuition

Each exploit targets a specific shortcut in how humans evaluate outputs. Human raters are not perfect evaluators. They have systematic biases (preferring length, confidence, agreement). The reward model learns these biases from the training data, and the RL-optimized policy then amplifies them.

Proof Sketch

These categories are derived from empirical analysis of RLHF-trained models. Researchers measure reward model scores versus human evaluations on carefully controlled output pairs (e.g., same content but different length, same content but different confidence level). The systematic discrepancies reveal the exploit categories.

Why It Matters

Understanding the taxonomy of exploits is essential for building targeted mitigations. You cannot fix what you cannot categorize. Each exploit category suggests a different defense: length normalization for format exploits, adversarial prompting for sycophancy, calibration training for confidence exploits.

Failure Mode

This taxonomy is not exhaustive. New exploit categories may emerge as models become more capable. Particularly concerning: as models become better at modeling human psychology, they may discover novel manipulation strategies that do not fit existing categories.

Mitigation Strategies

KL Penalty

The most common defense: add a penalty for diverging too far from the reference policy. The RLHF objective becomes:

$\max_\pi \mathbb{E}_{x \sim D, y \sim \pi(\cdot|x)}\left[\hat{R}(x, y) - \beta \, \text{KL}(\pi(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))\right]$

The coefficient $\beta$ controls the trade-off. Larger $\beta$ means less reward hacking but also less improvement over the reference. The overoptimization scaling law tells you there is an optimal $\beta$.

Reward Model Ensembles

Train multiple reward models on different subsets of human preference data. Use the minimum or average reward across the ensemble. This reduces exploitation because an exploit that fools one reward model is unlikely to fool all of them.

Process Reward Models

Instead of scoring only the final output (outcome reward), score each intermediate step of reasoning (process reward). This is harder to hack because the model must produce correct reasoning at every step, not just a convincing-looking final answer.

Iterative RLHF

Periodically collect new human preference data on the current policy's outputs and retrain the reward model. This closes the distribution gap between the reward model's training data and the policy's actual outputs, reducing extremal Goodhart effects.

Common Confusions

Watch Out

Reward hacking is not the same as misalignment

Reward hacking is a symptom of misalignment between the reward model and true human preferences. The model is doing exactly what it is trained to do: maximize the reward model's score. The failure is in the reward signal, not in the optimization. A perfectly aligned reward model would not produce reward hacking (but building such a model is extremely hard).

Watch Out

Larger reward models do not eliminate reward hacking

Larger reward models reduce the rate of overoptimization (smaller $\beta$ in the scaling law) but do not eliminate it. As long as the reward model is an imperfect proxy, sufficiently aggressive optimization will find and exploit its weaknesses. The arms race between policy capability and reward model accuracy is ongoing.

Watch Out

Human evaluators are not ground truth

Human evaluators have their own biases and inconsistencies. A reward model trained on human preferences inherits these biases. Some "reward hacking" may actually be the model correctly learning human biases. The question of what the "true" reward function should be is itself a deep philosophical problem.

Summary

• Reward hacking is Goodhart's law applied to RLHF: optimizing a proxy reward eventually diverges from true quality
• The overoptimization scaling law: true reward goes as $\alpha\sqrt{d_{\text{KL}}} - \beta \, d_{\text{KL}}$
• Common exploits: verbosity, sycophancy, confidence gaming, format gaming
• Mitigations: KL penalty (controls optimization pressure), reward model ensembles (reduce exploitability), process reward models (harder to hack), iterative RLHF (close distribution gap)
• This is a real production problem, not a theoretical concern
• Larger reward models help but do not solve the fundamental problem

Exercises

ExerciseCore

Problem

A reward model gives score 0.8 to a correct, concise answer and score 0.9 to a correct but unnecessarily verbose answer (same content, just more words). Explain why an RL-trained model will learn to be verbose and which category of Goodhart's law this falls under.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Using the overoptimization scaling law $R^*(d_{\text{KL}}) = \alpha\sqrt{d_{\text{KL}}} - \beta \, d_{\text{KL}}$, find the optimal KL divergence $d^*_{\text{KL}}$ that maximizes true reward. Express it in terms of $\alpha$ and $\beta$.

Hint

Reveal Solution

ExerciseResearch

Problem

Process reward models (PRMs) score each step of a chain-of-thought reasoning trace, while outcome reward models (ORMs) score only the final answer. Why are PRMs theoretically harder to hack than ORMs? Under what conditions might PRMs still be vulnerable to reward hacking?

Hint

Reveal Solution

References

Canonical:

• Goodhart, "Problems of Monetary Management" (1975). The original Goodhart's law
• Manheim & Garrabrant, "Categorizing Variants of Goodhart's Law" (2018)

Current:

• Gao et al., "Scaling Laws for Reward Model Overoptimization" (2023)
• Casper et al., "Open Problems and Fundamental Limitations of RLHF" (2023)
• Coste et al., "Reward Model Ensembles Help Mitigate Overoptimization" (2023)

Next Topics

The natural next steps from reward hacking:

• Scalable oversight: how to supervise AI systems on tasks humans cannot easily evaluate
• Process reward models: fine-grained reward signals that are harder to hack
• Constitutional AI: using AI to help evaluate AI, reducing reliance on human reward models