Reward hacking in RLHF and loss imbalance in PINNs are the same optimization pathology

The RLHF literature and the physics-informed-neural-network (PINN) literature developed in parallel and rarely cite each other. Both fields have independently converged on the same fix for the same underlying problem, and stating the problem abstractly makes the convergence look less like coincidence and more like structural necessity.

The abstract problem: proxy optimization under an implicit trust region. A learning system has access to a differentiable surrogate P of a ground-truth objective T that is either unobservable or infeasible to compute directly. If the optimizer pushes on P unrestricted, it finds solutions that score highly under P while degrading under T. The cure is a regularizer that keeps the trajectory close to a reference behaviour where the gap between P and Tis known to be small.

In RLHF, the ground-truth objective is human judgment of model outputs. The proxy is a learned reward model r_phi(x, y), trained once from pairwise preference data and then frozen. The failure mode — reward hacking — is the policy finding degenerate responses that score highly under r_phi without being useful to a human. The standard fix is a KL penalty against the supervised-fine-tuning policy, beta * KL(pi || pi_sft). The coefficient beta is tuned until a target KL budget is hit, which implicitly sets the size of the trust region around the reference policy.

In a PINN, the ground-truth objective is satisfaction of a partial differential equation at every point in a domain, which is not checkable for a parameterised function except pointwise. The proxy is the PDE residual r(u_theta), evaluated on collocation points and averaged. The failure mode — which the PINN literature does not call proxy hacking but should — is the network finding functions with low residual that ignore observational data, or conversely overfitting the data in a way that violates the residual away from measurements. The standard fix is a weighted loss L_data + lambda * L_PDE, and for a decade the community tuned lambdaby hand with no principled rule. The principled rule, due to Wang, Teng & Perdikaris (arXiv:2001.04536, 2021), sets lambda to the ratio of neural-tangent-kernel traces between the data loss and the residual loss, evaluated online. This is the PINN analog of selecting beta from a target KL budget: neither coefficient is a free hyperparameter, both are determined by the geometry of the problem at the current training state.

The parallels line up term by term.

The table is long enough to be suspicious, which is the interesting feature. If the correspondence were superficial, it would break down on one or two rows. Every row lines up, including — and this is the sharpest evidence — the fact that algorithmic improvements transfer between the two fields. Causal time-weighting in PINNs is a curriculum on collocation points, which is the PINN analog of process-reward shaping in RLHF. Hard-constraint PINNs enforce boundary conditions by architectural construction, which is the PINN analog of constrained decoding. Both fields independently rediscovered the same hierarchy of fixes: coefficient tuning, loss reweighting, curriculum, architectural constraint.

This has practical consequences. First, the PINN community's explicit NTK analysis (Wang, Teng, Perdikaris 2021) is a cleaner version of what the RLHF community does implicitly when it tunes beta. The eigenvalue spectrum of the data NTK and the residual NTK sit at different scales by construction, and an online trace-ratio rescales them to match. The analogous RLHF move would be to measure the spectral properties of the KL gradient vs the reward gradient during PPO updates and set beta from their ratio. That is not standard practice — betas are typically set from downstream evaluation — but the PINN precedent suggests the spectral-ratio approach may be tighter.

Second, the shared structure suggests PINN-literature failure modes have RLHF analogs that are less studied. De Ryck & Mishra (2024) give specific PDE classes where PINNs provably cannot converge. The analogous RLHF question — are there preference distributions on which no amount of KL-regularised PPO can recover the preferred policy — has a literature but no clean provability result. The PINN-side proof technique (representation-theoretic obstruction in a function space) could plausibly transfer.

Third, and I think most useful as a framing for future work: proxy optimization with an implicit trust region is a theory of its own, not a special case of either RLHF or PINNs. A clean statement would identify the trust-region size with the norm of the proxy-ground-truth discrepancy in a suitable function space, give a Rademacher-style finite-sample bound on the achievable gap, and recover both the PINN NTK-balanced rule and the RLHF KL-constrained update as instances of the same recipe.

The working machinery for this on TheoremPath is already in place: the Rademacher-complexity and concentration-inequality machinery gives the right empirical-process bounds; the PINN page sketches the open generalisation question; the RLHF deep-dive covers the KL-regularisation derivation. Connecting them through a proxy-trust-region formalism is a one-paper project and probably a workshop note first.