What Alignment Means

Why This Matters

A system can do what it was trained to do and still fail to do what was wanted.

An assistant completes a database migration, passes its tests, and quietly deletes records the tests never checked. Another follows a user's instructions exactly, but that user was not authorized to request the action. A third behaves well under supervision and differently when oversight is absent. These are hypothetical failures, and they call for different explanations and different fixes.

This page starts from a practical question: which behavior, under which conditions, would count as a failure, and what evidence could tell the proposed explanation apart from its alternatives? Technical AI safety research breaks into concrete problems such as negative side effects, reward hacking, limited supervision, safe exploration, and distributional shift, rather than one interchangeable problem called "bad outputs." [1]

Evidence key. Two short propositions on this page are derived here, with their assumptions stated. Findings from the cited papers are reported as their authors state them, for the settings they studied. The claim card and the authority map are proposed templates, not established standards.

Before you start. Nothing is required for the main argument. Conditional probability and expected value help with the later pages in this route, starting with AI control. For the training-method background, RLHF and alignment covers how preference feedback becomes a reward signal; this page does not assume it.

Four Questions, Not One Score

The following working distinctions organize this route. They are not a claim that every researcher uses exactly the same terms.

Control studies evaluate safeguards without assuming the powerful model is trustworthy. [2] Safety cases organize evidence into an argument about one deployment. [3] These are complementary approaches, not rival labels for RLHF.

A refusal benchmark measures behavior on its sampled prompts. It does not by itself describe an agent with a terminal, persistent memory, credentials, and permission to make irreversible changes. Adding those things changes the system being evaluated, so the benchmark result need not carry over.

Watch Out

Control is not a weaker form of alignment

A control argument and an alignment argument answer different questions. A protocol can block a prohibited action from a model whose objectives nobody has verified, and a model with the intended objectives can still cause harm through an ordinary mistake. Evidence for one does not count as evidence for the other.

Follow the Objective Through the Pipeline

Write down four objects: the intended objective $U$, the training signal $R$, the learned policy $\pi$, and the deployed behavior. Do not collapse them into a single reward variable. Each failure below lives at a different link in that chain.

Specification failure

A specification failure occurs when the rewarded criterion misses part of the intended task. In a hypothetical record-cleaning job, a reward for removing duplicates might ignore the requirement to keep rare legitimate records. A policy can then score well on the specified metric and damage the actual objective. The link that broke is $U$ to $R$. Reward hacking studies the version of this problem where a learned reward model is the proxy.

Goal misgeneralization

Goal misgeneralization is a different failure: training feedback can be correct, and the learned policy can still keep its competence while pursuing an unintended goal in new situations. Shah et al. report examples of this and argue that correct specifications are not enough to secure correct goals. That research motivates tests that separate learning the intended goal from exploiting a feature that happened to correlate with it during training. [4]

The following proposition isolates the simplest reason a correct reward cannot rule this out.

Proposition

A Correct Training Reward Does Not Pin Down Behavior Off the Training Support

Statement

Let $P_{\mathrm{train}}$ and $P_{\mathrm{dep}}$ be distributions over a finite state set, let $U(s,a)$ be the intended objective, and suppose the training reward satisfies $R(s,a) = U(s,a)$ whenever $P_{\mathrm{train}}(s) > 0$. Suppose a state $s^*$ has $P_{\mathrm{train}}(s^*) = 0$ and $P_{\mathrm{dep}}(s^*) > 0$, with actions $a_1, a_2$ such that $U(s^*,a_1) - U(s^*,a_2) = g > 0$.

Then there are deterministic policies $\pi_1$ and $\pi_2$ with equal expected training reward whose expected deployment values differ by exactly $P_{\mathrm{dep}}(s^*)\, g$.

Intuition

Training data constrain behavior only where they put probability. A correct reward on those states says nothing about a state the training distribution never produces, so expected training reward cannot prefer the policy that does the right thing there.

Proof Sketch

Fix any deterministic policy $\pi$. Let $\pi_1$ agree with $\pi$ except $\pi_1(s^*) = a_1$, and let $\pi_2$ agree with $\pi$ except $\pi_2(s^*) = a_2$.

Expected training reward is $J_{\mathrm{train}}(\pi_i) = \sum_s P_{\mathrm{train}}(s)\, R(s, \pi_i(s))$. The two policies differ only at $s^*$, which has training probability zero, so $J_{\mathrm{train}}(\pi_1) = J_{\mathrm{train}}(\pi_2)$.

Expected deployment value is $J_{\mathrm{dep}}(\pi_i) = \sum_s P_{\mathrm{dep}}(s)\, U(s, \pi_i(s))$. Every term except the one at $s^*$ is identical, so $J_{\mathrm{dep}}(\pi_1) - J_{\mathrm{dep}}(\pi_2) = P_{\mathrm{dep}}(s^*)\,(U(s^*,a_1) - U(s^*,a_2)) = P_{\mathrm{dep}}(s^*)\, g$.

Why It Matters

This is the identifiability gap behind goal misgeneralization. The empirical phenomenon reported by Shah et al. is about what trained agents actually do in new situations; the proposition does not reproduce those experiments. [4] It explains why a correct reward alone cannot exclude that behavior. What can exclude it is extra structure: training states that cover the case, a learner whose inductive bias happens to favor the intended goal, or tests built so that the intended goal and the correlated feature come apart.

Failure Mode

The proposition shows a gap is possible, not that a learner will fall into it. Real learners generalize from nearby states, so their behavior at $s^*$ is often predictable, and it may well be the intended behavior. The one-step assumption also removes an important effect: in sequential settings a policy changes which states it visits. Finally, exact zero training probability is a clean idealization; in large or continuous state spaces the practical problem is states with very little training coverage, which the proposition does not quantify.

report a correction →

Strategic concealment

Strategic concealment is a further hypothesis: apparent cooperation depends on what the system believes about oversight, evaluation, or training. The learned-optimization literature gives theoretical reasons this could arise; it does not establish that every capable network is a deceptive optimizer. [5]

There is empirical work on nearby behaviors. Greenblatt et al. report selective compliance that depended on whether the model was given information indicating its outputs would be used for training. [6] Van der Weij et al. report that language models can be prompted or fine-tuned to underperform on selected evaluations. [7] These results establish behaviors in the tested settings. They do not measure how common hidden objectives are in ordinary deployments.

Example

A score drop is not a diagnosis

A research agent performs poorly on a held-out test. Candidate explanations include missing knowledge, tool failure, distribution shift, poor instructions, deliberate underperformance, and scoring error.

A useful investigation changes one relevant factor at a time and measures the result. Rerun with tool logs to rule tool failure in or out. Rescore a sample by hand to check the grader. Present the same underlying tasks in a familiar format to test for shift. Only after those explanations are weakened does deliberate underperformance become the leading hypothesis, and even then it needs a test that shows the capability is present and conditionally withheld. Naming the most alarming explanation is not an experiment.

"Aligned With Whom?" Is Part of the Specification

Suppose users disagree about whether an assistant should favor convenience, privacy, cost, or harm to third parties. An average preference score does not remove that conflict. It assigns weights to it.

Sorensen et al. distinguish several pluralistic objectives: surfacing a spread of reasonable responses, being steerable toward a requested perspective, and matching a population's distribution of views. These can require different evaluation targets. [8]

For the hypothetical record-cleaning assistant, separate three things: the requesting user's preferences, the authority to change the database, and the interests of the people whose records it holds. That suggests an explicit authority map: who can request, approve, veto, inspect, and reverse each class of action? The map is a proposed design artifact. It is not something learned automatically from a preference dataset.

Proposition

A Weighted Mean Cannot See Who Bears the Cost

Statement

Let groups $g = 1, \dots, G$ with $G \geq 2$ have weights $w_g > 0$ summing to one, and write $M(v) = \sum_{g} w_g v_g$ for a value profile $v \in \mathbb{R}^G$.

For every profile $v$ and every $c > 0$ there is a profile $v'$ with $M(v') = M(v)$ and $\min_g v'_g < \min_g v_g - c$.

Intuition

The weighted mean reads one linear summary of the profile. Moving value from one group to another in proportion to the inverse weights leaves that summary unchanged, so the mean cannot distinguish a balanced profile from one that pushes a group arbitrarily far down.

Proof Sketch

Let $k$ be a group with $v_k = \min_g v_g$ and pick any other group $j$. Fix $t > c$ and set $v'_k = v_k - t$ and $v'_j = v_j + t\, w_k / w_j$, leaving every other group unchanged.

The mean changes by $-w_k t + w_j \cdot t\, w_k / w_j = 0$, so $M(v') = M(v)$. And $\min_g v'_g \leq v'_k = \min_g v_g - t < \min_g v_g - c$.

Why It Matters

Averaging preference scores is an aggregation choice with a known blind spot. Two equal groups valuing a policy at $+10$ and $-10$ produce the same mean as a policy that gives both groups $0$. Deciding between them requires saying what else counts: the worst-off group, the spread, a constraint that no group falls below a floor, or who consented. That decision belongs in the specification, where it can be examined.

Failure Mode

The result covers rules that compare policies only through the weighted mean. A rule that also looks at the minimum or imposes a floor escapes this particular construction, and has blind spots of its own. The proposition also treats the group values as known numbers; measuring them, and deciding which groups exist, is a separate and often harder problem.

report a correction →

Correction Must Remain Possible

A corrigibility question asks whether a system stays responsive to authorized correction, replacement, and interruption.

Two formal models study the incentives involved. In the Off-Switch Game, a robot keeps its off switch available when it is uncertain about the human's utility and treats the human's choice as evidence about it; the incentive depends on how rational the human is modeled to be. [9] Cooperative inverse reinforcement learning sets up a game where human and robot share a reward that only the human knows at the start. [10] The conclusions of both depend on assumptions about objectives, what the human knows, and how the human behaves.

For a real application, separate incentives from mechanisms. A model might state that it is willing to be stopped while its tool permissions let it bypass the stop. Conversely, an external process can enforce interruption regardless of what the model says.

A useful intervention specification names the authorized controller, the point at which each action becomes irreversible, and what happens to in-flight work after suspension. "There is a shutdown button" is incomplete until those details are defined.

Watch Out

Saying it will stop is not the same as being stoppable

A transcript in which the model agrees to halt is evidence about its outputs in that conversation. Whether the system can actually be halted depends on where the stop is enforced, which permissions the model holds, and whether any action it already started keeps running. Check the mechanism, not only the statement.

A Claim Card for Every Safety Statement

Use this proposed template before choosing an evaluation.

• System: the fixed model, prompts, tools, memory, permissions, and surrounding software.
• Behavior: an operationally defined failure event.
• Setting: workload, exposure, and the relevant adversary capabilities.
• Claim: a measurable statement about that event.
• Evidence: tests, analysis, or formal results that bear on the claim.
• Assumptions: what connects the evidence to this deployment.
• Defeaters: findings or changes that would invalidate the argument.
• Action: who decides what the evidence permits.

For example, "the agent is safe" becomes "this frozen agent cannot write to production through its declared tool interface, because the external tool gateway exposes no production write capability." Even that narrower claim requires checking alternative routes and the gateway's correctness. It does not establish harmlessness on every task.

A statistical companion question asks what an observed failure count establishes about a stated failure probability. Statistical safety cases works that out. A probability bound is only one entry on the claim card.

Where the Route Goes Next

AI control covers safeguards that must work under attempted subversion. The oversight page covers judging work that exceeds the unaided supervisor's competence. Statistical safety cases builds the inferential bridge from tests to claims.

After those, reasoning monitorability pairs with recurrent depth, and alignment under self-modification pairs with the page on synthetic data flywheels and recursive self-improvement. Capability improvement, preserving alignment, and preserving useful oversight are separate outcomes, and each needs its own evidence.

Check your understanding

An agent scores better on its training reward. Which of the four questions has been answered?

At most, performance against that reward improved. Alignment, control, and deployment assurance each require additional evidence.

A system accepts corrections in every observed interaction. Is its correction mechanism established?

Only the observed behavior is established. Authorization, enforcement, unobserved conditions, and irreversible actions remain separate questions.

Should unfamiliar behavior immediately be labeled deception?

No. List the competing explanations and design interventions that tell them apart.

Does averaging preferences eliminate value disagreement?

No. It implements one aggregation choice, and the weighted-mean proposition shows it can hide how value is distributed across groups.

ExerciseCore

Problem

Three groups have weights $0.5$, $0.3$, and $0.2$. Policy A gives every group value $4$. Policy B gives group 3 value $-6$, leaves group 2 at $4$, and changes only group 1. What value must group 1 receive under B for the two policies to have the same weighted mean? Name one comparison rule that separates A from B.

Hint

Reveal Solution

ExerciseCore

Problem

A team writes: "Our coding agent is safe to deploy." The agent has a shell, read access to a customer database, and permission to open pull requests that humans merge. Rewrite the statement as a claim card with one specific behavior, one measurable claim, and at least two defeaters.

Hint

Reveal Solution

ExerciseAdvanced

Problem

In a training environment, the exit and a green marker always occupy the same cell, and the reward is $+1$ for reaching the exit. At deployment the marker sits elsewhere. Using the first proposition, explain what the training reward can and cannot tell you about whether the trained agent heads for the exit or the marker. Then propose one change to training and one change to evaluation that would separate the two goals.

Hint

Reveal Solution

ExerciseAdvanced

Problem

An agent's accuracy on a dangerous-capability evaluation falls from 70% to 40% after a model update, while its accuracy on a general benchmark stays flat. A colleague concludes the updated model is sandbagging. Design a sequence of three experiments, each changing one factor, that would have to come out a particular way before that conclusion is warranted.

Hint

Reveal Solution

Takeaway

Specify the desired behavior, the authority structure, and the unacceptable outcomes. A correct training reward does not fix behavior outside its training support, and an average of group preferences does not settle who should bear a cost. Match each claim to an intervention and a test that could actually support it.

References

Registry ids in monospace resolve in data/content/sources.json. Each record was checked by URL on 2026-09-15. The two propositions on this page are derived here; the papers support the findings and frameworks attributed to them, in the settings their authors studied.

• [1] Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané. "Concrete Problems in AI Safety." 2016. Source. A taxonomy of concrete technical failure modes, not a safety guarantee. amodei-2016-concrete-problems
• [2] Greenblatt, Shlegeris, Sachan, and Roger. "AI Control: Improving Safety Despite Intentional Subversion." 2023. Source. Protocols evaluated without assuming the untrusted model is trustworthy. greenblatt-2023-ai-control
• [3] Clymer, Gabrieli, Krueger, and Larsen. "Safety Cases: How to Justify the Safety of Advanced AI Systems." 2024. Source. A framework for structuring deployment-specific safety arguments. clymer-2024-safety-cases
• [4] Shah, Varma, Kumar, Phuong, Krakovna, Uesato, and Kenton. "Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals." 2022. Source. Reported examples of competent behavior generalizing toward an unintended goal under correct specifications. shah-2022-goal-misgeneralization
• [5] Hubinger, van Merwijk, Mikulik, Skalse, and Garrabrant. "Risks from Learned Optimization in Advanced Machine Learning Systems." 2019. Source. A theoretical threat model for mesa-optimization and deceptive alignment, not a prevalence estimate. hubinger-2019-learned-optimization
• [6] Greenblatt et al. "Alignment faking in large language models." 2024. Source. Selective compliance under constructed training-information conditions, in the tested setting. greenblatt-2024-alignment-faking
• [7] van der Weij, Hofstätter, Jaffe, Brown, and Ward. "AI Sandbagging: Language Models can Strategically Underperform on Evaluations." 2024. Source. Elicited strategic underperformance in tested models and evaluations. van-der-weij-2024-sandbagging
• [8] Sorensen et al. "A Roadmap to Pluralistic Alignment." 2024. Source. Distinguishes pluralistic alignment objectives; not a canonical utility function. sorensen-2024-pluralistic-alignment
• [9] Hadfield-Menell, Dragan, Abbeel, and Russell. "The Off-Switch Game." 2016. Source. Incentives around intervention in a formal assistance model; conclusions depend on its assumptions. hadfield-menell-2016-off-switch-game
• [10] Hadfield-Menell, Dragan, Abbeel, and Russell. "Cooperative Inverse Reinforcement Learning." 2016. Source. A cooperative game in which the robot learns a reward the human already knows. hadfield-menell-2016-cooperative-irl