Alignment Under Self-Modification: What Must Survive the Next Update

Why This Matters

A successor can be more capable, less aligned, and harder to evaluate, all at once. None of those changes follows from the others.

The page on synthetic data flywheels and recursive self-improvement asks whether inherited improvements make future research more effective. This page asks what has to remain true while those improvements change the system.

A persistent update can change weights, prompts, code, tools, memory policy, the way tasks are generated, or the way outputs are evaluated. Each change can move the system outside the scope of the evidence gathered about its predecessor. A higher capability score does not certify the new configuration.

Evidence key. Four propositions are derived here from stated assumptions. They are conditional results, and none claims that a current system satisfies its hypotheses. Findings from cited work are attributed to their authors. The change manifest, the evidence-invalidation ledger, the proposed experiments, and the research agenda are proposals.

Before you start. This page builds on AI control for the trusted boundary and on statistical safety cases for zero-failure bounds and error budgets. Reasoning monitorability and martingales help with two sections, and the sequential test is developed from the statistics side in e-processes and confidence sequences.

Track the Whole Successor, Not Only Its Weights

Definition

Deployed configuration and change manifest (an accounting device)$S_j=({\theta }_j,{\pi }_j,{\mathcal{T}}_j,{\mathcal{M}}_j,V_j,P_j)$

Configuration $j$ lists model parameters $\theta_j$, agent policy and prompts $\pi_j$, tool interface $\mathcal T_j$, memory $\mathcal M_j$, evaluation machinery $V_j$, and permissions $P_j$. An evidence record $E_j$ states which of these components, and which assumptions, each safety claim concerns. A change manifest for the transition from $S_j$ to $S_{j+1}$ lists the components that changed, the claims whose scope they touch, the evaluations to repeat, the transfer assumptions being made, and who has authority to expand deployment.

The notation does not imply that all six components can be observed completely, or that a composition of individually checked components is safe. Its job is to stop a safety claim from silently referring to a system that no longer exists.

If a candidate can modify its own evaluator, agreement with that evaluator cannot be the approval criterion. An improved evaluator needs its own validity argument, using evidence the candidate did not determine. Running the evaluator on a separate model is not enough by itself, since separate models can share training data and blind spots.

Watch Out

A capability gain does not carry the safety case forward

A successor that scores higher on research tasks has passed a capability evaluation. Its predecessor's safety case was an argument about a different configuration. Which parts of that argument still apply depends on which components changed, and the change manifest is where that gets decided, not the capability score.

Three Different Things Can Be Preserved

Track all three. A mathematically valid test of the wrong version is as unhelpful as an invalid test of the right one.

Learned-optimization arguments and alignment-faking experiments give reasons to check whether apparent cooperation survives changes in training incentives and oversight information. Hubinger and colleagues set out a theoretical threat model in which a learned optimizer pursues an objective that differs from the training objective. [1] Greenblatt and colleagues told a model it would be trained only on conversations with free users. It complied with harmful queries from free users 14% of the time and almost never for paid users, and the authors note that they made this behavior easier by telling the model when and by what criteria it was trained. [2] The first is a threat model and the second is evidence from a constructed experiment. Neither shows that every self-improving system conceals its behavior.

What a Proof-Preserving Transition Guarantees

Let $\mathcal S$ be a set of fully specified system states, $I \subseteq \mathcal S$ a property, and $\mathcal A$ the set of authorized transitions, each a map from $\mathcal S$ to itself.

Proposition

An Inductive Invariant Survives Every Sequence of Authorized Transitions

Statement

Suppose $s_0 \in I$, and $F(s) \in I$ for every $F \in \mathcal A$ and every $s \in I$. Let $s_{j+1} = F_j(s_j)$, where each $F_j \in \mathcal A$ may be chosen in any way, including adaptively by the system being updated. Then $s_j \in I$ for every $j \ge 0$.

Intuition

The property is checked once at the start and once for each kind of change. Because every allowed change keeps it, no sequence of allowed changes, however cleverly chosen, can leave it.

Proof Sketch

Induction on $j$. The base case is $s_0 \in I$. If $s_j \in I$, then $F_j \in \mathcal A$ and the preservation hypothesis give $s_{j+1} = F_j(s_j) \in I$.

Why It Matters

The adaptivity clause is the point. A self-modifying system may choose its next change using everything it has learned, and the conclusion still holds, because the hypothesis covers every authorized transition from every state in $I$, not only the transitions anyone expected. This is why an approval boundary is best stated as an invariant enforced by a gateway the candidate cannot modify, for example "no reachable configuration grants the candidate write access to the evaluation store".

Failure Mode

The difficulty sits in the hypotheses. The property has to say something useful, the state model has to be faithful, and every real change has to be one of the modeled transitions. An operation outside $\mathcal A$, such as an unmodeled tool call or a manual configuration edit, falls outside the induction. A property can also hold on every state actually reached without being preserved from every state in $I$; proving it then needs a stronger inductive property, which the research exercise below constructs. A proved invariant about a restricted tool interface says nothing about the agent's honesty.

report a correction →

Example

A gateway invariant and the update that breaks it

Let $I$ say that no entry in the gateway's permission table grants write access to a protected store. Updating the weights preserves $I$: the table sits outside the weights, so no weight update can change it. Adding a tool from a fixed allowlist also preserves $I$ if no allowlisted tool can write to the table. Adding a general shell tool does not: from a state satisfying $I$ it can reach a state that violates it, so it fails the preservation check and cannot join $\mathcal A$ without a stronger invariant or a narrower tool. None of these checks says anything about whether the updated weights answer honestly.

Schmidhuber's Gödel machine is a deeper version of the same idea: a system that rewrites its own code only after proving, from axioms describing its hardware, its initial code, and a utility function, that the rewrite is useful. [3] Its optimality claims are relative to those axioms and that utility function, and the proof search has real resource costs. It is a framework for proof-triggered self-modification, not a procedure that turns an arbitrary utility function into alignment.

For a practical design, keep the approval boundary outside the candidate's modification privileges, unless changes to the boundary are themselves checked under an explicit trust model. A natural-language description of a proof is not a checked proof object.

Watch Out

A checked invariant is not a proof of alignment

A machine-checked proof that no reachable configuration can write to the evaluation store establishes exactly that, inside the model. It does not establish that the agent is honest, that its outputs are correct, or that the evaluation store was the only thing worth protecting. A proof-preserving update establishes general alignment only if the invariant actually expresses that claim in a faithful model, and that is a far harder statement to write down than an interface property.

Statistical Approval for an Adaptive Sequence of Candidates

Proof will not settle behavioral claims such as a failure rate on a task population. Suppose candidate $j$ has failure probability $p_j$ on a declared confirmation population and counts as unacceptable when $p_j \ge \epsilon_j$. The candidate and its threshold may depend on the whole earlier research history, but both are fixed before candidate $j$'s fresh confirmation data are drawn. Write $\mathcal G_{j-1}$ for the history at that moment.

Approval means rejecting the null hypothesis $H_{0,j}: p_j \ge \epsilon_j$. This puts the burden on showing a low rate, instead of treating a failure to find problems as approval.

Proposition

Error Spending Caps Lifetime False Approval Across Adaptive Candidates

Statement

Allot candidate $j$ the level $\alpha_j = \alpha/(j(j+1))$. Suppose that for every $j$, on the event $\{p_j \ge \epsilon_j\}$, the conditional probability of approving candidate $j$ given $\mathcal G_{j-1}$ is at most $\alpha_j$. Then

$$\Pr(\text{some unacceptable candidate is ever approved}) \le \alpha.$$

The same conclusion holds for any fixed nonnegative levels with $\sum_j \alpha_j \le \alpha$.

Intuition

Each candidate spends a slice of one fixed error budget, and the slices $\alpha/2, \alpha/6, \alpha/12, \ldots$ add up to $\alpha$ however many candidates arrive. Adaptivity is allowed because each slice is a guarantee conditional on whatever history produced that candidate.

Proof Sketch

Let $W_j$ be the event that candidate $j$ is unacceptable and approved. The event $\{p_j \ge \epsilon_j\}$ is determined by $\mathcal G_{j-1}$, so

$$\Pr(W_j) = \mathbb E\big[\mathbf 1\{p_j \ge \epsilon_j\} \Pr(\text{approve } j \mid \mathcal G_{j-1})\big] \le \alpha_j.$$

The union bound gives $\Pr(\bigcup_j W_j) \le \sum_j \alpha_j$, and since $1/(j(j+1)) = 1/j - 1/(j+1)$ the sum telescopes to $\alpha$. No independence between the candidates' approval errors is used.

Why It Matters

With $\alpha = 0.05$ and a target rate $\epsilon = 0.001$, a zero-failure test for candidate 1 runs at level 0.025 and needs 3,688 failure-free trials. Candidate 10 runs at about 0.00045 and needs 7,693, and candidate 100 needs 12,210. A single fixed candidate at level 0.05 needs 2,995. The requirement grows by roughly $2\log j/\epsilon$ trials, which is the price of letting a research process propose candidates indefinitely without inflating the chance of a false approval.

Failure Mode

Validity has to hold conditionally on the history that produced each candidate. If candidate $j+1$ is tuned on candidate $j$'s revealed confirmation data, those data are not fresh for $j+1$, and its test is not valid given its history. Adaptive reuse of data changes the inference problem; Dwork and colleagues study it and give mechanisms that keep estimates valid under further assumptions. [4] The proposition controls false approval of the stated rate claims only. It says nothing about harms outside the defined failure event, about populations other than the confirmation population, or about how much exposure an approved candidate receives; a later section takes up exposure.

report a correction →

Watch Out

Error spending does not require independent candidates

Successive checkpoints from one training run are strongly dependent, and that is fine here: the union bound needs no independence. What the proposition does need is that each candidate's test is valid given the history that produced the candidate. Dependence is harmless; reuse of confirmation data that shaped the candidate is not.

The synthetic data flywheels page applies the same allowance to a different claim, a candidate's mean score against a comparator. The error-spending step is identical; what changes is the null hypothesis each candidate's test is valid for.

An Anytime-Valid Confirmation Test

A fixed-sample test commits to $n$ in advance. A betting test lets the team look after every trial and stop at the first strong evidence. Drop the index $j$ and fix one candidate with threshold $\epsilon$ in $(0,1)$ and allotted level $\beta$. Let $X_1, X_2, \ldots$ be its failure indicators, and let $\mathcal F_t$ be the history after $t$ trials.

Proposition

A Betting Test for a Failure Rate Is Valid at Any Stopping Time

Statement

Set $M_0 = 1$ and

$$M_t = M_{t-1}\big(1 + \lambda_t(\epsilon - X_t)\big), \qquad 0 \le \lambda_t \le 1/(1-\epsilon),$$

with $\lambda_t$ determined by $\mathcal F_{t-1}$. Under the null hypothesis $p \ge \epsilon$, the process $(M_t)$ is a nonnegative supermartingale, and

$$\Pr\Big(\sup_{t \ge 0} M_t \ge 1/\beta\Big) \le \beta.$$

Approving the candidate the first time $M_t \ge 1/\beta$ therefore has false-approval probability at most $\beta$, however often the team looks and whenever it stops.

Intuition

$M_t$ is the wealth of a gambler betting that failures are rarer than $\epsilon$. Each unit of stake wins $\epsilon$ on a success and loses $1-\epsilon$ on a failure. If the true rate is at least $\epsilon$, every bet has nonpositive expected return, so wealth is not expected to grow, and multiplying the starting wealth by $1/\beta$ is correspondingly rare.

Proof Sketch

The stake bound keeps the failure factor $1 - \lambda_t(1-\epsilon)$ nonnegative, so $M_t \ge 0$. Because $\lambda_t$ is fixed before $X_t$ is seen,

$$\mathbb E[M_t \mid \mathcal F_{t-1}] = M_{t-1}\big(1 + \lambda_t(\epsilon - p)\big) \le M_{t-1}$$

under the null. Ville's inequality for a nonnegative supermartingale with $M_0 = 1$ gives $\Pr(\sup_t M_t \ge 1/\beta) \le \beta$. Howard and colleagues state this inequality as Lemma 1 and build their time-uniform crossing bounds on it. [5]

Why It Matters

At the largest stake $\lambda = 1/(1-\epsilon)$, a success multiplies wealth by $1/(1-\epsilon)$ and a failure sends it to zero. A failure-free run then crosses $1/\beta$ exactly when $(1-\epsilon)^t \le \beta$, the same sample size as the fixed zero-failure bound: 2,995 trials for $\epsilon = 0.001$ and $\beta = 0.05$. At half that stake a failure halves wealth instead of ending the test. A clean run then needs 5,987 trials, each failure costs about 1,385 extra successes, and one failure at trial 10 moves approval to trial 7,374. The stake trades speed on a clean run against tolerance of occasional failures.

Failure Mode

Validity needs one fixed candidate and a constant failure probability given the past. Swapping in a new checkpoint partway through changes the parameter being tested, and the supermartingale property no longer refers to any one system. The same algebra works under a weaker null in which every conditional failure probability is at least $\epsilon$, but then approval rejects only that weaker statement; it does not show that every future conditional rate is below $\epsilon$. The test also inherits the sampling assumptions of the confirmation population. Optional stopping protects against peeking, not against distribution shift.

report a correction →

Running a betting test with $\beta = \alpha_j$ for each candidate $j$, on fresh trials, meets the conditional hypothesis of the error-spending proposition. The two results then combine into a lifetime false-approval guarantee for a process that tests candidates one after another, each monitored continuously.

Watch Out

Anytime-valid does not mean checkpoint-agnostic

Optional-stopping validity lets the team look after every trial and stop whenever the evidence is strong. It does not let the system under test change during the run. A stream of outcomes from a sequence of checkpoints is not a sample from the checkpoint finally approved, and no stopping rule repairs that. Give each new candidate its own test with its own slice of the error budget.

False-Approval Error Is Not Operational Risk

A correctly approved candidate with a small failure rate is still not safe under unlimited use. Suppose deployment runs in stages, stage $j$ uses one approved candidate for at most $N_j$ trials, and each cap is fixed in advance and enforced outside the candidate.

Proposition

Lifetime Violation Risk Splits Into Approval Error Plus Exposure

Statement

Let $V$ be the event of at least one target violation during deployment. Under the assumptions, with each stage threshold $\epsilon_j$ a constant declared before deployment (thresholds chosen from the research history must be replaced by an upper bound fixed in advance),

$$\Pr(V) \le \min\Big(1, \alpha + \sum_j N_j \epsilon_j\Big).$$

If every deployed candidate is known to satisfy the conditional bound, the approval term drops and $\Pr(V) \le \min(1, \sum_j N_j \epsilon_j)$.

Intuition

There are two routes to a violation. A bad candidate slipped through approval, which has probability at most $\alpha$, or the approvals were right and a good candidate failed during the trials it was allowed to run. The second term is an exposure budget, not a statistical error.

Proof Sketch

Let $W$ be the event that some unacceptable candidate is approved, and let $G_j$ be the event that stage $j$ runs a candidate with $p_j \lt \epsilon_j$; $G_j$ is determined by the history before the stage. Only approved candidates are deployed, so outside $W$ every running stage satisfies $G_j$, and $V \setminus W$ is contained in the union over trials $i$ of $V_i \cap G_{j(i)}$, where $V_i$ is a violation at trial $i$ and $j(i)$ is its stage. For each trial,

$$\Pr(V_i \cap G_{j(i)}) = \mathbb E\big[\mathbf 1_{G_{j(i)}} \Pr(V_i \mid \mathcal F_{i-1})\big] \le \epsilon_{j(i)}$$

by the transfer hypothesis. Summing over at most $N_j$ trials per stage gives $\Pr(V \setminus W) \le \sum_j N_j \epsilon_j$, and $\Pr(V) \le \Pr(W) + \Pr(V \setminus W)$. A probability is also at most 1.

Why It Matters

A single candidate correctly approved at $\epsilon = 0.001$ and then run for 10,000 trials has an exposure term of $10{,}000 \times 0.001 = 10$, so the bound is vacuous: approval at a rate threshold is not a license for unlimited use. Budgeting exposure fixes this. Take $\alpha = 0.05$ and three stages with caps of 1,000, 10,000, and 100,000 trials at thresholds $10^{-4}$, $10^{-5}$, and $10^{-6}$. Each stage contributes 0.1 of exposure, and the bound is $0.05 + 0.3 = 0.35$. Exposure dominates, and tightening $\alpha$ would barely move the total. The calculation does not choose an acceptable risk; it shows what follows from a declared threshold and exposure policy.

Failure Mode

The transfer hypothesis carries the argument. It is a conditional bound after every history, which is stronger than a low historical average, and it has to survive changes in workload, interventions, and anything an adaptive system does with what it has observed, as in the stepwise bound on the AI control page. The confirmation population and the deployment population need a justified link, such as a total variation bound, and that link changes the thresholds that apply. If the caps are not enforced by a mechanism outside the candidate, the sum has no fixed value.

report a correction →

Watch Out

A small false-approval probability is not a small probability of harm

The error-spending bound says that wrong approvals are rare. It says nothing about how often a rightly approved candidate causes a violation, which is governed by its rate and its exposure. A report that gives only the false-approval level has left out the term that usually dominates.

Measure Safety Transfer Instead of Assuming It

This section describes a proposed experiment, not a reported result.

Compare the predecessor and the successor on capability, target violations, caught failures at a fixed monitoring budget, and response to authorized intervention. Keep the workload and permissions comparable before studying an expanded deployment separately.

For monitoring, compare the inherited monitor, a newly trained monitor, and an independently developed one. Detector-transfer research gives a concrete reason: Hopkins and colleagues report lie detectors that did well in distribution but transferred poorly to held-out lie categories in their settings. [6]

For alignment, vary incentives and oversight information in controlled tasks, and include ordinary mistakes and scoring errors as competing explanations. For control, test the inherited protocol against newly developed attack families instead of rerunning only the attacks used to design it.

Record capability gains and safety regressions separately. Choosing a successor by a weighted average of the two can hide a large regression in a property meant as a hard constraint, which is the weighted-mean blind spot from what alignment means.

A Proposed Evidence-Invalidation Ledger

These rules are proposals for this route, not results from a controlled deployment study. An immutable version record and a reproducible evaluation protocol make the argument inspectable; they do not make its assumptions true.

A Research Agenda for 2027 and Later

These are conditional research targets. They assert no timeline and no probability of runaway improvement.

Safety-preserving research transfer. An inherited improvement procedure keeps producing useful successors while keeping defined constraints and independently assessed monitoring quality across new task families.

Evaluator improvement without evaluator capture. A system develops better tests, and adoption depends on evidence that the new tests catch previously missed failures instead of relabeling them. Proposed experiment: hold back a sealed set of cases with independently verified failures that the system never sees, and adopt a proposed evaluator only if it catches sealed failures the old evaluator missed while keeping its verdicts on preserved, already-adjudicated cases.

Compositional oversight. Local guarantees stay useful across delegation, shared memory, repeated updates, and multiple agents. The invariant and exposure propositions above are small examples of composition, and their hypotheses show how much composition demands.

Check your understanding

Does a more capable successor inherit its predecessor's safety case?

Not automatically. The components that changed determine which claims need new evidence.

What does the error-spending proposition control?

The probability that any unacceptable candidate is ever approved, given tests that are valid conditional on each candidate's history. It does not bound harm from correctly approved candidates.

Does optional-stopping validity allow checkpoint changes inside one test?

No. The candidate and the sampling assumptions have to stay fixed for the whole run.

Can proof-preserving updates establish general alignment?

Only if the proved invariant expresses that claim in a faithful model. A narrow interface invariant does not.

ExerciseCore

Problem

A lab approves candidates by error spending with $\alpha = 0.05$, using fixed-sample zero-failure tests at $\epsilon = 0.001$. Find the level and the number of failure-free trials for candidates 1 and 2, and the share of the budget spent after 10 candidates. Then explain why giving every candidate the full level 0.05 fails, using 20 unacceptable candidates whose test errors are independent.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Run the betting test with $\epsilon = 0.001$, $\beta = 0.05$, and the constant stake $\lambda = 1/(2(1-\epsilon))$. Show that a failure halves wealth and find the success factor. Derive the number of trials to approval with zero, one, and two failures. Then take an acceptable candidate with true rate $p = 0.0002$ and compute the probability that at least one failure occurs in the first 2,995 trials. What does that say about using the largest stake?

Hint

Reveal Solution

ExerciseAdvanced

Problem

Candidate $a$ has failure rate $p_a = 0.02$ against a threshold $\epsilon = 0.01$, so it is unacceptable. The team runs the betting test with the constant stake $\lambda = 0.5$ and $\beta = 0.025$. After 200 trials, without resetting the wealth, it swaps in candidate $b$, which never fails on this workload, and keeps betting. What is the probability that the test eventually approves? Which assumption failed, and what is the valid procedure?

Hint

Reveal Solution

ExerciseResearch

Problem

A system state is a pair of integers $(x, y)$ with initial state $(0, 1)$, and the only authorized transition is $F(x, y) = (x + y, y)$. Let $I$ be the property $x \ge 0$. Show that $I$ holds on every reachable state, show that the invariant proposition cannot be applied to $I$ directly, and find a stronger property that it can be applied to. Then say what this example implies for an approval gateway whose invariant has only been checked on the configurations observed in testing.

Hint

Reveal Solution

Takeaway

Every successor needs an account of what improved, what stayed constrained, and which evidence still applies. Approval error, exposure, and the scope of each proof are separate budgets, and a gain in research capability pays for none of them.

References

Registry ids in monospace resolve in data/content/sources.json. Each source was checked by URL on 2026-09-15. The four propositions are derived on this page; the sources support the threat models, findings, and frameworks attributed to them.

• [1] Hubinger, van Merwijk, Mikulik, Skalse, and Garrabrant. "Risks from Learned Optimization in Advanced Machine Learning Systems." 2019. Source. A theoretical threat model for mesa-optimization and deceptive alignment, not a prevalence estimate. hubinger-2019-learned-optimization
• [2] Greenblatt et al. "Alignment faking in large language models." 2024. Source. Selective compliance under constructed training-information conditions, in the tested setting. greenblatt-2024-alignment-faking
• [3] Schmidhuber. "Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements." 2003 preprint, revised 2006. Source. Proof-triggered self-rewrites relative to formal axioms and a utility function. schmidhuber-2003-godel-machines
• [4] Dwork, Feldman, Hardt, Pitassi, Reingold, and Roth. "Preserving Statistical Validity in Adaptive Data Analysis." 2014. Source. Studies how statistical validity is lost when analyses adaptively reuse the same data, and gives mechanisms that restore it under further assumptions. dwork-2015-adaptive-data-analysis
• [5] Howard, Ramdas, McAuliffe, and Sekhon. "Time-uniform Chernoff bounds via nonnegative supermartingales." 2018 preprint; Probability Surveys, 2020. Source. Ville's inequality as Lemma 1 and the nonnegative-supermartingale framework behind time-uniform crossing guarantees. howard-2020-time-uniform-chernoff
• [6] Hopkins, Khullar, Wang, and Roger. "Fine-Tuned Lie Detectors Failed to Generalize." August 21, 2026. Source. Strong in-distribution and weak cross-category detection of elicited lies. hopkins-2026-lie-detectors