AI Control: Safety Without Assuming Alignment

Why This Matters

Can a useful system stay controlled even when a capable component is trying to defeat its safeguards?

That is the threat model AI control studies. Instead of assuming a capable model has the intended objectives, you evaluate the surrounding protocol against deliberate attempts to produce an unacceptable outcome. The original control experiments use programming tasks with a powerful untrusted model, a weaker trusted model, and a limited budget of high-quality oversight; the outcome they guard against is submitted code containing a hidden backdoor. [1]

"Trusted" is an assumption relative to a threat model. It does not mean infallible, perfectly calibrated, or equally capable. A trusted model may make ordinary mistakes while being assumed not to coordinate with the attacker on purpose.

Evidence key. The audit-budget model on this page is an original teaching model, and its three propositions are derived here under stated assumptions. It is not a fit to any deployed system. Protocol descriptions are attributed to their sources. The research protocol near the end is a proposal.

Before you start. Read what alignment means for the distinction between alignment and control, and be comfortable with conditional probability. Classification thresholds and LLM application security are helpful background. No reinforcement-learning derivation is needed to use the lab.

Draw the System Boundary First

Consider a hypothetical code-maintenance assistant. An untrusted component proposes changes. A monitor assigns each change a suspicion score. Some changes are audited, some are redirected to a trusted implementation path, and some are accepted. An external gateway controls which operations can actually run.

A control evaluation must specify what the attacker can change and observe. Can it edit only the proposed patch, or also the monitor, the tests, the logs, the tool configuration, and the approval state? A protocol that works in the first setting may fail in the second.

For this page, the trusted boundary contains the evaluator, the audit randomization, the permission gateway, and the record of approvals. Text generated by the candidate cannot change those components. That is a design assumption to test in an implementation, not a property conferred by writing the word "trusted" in a prompt.

The target violation is one specific prohibited side effect. Other bugs remain possible. Usefulness is defined separately, on ordinary authorized tasks, so that refusing all work is not mistaken for solving the problem.

Definition

Control evaluation (working definition)

A control evaluation fixes a protocol: the full procedure around the untrusted model, including monitors, audits, routing rules, permissions, and responses. It also fixes a trusted boundary of components the attacker cannot modify, a target violation, a class of attack policies, and a separate honest workload.

It reports two things for that boundary and attack class only: the risk of the target violation when the untrusted model follows an attack policy, and task usefulness when it behaves honestly.

Compare Protocols, Not Isolated Classifiers

The original work examines trusted monitoring, deferral to the trusted model, trusted editing, and untrusted monitoring with measures meant to limit collusion. The object being evaluated is the entire decision procedure, including what happens after an alert. [1]

ControlArena, a library from the UK AI Security Institute and Redwood Research, provides evaluation settings with main and side tasks, honest and attack evaluations, monitors, protocol components, and safety and usefulness analysis. [2] It is a research tool, not a certificate that an application built with it is controlled.

In the safety-case literature, a control argument is one kind of argument alongside arguments from inability, trustworthiness, and deference to credible AI advisors. [3] A control evaluation supplies evidence for that kind of argument and not the others.

Watch Out

A strong monitor does not decide what happens next

A monitor with high recall and no remaining audit capacity creates a policy question: should suspicious work be released, deferred, or stopped? The answer changes the outcome even when the monitor is unchanged. The next section puts a number on that difference.

An Explicit Audit-Budget Model

The following is an original teaching model. Every rate is supplied by the reader. The model describes stationary routing of a stream of proposals; it does not simulate an adaptive attacker or a real review queue.

Let $p$ be the fraction of proposals that contain the target violation, $s$ the probability the monitor flags such a proposal, and $f$ its false-positive rate on harmless proposals. Let $b$ be review capacity as a fraction of all proposals, and $h$ the probability an audit detects the violation in a reviewed, flagged proposal.

The flagged fraction, the reviewed fraction, and the probability that a flagged proposal is reviewed are

$$z = ps + (1-p)f, \qquad a = \min(b, z), \qquad q = \begin{cases} a/z, & z > 0, \\ 0, & z = 0. \end{cases}$$

Audited proposals are a uniform random selection from flagged proposals, so in expectation their violating fraction matches the flagged pool. If review instead prioritizes the highest suspicion scores, use the conditional rates for that selection rule; do not reuse these formulas silently.

Proposition

Release and Deferral Give Different Violation Rates for the Same Monitor

Statement

Define $z$, $a$, and $q$ as above. If unreviewed flagged proposals are released, the per-proposal probability of the target violation is $r_{\mathrm{release}} = p(1 - sqh)$. If they are deferred to the trusted fallback, it is $r_{\mathrm{defer}} = p(1-s) + psq(1-h)$. The difference is $r_{\mathrm{release}} - r_{\mathrm{defer}} = ps(1-q) \geq 0$.

Intuition

The two policies route every proposal identically except the flagged ones that review never reaches. That path carries probability $ps(1-q)$ of a violation. Release lets it through; deferral blocks it.

Proof Sketch

Partition violating proposals into three disjoint paths. Unflagged: probability $p(1-s)$, always accepted. Flagged and reviewed: probability $psq$, since uniform selection reviews each flagged proposal with probability $q$; these pass only when the audit misses, with probability $1-h$. Flagged and unreviewed: probability $ps(1-q)$; these pass under release and are blocked under deferral.

Adding the paths that pass gives $r_{\mathrm{defer}} = p(1-s) + psq(1-h)$ and

$$r_{\mathrm{release}} = p(1-s) + psq(1-h) + ps(1-q) = p - psqh = p(1 - sqh).$$

Subtracting gives $ps(1-q)$, which is nonnegative because every factor lies in $[0,1]$.

Why It Matters

A backlog can silently switch a protocol from deferral to release, and $ps(1-q)$ is the price of that switch. With the lab's starting values below it is about 59 violations per 10,000 proposals, more than five times the entire deferral rate.

Failure Mode

This is an accounting identity for expected rates in a stationary stream. It does not describe an attacker who times attempts for moments when the queue is full, a review rule that picks the most suspicious proposals first, review delays, or a fallback that fails on the target violation. Under an adaptive attacker, $p$ and $s$ are not fixed constants, and the flag rate $z$ that sets $q$ is partly under the attacker's influence. The identity is not a theorem that any real fallback is safe.

report a correction →

For deferral, the trusted fallback is assumed to prevent this particular target violation. It need not solve every task and is not assumed free of unrelated errors.

The Lab: What Happens When the Review Budget Runs Out?

The lab starts with 1% violating proposals, 90% monitor recall, a 5% false-positive rate, a 2% review budget, and 99% audit sensitivity. The flagged fraction is 5.85%, so review covers about 34.19% of flagged proposals.

Under those inputs, releasing unreviewed alerts gives about 69.54 target violations per 10,000 proposals, and deferring them gives about 10.31. The breakdown line shows where the deferral figure comes from: 10.00 unflagged, 0.31 missed in review, and 0.00 released. These are exact expectations from the model, rounded for display. They are not measured attack rates.

Use the Monitor shift preset to drop recall to 30% without changing the review budget. Under deferral the rate rises to 70.11, of which 70.00 comes from the unflagged path. Extra review of already-flagged proposals cannot recover violations that never enter the queue.

Use the No signal preset to set recall equal to the false-positive rate, 5%. The monitor then provides no discrimination between violating and harmless proposals. Random review and deferral still stop a few violations (95.02 per 10,000 under deferral, 98.02 under release), but the score contributes no sorting advantage.

Usefulness is evaluated in a different mode

The lab separately models an honest run, where no proposal contains the target violation. Let $u$ and $v$ be the task-success probabilities of the original path and the fallback path.

Proposition

Honest-Mode Task Success Under Deferral

Statement

Under the deferral policy, the fraction of honest proposals sent to the fallback is $d = \max(0, f - b)$, and expected task success is $U_{\mathrm{honest}} = (1-d)\,u + d\,v$. Under the release policy nothing is deferred and $U_{\mathrm{honest}} = u$.

Intuition

In honest mode every flag is a false positive. Review absorbs up to $b$ of them, and only the overflow goes to the weaker path, so usefulness is a mixture of the two success rates weighted by how much overflows.

Proof Sketch

With no violating proposals, the flagged fraction is $f$. Review covers $\min(b, f)$, so the deferred fraction is $f - \min(b, f) = \max(0, f - b) = d$.

A proposal that is not deferred keeps its original answer, because audits of harmless proposals do not change answers; by independence it is correct with probability $u$. A deferred proposal is correct with probability $v$. The law of total probability gives $(1-d)u + dv$. Under release, $d = 0$.

Why It Matters

With $u = 0.9$, $v = 0.65$, and the lab's starting routing values, $d = 0.03$ and task success is 89.25%, against 90% when alerts are released. Deferral buys a drop from 69.54 to 10.31 violations per 10,000 at a cost of 0.75 percentage points of honest task success. Report both numbers side by side; the lab deliberately does not combine them into one unexplained safety score.

Failure Mode

If the monitor flags hard tasks more often than easy ones, the independence assumption fails: both paths do worse than average on the deferred slice, and the formula is off by an amount that depends on that correlation. Real review can also add delays, false rejections, and correlated errors. None of those mechanisms is in the model, and each needs its own measurement.

report a correction →

An Adaptive Adversary Changes the Evaluation Problem

The rate $p$ is not a known natural constant, and monitor recall may change once an adversary adapts to the protocol. Use the lab to reason about bottlenecks, not to estimate the intentions of a model.

Definition

Worst-case control objective (a proposed formulation)${\min }_{\pi \in \Pi }{\sup }_{\alpha \in \mathcal{A}}R(\pi ,\alpha )$

Choose a protocol $\pi$ from a set $\Pi$ to minimize the worst-case violation risk $R(\pi, \alpha)$ over a declared class $\mathcal{A}$ of attack policies $\alpha$, subject to a usefulness floor $U(\pi) \geq u_0$ and a resource budget $C(\pi) \leq B$. The formulation makes the attack class explicit; it is a way to state the goal, not a result.

Finite red-team testing estimates the maximum risk over the attacks actually tried. For any tested set $T \subseteq \mathcal{A}$, the true maximum over $T$ is at most the supremum over $\mathcal{A}$, so the tested maximum is a lower bound on worst-case risk, not a conservative upper bound. Sampling error in each tested risk adds uncertainty on top of that.

Watch Out

The strongest attack you found is not the strongest attack

A red team that tried many strategies and found none above 2% has shown that those strategies stay below 2% (up to sampling error). An untested strategy can do better. Treat the tested maximum as a floor on worst-case risk, and widen the attack class before narrowing the claim.

Deployments also run many steps, and an attacker's choices at one step can depend on what happened before. The following bound handles that dependence.

Proposition

A Lifetime Violation Bound That Does Not Need Independent Steps

Statement

Let $H_i$ be the event of a target violation at step $i$, and let $\mathcal{F}_{i-1}$ be the history before step $i$, with $H_1, \dots, H_{i-1}$ determined by $\mathcal{F}_{i-1}$. If $\Pr(H_i \mid \mathcal{F}_{i-1}) \leq r_i$ almost surely for each $i = 1, \dots, N$, then

$$\Pr\left(\bigcup_{i=1}^{N} H_i\right) \leq 1 - \prod_{i=1}^{N} (1 - r_i) \leq \sum_{i=1}^{N} r_i.$$

Intuition

Dependence is allowed. The attacker may adapt to everything it has seen, so long as the protocol caps its chance of success at each step whatever happened before. Under that cap, risk accumulates no faster than it would for independent steps at the capped rates.

Proof Sketch

Let $E_k$ be the event of no violation in the first $k$ steps, with $E_0$ certain. Because $E_{k-1}$ is determined by the history before step $k$,

$$\Pr(E_k) = \mathbb E\left[\mathbf{1}_{E_{k-1}} \Pr(H_k^c \mid \mathcal F_{k-1})\right] \geq (1 - r_k)\Pr(E_{k-1}).$$

Induction gives $\Pr(E_N) \geq \prod_{i} (1 - r_i)$, and taking complements gives the first inequality. The second, $1 - \prod_i (1 - r_i) \leq \sum_i r_i$ for $r_i \in [0,1]$, follows by induction on $N$. The outer bound also follows directly from the union bound, because $\Pr(H_i) = \mathbb{E}[\Pr(H_i \mid \mathcal{F}_{i-1})] \leq r_i$; that route does not need the history to record earlier violations.

Why It Matters

With $N = 10{,}000$ steps and a per-step bound of $10^{-5}$, the sum gives $0.1$ and the product form gives about $0.0952$. So the familiar expression $1 - (1-r)^N$ is a valid upper bound here without any independence assumption. Read as an exact risk rather than a bound, it does need independent steps that each fail with probability exactly $r$.

Failure Mode

The difficult part is the hypothesis. A historical average recall, or a failure rate measured on past attacks, does not establish that the conditional bound holds after every history, and a strategic attacker looks for exactly the histories where the protocol is weakest, such as a full audit queue. If the per-step bound fails on histories of positive probability, the conclusion can fail. The bounds must also be fixed before the steps run, not chosen after seeing outcomes.

report a correction →

A Research Protocol Worth Running

What follows is a proposed experiment design. The local lab does not run these agent experiments.

Freeze a candidate control protocol after development. Evaluate ordinary task usefulness and adversarial target violations separately. Hold out attack families, not merely random seeds. Count the resources available to the attacker, the monitor, the fallback, and the reviewers.

Include ablations for no monitoring, random auditing, monitor-guided auditing, and deferral, with the same authorized task distribution and comparable budgets. A protocol that improves only against the attacks used to design it has not established general protection. Reusing one evaluation set to both choose and assess a procedure breaks the usual statistical guarantees, which is why the held-out attack families matter. [4] Statistical safety cases covers how to turn the resulting failure counts into claims, and red-teaming and adversarial evaluation covers attack discovery.

Record each alert, routing decision, audit result, final artifact, and irreversible action. Evaluate the final artifact rather than assuming a successful intermediate intervention resolved the whole trajectory.

Failure Modes Worth Making Visible

A monitor and a generator may share blind spots. Auditors may see only the information the candidate chooses to present. A backlog may quietly change the intended policy from deferral to release. A permission change may open an untested path around a previously effective safeguard.

These examples lead to three separate questions: did the monitor notice, did the protocol act, and did the environment enforce the action? Aggregate classifier accuracy answers none of them on its own. When the observation channel is the model's own reasoning text, the question of what a monitor can see gets harder; reasoning monitorability takes it up.

Check your understanding

Why does higher audit sensitivity fail to remove all modeled risk?

Because unflagged violating proposals never reach review. The $p(1-s)$ term does not depend on $h$.

Does "trusted fallback" mean the fallback solves every problem?

No. The model assumes it prevents the specified target violation while allowing lower ordinary task success.

Is the strongest tested attack a worst-case upper bound?

Not in general. Untested attacks may do better, so the tested maximum is a lower bound on worst-case risk.

Why must the audit queue's overflow policy be specified?

Because releasing and deferring the same unreviewed alerts produce different violation probabilities, differing by $ps(1-q)$ in the model.

ExerciseCore

Problem

Using the lab's starting values ($p = 0.01$, $s = 0.9$, $f = 0.05$, $b = 0.02$, $h = 0.99$, $u = 0.9$, $v = 0.65$), compute $z$, $q$, both violation rates per 10,000 proposals, and honest task success under each policy.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Keep every starting value but raise the review budget from 2% to 6% of proposals, under the deferral policy. Compute the new violation rate per 10,000 and the new honest task success. The violation rate goes up. Explain why, using the derivative of $r_{\mathrm{defer}}$ with respect to $q$, and say what the result does and does not imply about buying more review.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Under the Monitor shift preset ($s = 0.3$, other values at their starting settings), find the smallest and largest values $r_{\mathrm{defer}}$ can take as the review budget ranges over all values. What does the answer say about where to spend effort?

Hint

Reveal Solution

ExerciseResearch

Problem

A deployment will run $N = 10{,}000$ steps. Past evaluations show the monitor catches 90% of attacks on average, and the team argues that a per-step violation bound of $10^{-5}$ holds. First, compute the lifetime bounds from the stepwise proposition. Second, describe an attack policy under which the average recall stays near 90% while the conditional bound fails, and say what evidence would be needed to support the hypothesis instead.

Hint

Reveal Solution

Takeaway

Control belongs to the full protocol: detection, review, routing, permissions, and response. The same monitor gives different risk under release and deferral, review cannot reach what the monitor never flags, and a lifetime bound needs a per-step guarantee that holds after every history. Evaluate the protocol against a stated adversary, report usefulness next to risk, and state what remains outside the test.

References

Registry ids in monospace resolve in data/content/sources.json. Each record was checked by URL on 2026-09-15. The audit-budget model and the three propositions are derived on this page; the sources support the protocol descriptions, framework, and statistical point attributed to them.

• [1] Greenblatt, Shlegeris, Sachan, and Roger. "AI Control: Improving Safety Despite Intentional Subversion." 2023. Source. Control protocols with trusted and untrusted models, monitoring, editing, and limited high-quality oversight on programming tasks. greenblatt-2023-ai-control
• [2] UK AI Security Institute and Redwood Research. "ControlArena." Living documentation; accessed 2026-09-15. Source. A library for control experiments with settings, honest and attack evaluations, and protocol components; not a certificate for any application. control-arena-documentation
• [3] Clymer, Gabrieli, Krueger, and Larsen. "Safety Cases: How to Justify the Safety of Advanced AI Systems." 2024. Source. A framework that places control arguments among other categories of safety argument. clymer-2024-safety-cases
• [4] Dwork, Feldman, Hardt, Pitassi, Reingold, and Roth. "Preserving Statistical Validity in Adaptive Data Analysis." 2014. Source. Loss of standard statistical validity when analyses adaptively reuse the same data. dwork-2015-adaptive-data-analysis