Reasoning Monitorability: What an Overseer Can Observe

Why This Matters

A model can get better at a task while an overseer sees less of how it got there. Those are separate axes, and progress on the first says nothing about the second.

Monitoring a model's written reasoning has been proposed as an imperfect but useful source of oversight evidence. Korbak and colleagues argue that chain-of-thought monitoring is imperfect, since some misbehavior goes unnoticed, but promising. They add that the opportunity may be fragile, and they recommend that developers consider how their development decisions affect it. [1] Whether readable reasoning persists is a research question, not a property reasoning models are guaranteed to keep.

This page is the safety companion to the loop inside the model, which covers architectures that spend extra computation inside the network before emitting a token. Recurrent depth asks how much computation happens. Monitorability asks whether the available observations let someone detect a specified failure early enough to act.

Evidence key. Two propositions are derived here from stated assumptions. Empirical findings are attributed to their authors and scoped to the settings they studied. The regression suite and the research targets near the end are proposals.

Before you start. Read what alignment means first. Conditional probability is all the probability the derivations use, and AI control is helpful, since its audit-budget model reappears below. Latent reasoning and mechanistic interpretability are useful side branches, not requirements.

Readable, Faithful, and Useful Are Different Properties

Definition

Three properties of an observation channel (working definitions)

A trace is readable when a person or a monitor can parse what it says. It is faithful, in a stated sense, when it reflects the features that actually caused the answer. It is useful for monitoring a specified failure when a decision rule that reads it detects that failure at an acceptable cost and early enough to intervene.

These are working definitions for this page; researchers use related terms in different ways. The first property is about language, the second is about causation, and the third is about a decision problem. None of them implies another.

A readable justification can rationalize a decision made for a different reason. Turpin and colleagues added biasing features to prompts, for example reordering the multiple-choice options in a few-shot prompt so the answer was always "(A)", and found that models systematically failed to mention those features in their explanations. [2] When the bias pushed toward a wrong answer, models often wrote explanations rationalizing it, and accuracy dropped by as much as 36% on 13 BIG-Bench Hard tasks with GPT-3.5 and Claude 1.0. The study concerns prompted explanations from 2023 models. It shows that readability does not establish faithfulness; it does not show that every later trace is unfaithful.

The two directions do not collapse into one. A partially unfaithful trace can still carry a warning signal: a trace that rationalizes a sabotage decision may still name the file being edited. A faithful account of a narrow computation can omit the system-level consequence a monitor needs, such as the effect of a correct-looking patch on a permission check elsewhere. The target is not a general "transparent" label. It is a measured detection rate for a named failure.

Watch Out

A readable trace is not a faithful trace

Fluent step-by-step text shows what the model wrote, not why it answered. An explanation can be accurate about every step it states and still leave out the input feature that decided the answer. Faithfulness is a causal claim and needs a causal test, such as changing the suspected feature and checking whether the explanation reports the change.

Timing belongs in the evaluation. A warning emitted after an irreversible action is a forensic finding, not a control. An evaluation should record when the monitor could first have flagged the trajectory and whether an intervention was still available at that point, not only whether a classifier can label the finished transcript.

An Information Result and What It Leaves Out

Fix a joint distribution of an action record $A$, a trace $T$, and a failure label $Y$ taking the values 0 and 1. A monitor chooses $d = 1$ (flag) or $d = 0$ (pass). A missed failure costs $c_1 > 0$ and a false alarm costs $c_0 > 0$, so the loss is

$$\ell(d, y) = c_1 \mathbf 1\{d = 0, y = 1\} + c_0 \mathbf 1\{d = 1, y = 0\}.$$

Write $\eta_A = \Pr(Y = 1 \mid A)$ and $\eta_{A,T} = \Pr(Y = 1 \mid A, T)$ for the conditional failure probabilities, and $\varphi(u) = \min(c_1 u, c_0(1-u))$ for $u$ in $[0,1]$.

Proposition

Extra Observations Cannot Raise the Optimal Monitoring Risk

Statement

Let $R_A^*$ be the smallest expected loss over rules that read only $A$, and $R_{A,T}^*$ the smallest over rules that read $(A, T)$. Then

$$R_A^* = \mathbb E[\varphi(\eta_A)], \qquad R_{A,T}^* = \mathbb E[\varphi(\eta_{A,T})], \qquad R_{A,T}^* \le R_A^*.$$

Let $u^* = c_0/(c_0 + c_1)$. If, given $A$, the probability $\eta_{A,T}$ almost surely stays at or below $u^*$, or almost surely stays at or above $u^*$, then $R_{A,T}^* = R_A^*$: the trace changes no optimal decision.

Intuition

Reading more cannot hurt an ideal decision-maker, because it can always ignore the extra input. The extra input helps only when it moves the conditional failure probability across the point where flagging becomes worth its false-alarm cost.

Proof Sketch

Given an observation $O$ with conditional failure probability $\eta_O$, flagging has conditional expected loss $c_0(1-\eta_O)$ and passing has $c_1 \eta_O$. Every rule has conditional loss at least the smaller of the two, $\varphi(\eta_O)$, and the measurable rule that flags exactly when $\eta_O > u^*$ attains it. Taking expectations gives both formulas.

By the tower property, $\eta_A = \mathbb E[\eta_{A,T} \mid A]$. The function $\varphi$ is the minimum of two linear functions, so it is concave, and conditional Jensen gives $\mathbb E[\varphi(\eta_{A,T}) \mid A] \le \varphi(\eta_A)$. Taking expectations gives the inequality. On each side of $u^*$ the function $\varphi$ is linear, so Jensen holds with equality when $\eta_{A,T}$ stays on one side, which is the equality case.

Why It Matters

The inequality is the formal version of "more information cannot hurt", and the equality case is the part that gets forgotten. A trace can be informative, in the sense that it changes the failure probability, and still be worth nothing to a monitor with these costs, because every change stays on one side of the flagging threshold. The worked example below builds one.

Failure Mode

The result concerns an ideal rule on a fixed distribution. A monitor learned from finite data can do worse with more inputs, for example by fitting spurious cues in long traces. The statement has no budget, while reading long traces costs review time that could cover more actions. Most important, the distribution is held fixed. If traces are exposed to training pressure, or an architecture moves computation out of text, the joint law of $(A, T, Y)$ changes and the inequality says nothing about the new system. It cannot be used to argue that an architecture producing more text is safer.

report a correction →

Example

An informative trace that changes no decision

Take failure prevalence $p = 0.01$, a missed-failure cost $c_1 = 20$, and a false-alarm cost $c_0 = 1$, so $u^* = 1/21 \approx 0.0476$. Suppose the action record carries no signal, so $\eta_A = 0.01$. Since $0.01 \lt u^*$, the best action-only rule never flags, with risk $c_1 p = 0.2$.

A binary trace feature $T$ fires on 60% of failing trajectories and 5% of harmless ones. Then $\Pr(T = 1) = 0.006 + 0.0495 = 0.0555$ and $\Pr(Y = 1 \mid T = 1) = 0.006/0.0555 \approx 0.108$, above $u^*$. The best rule flags exactly when $T = 1$, with risk $20 \times 0.004 + 1 \times 0.0495 = 0.1295$. The trace is worth $0.0705$.

Now let the feature fire on only 20% of failing trajectories and still 5% of harmless ones. It remains informative: a firing raises the failure probability from 1% to $0.002/0.0515 \approx 3.9\%$. That is still below $u^*$, so the best rule still never flags, and the trace is worth exactly zero to this monitor. Flagging on $T = 1$ changes the risk by $c_0(1-p)f - c_1 p s$, so the break-even recall is $s = 0.0495/0.2 = 0.2475$.

Evaluate the Monitor at Its Operating Point

A monitor report should name the failure event, the observation channel, when the decision is made, the threshold, the false-positive rate, the recall, and the review budget. Action-only, trace-only, and combined monitors should be compared on the same trajectories and labels whenever the question allows it.

Once review is scarce, recall is not the number that decides how many failures get caught. Use the audit-budget quantities from AI control: prevalence $p$, recall $s$, false-positive rate $f$, review capacity $b$ as a fraction of all trajectories, and audit sensitivity $h$. The flag rate is $z = ps + (1-p)f$, and when $z > 0$ the precision is $\pi = ps/z$ by Bayes' rule.

Proposition

Under a Binding Review Budget, Caught Failures Track Precision, Not Recall

Statement

The expected fraction of failing trajectories that are reviewed and detected is

$$C = h\, s \min(1, b/z).$$

When the budget binds, $z \ge b$, this equals $C = h\, b\, \pi / p$. Among monitors whose flag rates all exceed the budget, the one with the highest precision catches the most failures, whatever their recalls.

Intuition

When not every alert can be reviewed, the review queue is a random sample of alerts. The number of failures in that sample is the sample size times the fraction of alerts that are real, which is precision. Extra recall bought with many extra false alarms dilutes the queue.

Proof Sketch

A failing trajectory is flagged with probability $s$. Uniform selection reviews a flagged trajectory with probability $\min(b, z)/z = \min(1, b/z)$, and review detects the failure with probability $h$. Multiplying gives $C$. When $z \ge b$, $C = h s b / z$, and $s/z = \pi/p$ because $\pi = ps/z$.

Why It Matters

Take $p = 0.01$, $b = 0.02$, and $h = 1$. The trace monitor has 90% recall with a 10% false-positive rate. The action-only monitor has 60% recall and a 2% false-positive rate. The trace monitor flags 10.8% of trajectories at 8.3% precision and catches 16.7% of failures. The action-only monitor flags 2.58% at 23.3% precision and catches 46.5%. The lower-recall monitor catches almost three times as many failures under this budget, which a recall comparison would hide.

Failure Mode

Precision depends on prevalence. An evaluation that elicits many deliberate violations runs at a much higher $p$ than deployment, so its precision, and any caught-failure figure computed from it, will look better than deployment. The formula also assumes uniform selection among alerts. A queue that reviews the most suspicious alerts first follows a different rule, and an adaptive attacker can change $s$ and $z$ at the moments that matter, as the control page discusses.

report a correction →

Report recall at a fixed false-positive rate or a fixed review budget, not only a threshold-free ranking metric such as AUROC. Then break recall down by failure family. A high average can hide an entire category the monitor never catches, and the category a deployment meets most often need not be the one the evaluation sampled most often.

Re-scoring saved trajectories with alternative monitors is a cheap way to compare them on fixed data. ControlArena, a library for control experiments, documents a function, evaluate_monitors_post_hoc, that applies several monitors to completed evaluation logs, so monitors can be compared on the same trajectories without re-running the evaluations. [6] That comparison and a deployment evaluation answer different questions.

Watch Out

Re-scoring saved transcripts is not a deployment test

A monitor scored on saved transcripts faces trajectories produced without it. Once deployed, a monitor can change what happens next: flagged work is rerouted, and an attacker that learns what gets flagged changes its behavior. Offline re-scoring answers which monitor sorts a fixed dataset better. The interactive effect needs its own experiment.

Architecture and Latent Computation

On September 10, 2026, Redwood Research published a proposal that AI developers report how their architectures bear on monitorability. [3] The concern is architectures with opaque recurrence, or with instances of a model communicating through latent states, since these could make chains of thought and messages between agents much harder to monitor. The proposal asks developers to disclose enough about their architectures for outside scientists to judge whether such latent reasoning or communication is possible, including whether agents communicate in latents, store shared memories in latents, or receive parts of other agents' KV caches. It also asks for third-party verified reports of opaque serial depth across near-frontier models, and for monitorability evidence, ideally including stress tests, to be shared and reviewed by third parties on a regular schedule and potentially whenever a new architecture or training method produces a near-frontier model with much higher opaque serial depth. Finally, it asks each developer to publish a policy on architectures that could enable more latent reasoning. It is a transparency and measurement proposal, not a finding that any particular recurrent architecture defeats oversight.

A companion post by Sheffield and colleagues makes the measure concrete. [4] Opaque serial depth, a notion they credit to earlier work at Google DeepMind, is roughly the longest path of computation that does not pass through an interpretable bottleneck. If chain-of-thought tokens count as interpretable and hidden states do not, the opaque serial depth of a standard transformer is proportional to its number of layers. The post proposes a specific standard for which text-producing nodes count as bottlenecks, and presents the resulting quantity, NLS depth, as a good but imperfect proxy for how much unverbalized serial computation a model can perform. The authors call it a complement to measuring monitorability, not a replacement. A proxy for computational opportunity is not a measure of intent or of risk.

The connection to recurrent depth is direct. A looped block adds serial computation between tokens, and computation can move from written reasoning into latent state, shared memory, or messages between agents without leaving a matching trail in any transcript. Whether that move lowers a particular monitor's detection rate is an empirical question for the specific architecture, task, and monitor. The architecture diagram does not settle it.

Watch Out

Opacity is not evidence of intent

An architecture that computes more in latent state reduces what a text monitor can see. That is a statement about the observation channel. It says nothing about what the model is trying to do, and it is not evidence of deception. The testable safety claim is narrower: detection of a named failure at a fixed cost either got worse or it did not.

Detector Transfer Is a Separate Target

In an August 21, 2026 study, Hopkins and colleagues elicited on-policy lies from open-weight models in 12 settings and fine-tuned Gemma-3 models to classify whether they had lied, training on half of the lie types and testing on the rest. [5] They report that in-distribution detection worked well, with AUROC going from 0.60 to 0.95, while cross-category transfer reached only about 0.70 to 0.75, and that zero-shot prompting of larger models often did better than the fine-tuned detectors. Transfer tracked surface similarity: detectors trained on denials of ASCII-art ability transferred to chess sandbagging, while detectors trained on factual lies did not transfer to roleplay. About 25% of candidate labels changed after a relabeling pass by a GPT-5 judge, which the authors read as a sign that deception is hard to tell apart from confusion. The authors state the limits: they used LoRA rather than full fine-tuning, tested Gemma and Qwen models, and studied controlled settings, not alignment faking, reward hacking, or strategic deception during actual use. These experiments on elicited lies do not measure how often deployed models deceive.

The lesson is not that detection is hopeless. In-distribution accuracy and transfer to new failure categories are different quantities, and a monitor validated on one set of categories has made no claim about the next. A successor model, a new tool interface, or a new task family can move a deployment outside the conditions under which its monitor was checked.

A Proposed Monitorability Regression Suite

Everything in this section is a proposed design, not a reported result.

Freeze the failure taxonomy, the workload families, the adjudication rules, and the review budget. Evaluate a predecessor model and its successor separately, and do not assume that matching transcript length means matching computation. Where feasible, report both a capability-matched and a resource-matched comparison, since they answer different questions.

Include ordinary failures, deliberately elicited violations in contained environments, and failure mechanisms held out from monitor development. Test action-only and action-plus-trace monitors. One ablation removes parts of the visible trace before the monitor reads it; that measures how much the monitor depends on those parts. A second arm runs the agent itself with a constrained or absent trace; that measures how the agent's behavior changes, though a loss of capability can be mistaken for computation moving out of the trace. Feeding a monitor an altered trace is not the same as changing what the agent computes, which is why the two arms are separate.

Separate monitor development from confirmation. Run the old monitor unchanged on the successor, then a monitor retrained for the successor, then an independently developed alternative, and record how much new supervision it took to restore the previous operating point. Report recall at a fixed false-positive rate and the flag rate next to precision. Precision rises and falls with prevalence, so a successor that fails less often shows lower precision even when the monitor is unchanged. The outcome that matters is whether caught failures at a fixed budget survive the change, and whether the control protocol still works when monitoring degrades. Turning the resulting counts into a claim is the job of statistical safety cases.

Proposed Research Targets for 2027 and Later

These are research targets. The dates are planning horizons, not forecasts.

One target is a monitoring evaluation that transfers across task families instead of exploiting one benchmark's surface cues. Proposed experiment: build the regression suite over several task families and report leave-one-family-out recall at a fixed false-positive rate, with the worst held-out family printed next to the average.

A second target is oversight that stays useful as parts of the computation leave natural language. Conditional agenda: if an architecture with larger opaque serial depth reaches near-frontier capability, run the same control setting on it and on a lower-depth predecessor at matched capability, and compare how often each completes a covert side task without being flagged at a fixed review budget. A difference would be evidence about that pair of systems in that setting, not a ranking of architectures.

A third target is a monitor that can detect its own loss of coverage early enough for an external controller to reduce exposure, which is the problem alignment under self-modification takes up across model updates. Proposed experiment: insert labeled canary failures into deployment-like traffic and track the monitor's recall on them with an anytime-valid sequential test; the controller cuts permissions when the test rejects the hypothesis that canary recall is at least its validated level. The test's false-alarm guarantee holds when the canary schedule and the validated recall level are fixed before monitoring starts and each canary is detected with probability at least that level given everything observed earlier when the hypothesis holds. That guarantee is about canary recall only; reading the result as evidence about the monitor on real traffic also needs the agent to be unable to tell canaries from natural traffic. Canary recall is evidence about failures that resemble the canaries; natural failures that differ from them need separate evidence.

Mechanistic probes, behavioral monitors, formal checks on tool interfaces, and permission restrictions supply different kinds of evidence. None of them should be declared a universal substitute for the others.

Check your understanding

Does an unfaithful explanation have zero monitoring value?

No. It can still carry a predictive signal for a specific failure, and whether it does has to be measured.

Does the information proposition guarantee that a real combined monitor beats an action-only monitor?

No. It concerns an ideal rule on a fixed distribution with no cost for reading, and even there the gain can be exactly zero.

Does architectural opacity show malicious intent?

No. It can reduce what a monitor observes without establishing any objective or behavior.

Why is a monitorability regression operationally important?

It can weaken a control protocol even while task capability improves.

ExerciseCore

Problem

Failures occur in 2% of trajectories, and review capacity covers 3% of trajectories with perfect detection, selected uniformly from alerts. Monitor X has 80% recall and a 6% false-positive rate. Monitor Y has 50% recall and a 1% false-positive rate. For each, compute the flag rate, the precision, and the fraction of failures caught. Which catches more under this budget, and at what budget does the answer change?

Hint

Reveal Solution

ExerciseAdvanced

Problem

Return to the worked example with $p = 0.01$, $c_0 = 1$, $c_1 = 20$, and a feature that fires on 5% of harmless trajectories. Show that the feature has positive value exactly when its recall $s$ exceeds 0.2475, and compute its value at $s = 0.4$. Then keep $s = 0.2$ but raise the missed-failure cost to $c_1 = 50$. Does the feature now have value, and what does that say about "informative" versus "useful"?

Hint

Reveal Solution

ExerciseResearch

Problem

A lie detector is validated on three failure families and catches all 60 held-out attempts in each. Deployment will also meet a fourth family never seen in evaluation, making up 20% of attempts. First, treating each family's 60 attempts as independent draws from that family's attempt distribution, give a 95% lower confidence bound on recall for one family, and a bound that holds for all three families at once. Second, bound overall deployment recall using only this evidence. Third, explain what would have to be true of the 60 attempts for these bounds to mean anything against an adaptive attacker, and design an evaluation that measures transfer to unseen families instead of assuming it.

Hint

Reveal Solution

Takeaway

Measure what a monitor detects, at what cost, and before which action. Readable reasoning is not evidence that an oversight channel works. A detection rate at a stated operating point, checked on failure families the monitor was not built on, is.

References

Registry ids in monospace resolve in data/content/sources.json. Each source was checked by URL on 2026-09-15. The two propositions are derived on this page; the sources support the findings and proposals attributed to them, in the settings their authors studied.

• [1] Korbak, Balesni, Barnes, Bengio, et al. "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety." 2025. Source. Position paper: chain-of-thought monitoring is imperfect but promising, and may be fragile to development decisions. korbak-2025-cot-monitorability
• [2] Turpin, Michael, Perez, and Bowman. "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting." NeurIPS 2023. Source. Biasing features change answers without being mentioned in the explanations; tested on BIG-Bench Hard with GPT-3.5 and Claude 1.0. turpin-2023-unfaithful-cot
• [3] Redwood Research. "Proposal for tracking the effects of architecture on monitorability." September 10, 2026. Source. Reporting, third-party verification, and stress-testing proposals; not a demonstrated result about any architecture. redwood-2026-architecture-monitorability
• [4] Sheffield, Westover, Finnveden, Pan, Stastny, and Greenblatt. "An operationalization of opaque serial depth." September 10, 2026. Source. Serial depth between text bottlenecks as a proxy for unverbalized serial computation; an incomplete proxy, not a monitorability guarantee. sheffield-2026-opaque-serial-depth
• [5] Hopkins, Khullar, Wang, and Roger. "Fine-Tuned Lie Detectors Failed to Generalize." August 21, 2026. Source. Strong in-distribution and weak cross-category detection of elicited lies; not deployment-prevalence evidence. hopkins-2026-lie-detectors
• [6] UK AI Security Institute and Redwood Research. "ControlArena." Living documentation; accessed 2026-09-15. Source. A library for control experiments whose evaluation guide describes applying monitors post hoc to completed evaluation logs; not a certificate for any application. control-arena-documentation