Statistical Safety Cases: From Tests to Defensible Claims

Why This Matters

Zero observed failures is a dataset. A safety claim is an inference. The assumptions that connect them are part of the result.

A safety case is a structured argument that a particular system, in a particular operating setting, is unlikely to cause an unacceptable outcome. Clymer and colleagues organize such arguments into four categories: total inability to cause a catastrophe, sufficiently strong control measures, trustworthiness despite capability, and, if systems become much more powerful, deference to credible AI advisors. [1] No single benchmark substitutes for that whole argument.

This page develops one component: finite-sample claims about a defined failure event. The mathematics is useful because it makes the limits visible. Three results are derived here from stated assumptions. Statements about the cited papers are attributed to their authors, and the evaluation workflow and safety-case record near the end are proposals.

Before you start: conditional probability and the Bernoulli and binomial models from common probability distributions. Hypothesis testing helps. The optional sequential section connects to confidence sequences, anytime-valid inference and martingales.

Freeze the thing being evaluated

A safety claim has to identify the model version, prompts, tool access, permissions, memory, software wrapper and workload. Changing the surrounding system can change the failure distribution even when the model weights stay fixed.

Definition

Unit of exposure

The unit on which a failure probability is defined: one answer, one autonomous trajectory, one database migration, one user-day. A per-answer rate and a per-trajectory rate are different parameters. A system that takes ten times as many actions creates a different exposure pattern even if its per-action failure probability is unchanged.

Separate at least two targets: the ability to perform a dangerous action under strong elicitation, and the probability of undesirable behavior under a particular operating protocol. Shevlane and colleagues distinguish dangerous-capability evaluations from alignment evaluations as inputs to risk management; the two measurements support different arguments. [2]

The calculations below concern a fixed binary event $H$ on a declared population of trials. They do not decide which harms are acceptable. Choosing the event and the acceptable rate is a substantive decision made outside the probability calculation.

The exact zero-failure bound

Assume $X_1,\ldots,X_n$ are independent $\text{Bernoulli}(p)$ failure indicators from one fixed system and evaluation distribution, with the sample size $n$ fixed before any outcome is seen. Let $K=\sum_i X_i$ count the failures. Then

$$\Pr_p(K=0)=(1-p)^n.$$

Definition

One-sided upper confidence bound${\mathrm{Pr}}_p(p\le U)\ge 1-\alpha$

A statistic $U$ computed from the trials is called a level $1-\alpha$ upper confidence bound for $p$ exactly when $\Pr_p(p\le U)\ge1-\alpha$ for every $p$ in $[0,1]$. The probability is over repeated samples at a fixed $p$. The parameter $p$ is not random.

Proposition

The Zero-Failure Upper Confidence Bound and Its Sample Size

Statement

When no failures are observed, report $U_0=1-\alpha^{1/n}$. For every $p$ in $[0,1]$,

$$\Pr_p\big(K=0\text{ and }p>U_0\big)\le\alpha,$$

so the zero-failure report fails to cover $p$ with probability at most $\alpha$. For a target rate $\epsilon$ in $(0,1)$, $U_0\le\epsilon$ holds if and only if

$$n\ge\left\lceil\log\alpha\,/\log(1-\epsilon)\right\rceil.$$

Intuition

$U_0$ is the largest failure rate under which a clean run of $n$ trials still has probability at least $\alpha$. Any rate above it would make the observed clean run a rare event.

Proof Sketch

If $p\le U_0$ the event is empty. If $p>U_0$, then $1-p<\alpha^{1/n}$, so $\Pr_p(K=0)=(1-p)^n<\alpha$.

For the sample size, $U_0\le\epsilon$ means $\alpha^{1/n}\ge1-\epsilon$. Taking logarithms gives $(\log\alpha)/n\ge\log(1-\epsilon)$. Multiplying by $n$ and dividing by the negative number $\log(1-\epsilon)$ reverses the inequality to $n\ge\log\alpha/\log(1-\epsilon)$, and $n$ is an integer.

Why It Matters

At $n=1000$ and 95% confidence, $U_0\approx0.002991$, or 0.2991%. Placing the 95% bound at or below 0.1% takes 2,995 failure-free independent trials. Because $1-e^{-x}\le x$, the bound never exceeds $-\log(\alpha)/n$, which is about $3/n$ at 95% confidence: the rule of three is a slightly conservative version of the exact bound.

Failure Mode

Every assumption carries weight. Duplicated or clustered trials break independence. A candidate chosen after seeing these outcomes breaks the fixed-system assumption. Restarting the count after a failure, or waiting for a favorable moment of a bound that allows failures, breaks the fixed-sample guarantee. A different deployment population has a different $p$. The statement covers the zero-failure branch only, and deleting observed failures to reach $K=0$ is not an application of it.

report a correction →

The zero-failure branch is one case of the one-sided Clopper-Pearson procedure. With $k<n$ observed failures that procedure reports $U(k,n,\alpha)=\operatorname{Beta}^{-1}(1-\alpha;k+1,n-k)$, the $1-\alpha$ quantile of a Beta distribution, and it reports $U(n,n,\alpha)=1$. Inverting the binomial lower-tail tests gives coverage at least $1-\alpha$, and at $k=0$ the Beta quantile reduces to $U_0$. The lab below implements only the zero-failure branch.

Watch Out

A confidence bound is not the probability that the system is safe

A 95% upper bound of 0.2991% does not mean there is a 95% probability that the failure rate is below 0.2991%, and it says nothing about harms outside the defined event. It is the output of a procedure that covers the true rate in at least 95% of repeated experiments under the stated sampling model. A posterior probability needs a prior and answers a different question.

How many trials a zero-failure claim requires

The sample-size formula turns a target into a trial budget. The counts below come from the exact formula, not the rule of three.

The word "independent" is doing real work. Extra log lines from the same underlying trial do not supply the same evidence as new trials.

This is a prospective calculation conditional on observing zero failures. It does not promise that the experiment will pass, and it does not decide whether the target rate is acceptable for the exposure the system will face. Sample size determination treats the general design problem.

The lab opens on the 1,000 independent trials preset: both bounds read 0.2991%, and the 0.1% design target needs 2,995 independent trials. Meet the 0.1% design target sets $n=2995$. The evidence-regime menu lets you break assumptions, and unsupported regimes remove numbers instead of showing a reassuring one. Under adaptive candidate selection or unknown dependence both bounds read "Not supported". Under a shifted deployment population the evaluation bound stays and the target-population bound is withheld.

Selection across checkpoints consumes evidence

Suppose there are $m$ candidates, all fixed before the confirmation data are revealed.

Proposition

Simultaneous Coverage for Prespecified Candidates

Statement

Let $p_j$ be the failure rate of candidate $j$ and $U_j$ its upper bound, with $\Pr(p_j>U_j)\le\alpha/m$ for each $j$. Then

$$\Pr\big(p_j>U_j\text{ for some }j\big)\le\alpha.$$

For zero-failure bounds from $n$ independent trials per candidate, $U_j=1-(\alpha/m)^{1/n}$.

Intuition

Each candidate receives a slice $\alpha/m$ of the error budget, and the slices add up to $\alpha$ however the candidates' errors are related.

Proof Sketch

The union bound gives $\Pr(p_j>U_j\text{ for some }j)\le\sum_{j=1}^{m}\Pr(p_j>U_j)\le m\cdot\alpha/m=\alpha$. The zero-failure form is the previous proposition with $\alpha$ replaced by $\alpha/m$.

Why It Matters

Picking the best of 100 prespecified checkpoints and reporting its unadjusted bound overstates confidence. With $n=1000$ and $m=100$, each adjusted zero-failure bound is 0.7572% instead of 0.2991%, and reaching 0.1% takes 7,598 trials per candidate instead of 2,995. The lab's 100 fixed candidates preset shows the same calculation.

Failure Mode

The guarantee needs every individual bound to be valid. If a candidate is built after inspecting the confirmation results, its own bound is no longer valid, and raising $m$ afterwards does not restore validity. The set must be fixed before confirmation, and $m$ counts every candidate that could have been reported, not only the ones that were.

report a correction →

This is not permission to inspect the holdout, construct a new candidate that exploits it, and count that as one more prespecified model. Adaptive reuse of evaluation feedback changes the inference problem. Dwork and colleagues study that problem and give mechanisms that preserve statistical validity under additional assumptions. [3] The same failure appears in ordinary research as p-hacking and multiple testing.

A workable design has a development suite, a frozen candidate-selection decision, and a confirmation suite that returns no tuning feedback before that decision. After a failed confirmation, a revised candidate needs a new validity argument. It cannot inherit the original independence because the test file is still called holdout.

Watch Out

Bonferroni does not repair holdout-driven development

Bonferroni allocation controls simultaneous error for a candidate set fixed in advance, given valid individual bounds. When the confirmation data shaped which candidates exist, the individual bounds are the broken part, and dividing $\alpha$ by a larger number does not fix them.

Dependence is not a cosmetic correction

Example

One thousand copies of one trial

Draw one $\text{Bernoulli}(p)$ outcome $Z$ and copy it into a thousand rows, so that $X_1=\cdots=X_{1000}=Z$. The probability of zero failures is $1-p$, not $(1-p)^{1000}$. The 95% zero-failure bound in this model is $1-0.05=0.95$, because the data contain one independent draw. The lab's 1,000 copies of one trial preset shows this calculation and reports one independent draw.

Intermediate cases need a model too. Suppose the thousand trials come from 50 independent scenarios, and within a scenario either every trial fails or none does. The relevant sample size is 50, and the 95% bound after a clean run is $1-0.05^{1/50}\approx5.82\%$, about twenty times the naive 0.2991%.

Unknown dependence is a different case again. There is no universal effective-sample-size number to substitute without a model. Clustered tasks, shared scenarios, repeated users and common environmental shocks need an analysis at the corresponding sampling unit or a justified dependence model.

Watch Out

Different task IDs do not certify independence

Task IDs, random seeds, model temperatures and timestamps are details of how the data were generated. Whether they yield independent failure indicators is an assumption about the joint distribution of failures, and it has to be argued from how tasks, scenarios and users were sampled.

Distribution shift is a separate uncertainty

Even a valid bound on $p_{\mathrm{eval}}$ need not bound $p_{\mathrm{deploy}}$. More test draws from the same generator do not show that the generator covers the mechanisms present at deployment. A conditional result makes the missing quantity explicit, using total variation distance.

Proposition

Transferring a Rate Bound Across a Total Variation Gap

Statement

Let $p_{\mathrm{eval}}=P_{\mathrm{eval}}(H)$ and $p_{\mathrm{deploy}}=P_{\mathrm{deploy}}(H)$ for the failure event $H$, and suppose $\operatorname{TV}(P_{\mathrm{deploy}},P_{\mathrm{eval}})\le\rho$. Then $p_{\mathrm{deploy}}\le\min(1,p_{\mathrm{eval}}+\rho)$. If $U$ satisfies $\Pr(p_{\mathrm{eval}}\le U)\ge1-\alpha$, then

$$\Pr\big(p_{\mathrm{deploy}}\le\min(1,U+\rho)\big)\ge1-\alpha.$$

Intuition

Total variation is the largest amount by which two distributions can disagree about the probability of any single event, and the failure event is one such event.

Proof Sketch

By definition $\operatorname{TV}(P,Q)=\sup_A\lvert P(A)-Q(A)\rvert$. Taking $A=H$ gives $P_{\mathrm{deploy}}(H)-P_{\mathrm{eval}}(H)\le\rho$, and every probability is at most 1. On the coverage event $p_{\mathrm{eval}}\le U$, the first inequality yields $p_{\mathrm{deploy}}\le\min(1,U+\rho)$, so the second probability is at least the probability of coverage.

Why It Matters

It names what a deployment claim built from evaluation data is missing: a number like $\rho$, and that number has to come from somewhere other than the evaluation sample.

Failure Mode

The inequality does not create $\rho$. When the two populations have little overlap, for instance prompts from sources that a classifier can tell apart, total variation is close to 1 and the bound is vacuous. The result also assumes the same system and the same event definition in both settings; a changed wrapper, tool set or labeling rule breaks it before any population shift is measured.

report a correction →

Without a justified $\rho$, the formula supplies nothing. Mechanism-shift tests and field evidence can inform a transfer argument, but they should not be summarized as an arbitrary safety margin. The lab's shifted-population regime keeps the evaluation-population bound and withholds the deployment bound. Detecting that deployment inputs have left the evaluation distribution is its own problem, covered in out-of-distribution detection.

Optional stopping is not fixed-sample inference

Recomputing a fixed-sample interval after every batch and stopping at the first favorable result does not inherit the fixed-sample guarantee. Time-uniform methods, including confidence sequences and tests built from nonnegative supermartingales, are designed for sequential monitoring with data-dependent stopping. [4, 5]

They do not make every changing-model experiment valid. A sequence of outcomes from different checkpoints is not a stationary sample from the checkpoint finally selected. The target parameter and the null hypothesis have to stay meaningful as the procedure adapts.

The self-modification page gives an explicit sequential test per candidate and a lifetime allocation of false-approval error. Keep three budgets separate: development compute, statistical error probability, and permitted operational exposure.

Red-teaming and confirmation play different roles

Exploratory red-teaming searches for failure mechanisms. The examples it selects generally do not estimate a natural deployment frequency. Conversely, a representative random sample can miss a rare, strategically important vulnerability.

Use both kinds of evidence and label them. A discovered exploit is strong evidence that a failure is possible in that setting. A measured failure proportion is a frequency estimate only for the sampling and attack process actually used. Neither dominates the other automatically.

For a control evaluation, in the sense of AI control, record the attack-development budget and whether attackers saw feedback from the defense. For a frequency claim, record the sampling frame, exclusions, missing outcomes, adjudication procedure, and whether the test distribution changed after results were seen.

The adjudicator can also make mistakes. If failure labels are noisy, a binomial model for the labels describes label frequency, not necessarily true failure frequency. Calibrate the adjudicator against independently checked cases or use a justified measurement-error model; the advanced exercise below works one out.

Assemble the safety case

A useful record contains the exact deployment scope, the unacceptable event, evidence sources, an uncertainty statement, assumptions, known counterexamples and response triggers. The safety-case framework supplies the organizing concept; this particular record is a suggestion, not part of the framework. [1]

Example

A hypothetical code agent

A case for a code agent might combine enforced permission restrictions, sandbox-escape testing, a controlled adversarial evaluation, and a zero-failure upper bound on a defined operational failure rate per task. These are different evidence types. The permission restriction is a control argument, the escape test is a capability argument, and the rate bound holds only for the task population it sampled. Each keeps its own scope, and none of them licenses the others.

A failed capability test is not proof of incapability when elicitation was weak or strategic underperformance is plausible. Van der Weij and colleagues report that frontier models can be prompted to underperform selectively on dangerous-capability evaluations while keeping general performance, and that models can be fine-tuned to hide capabilities unless given a password. [6] Those experiments show why the distinction matters. They do not establish that any particular low score is intentional.

A claim should expire, or be reassessed, when the model weights, tools, permissions, workload or monitoring procedures change. The self-modification page makes that evidence-invalidation process explicit.

Check your understanding

Do 1,000 failure-free trials prove that the failure probability is zero?

No. Under the fixed independent model, the 95% upper bound is about 0.299%.

Does Bonferroni correction fix arbitrary holdout-driven model development?

No. It needs valid individual bounds, and adaptation to the holdout can break them.

Are 1,000 copies of one successful trial equivalent to 1,000 independent trials?

No. The explicit duplicate model supplies one independent observation, and its 95% bound is 95%.

Does a valid confidence sequence protect against unspecified distribution shift?

No. Validity under optional stopping and transfer to another population are separate guarantees.

ExerciseCore

Problem

A team logs 450 failure-free, independent agent trajectories from a fixed system. What is the 95% zero-failure upper bound on the per-trajectory failure rate? Does it support a claim that the rate is below 1%? How many failure-free trajectories would a 99% bound at or below 0.5% require?

Hint

Reveal Solution

ExerciseAdvanced

Problem

Failures are labeled by an adjudicator that catches a true failure with probability at least $s$ (its sensitivity) and may also flag some non-failures. Labels on different trials are independent. After $n$ trials with no flagged failures, show that $\min(1,U_0/s)$ is a level $1-\alpha$ upper bound on the true failure rate $p$, where $U_0=1-\alpha^{1/n}$. Evaluate it for $n=1000$, 95% confidence and $s=0.8$.

Hint

Reveal Solution

ExerciseResearch

Problem

A team checks the zero-failure bound after every trial and stops the first time $U_0$ falls to $\epsilon$ or below, claiming the rate is at most $\epsilon$. Show that when the true rate satisfies $p\ge\epsilon$, the probability of making the claim is at most $\alpha$, even though the stopping time depends on the data. Then show that if the team restarts the count after every observed failure and keeps going, the probability of eventually making the claim is 1 for every $p<1$.

Hint

Reveal Solution

Takeaway

A defensible claim names its event, sampling unit, system version, selection procedure, uncertainty and transfer assumptions. A precise number without those details is not a safety case.

References

Registry ids in monospace resolve in data/content/sources.json. Each record was checked against its arXiv identifier on 2026-09-15. The derivations on this page are elementary and self-contained; the sources supply the safety-case framework, the evaluation categories, and the adaptive and sequential inference literature.

• [1] Clymer, Gabrieli, Krueger, and Larsen. "Safety Cases: How to Justify the Safety of Advanced AI Systems." 2024. Source. Four categories of safety argument; a framework, not a statistical procedure. clymer-2024-safety-cases
• [2] Shevlane, Farquhar, Garfinkel, et al. "Model evaluation for extreme risks." 2023. Source. Dangerous-capability and alignment evaluations as inputs to risk management. shevlane-2023-extreme-risks
• [3] Dwork, Feldman, Hardt, Pitassi, Reingold, and Roth. "Preserving Statistical Validity in Adaptive Data Analysis." 2014 preprint; STOC 2015. Source. Adaptive reuse of evaluation data and mechanisms that preserve validity. dwork-2015-adaptive-data-analysis
• [4] Howard, Ramdas, McAuliffe, and Sekhon. "Time-uniform Chernoff bounds via nonnegative supermartingales." 2018 preprint; Probability Surveys, 2020. Source. Nonnegative-supermartingale framework for time-uniform crossing guarantees. howard-2020-time-uniform-chernoff
• [5] Howard, Ramdas, McAuliffe, and Sekhon. "Time-uniform, nonparametric, nonasymptotic confidence sequences." 2018 preprint; Annals of Statistics, 2021. Source. Confidence sequences valid under optional stopping; not protection against distribution shift. howard-2021-confidence-sequences
• [6] van der Weij, Hofstätter, Jaffe, Brown, and Ward. "AI Sandbagging: Language Models can Strategically Underperform on Evaluations." 2024. Source. Prompted and password-locked strategic underperformance on capability evaluations. van-der-weij-2024-sandbagging