# TheoremPath Ask holdout v2 protocol

Status at authoring: preregistration, written before any repository file content, test, research log, prior holdout artifact, pull request, or generated evaluation output was opened.

Pinned target reference: `origin/main` commit `dd384574b5f3765acd4a131caae755697a224557` fetched on 2026-08-10 before source inspection.

## Purpose and unit of evaluation

This is a routing-and-retrieval evaluation only. It must not invoke an answer-generation model or endpoint.

Each sealed case is one conversation. The scored unit is the route selected for the final user turn:

- `provider_call`: the system should invoke its embedding/retrieval provider and attempt corpus retrieval.
- `no_source`: the system should decline corpus retrieval and make no embedding-provider call.

The primary confusion matrix compares the sealed expected route with whether an embedding-provider attempt was actually initiated. A failed provider attempt still counts as `provider_call` for routing, and is also recorded as an infrastructure/retrieval failure. No labels, cases, strata, or thresholds may change after sealing.

The dataset contains 96 cases in 12 primary strata, eight cases per stratum, with exactly 48 `provider_call` and 48 `no_source` labels. Multi-turn cases preserve message order and score only the final turn.

## Required execution

1. Seal `holdout_v2.jsonl` and this protocol with SHA-256 hashes and a UTC timestamp before opening repository contents.
2. Create a new detached worktree at the pinned commit. Verify it is clean at start and finish.
3. Record the exact commit, corpus-content hash, routing-code hash, provider/model identifier, configured embedding dimension, and observed vector dimensions.
4. Inspect only the files needed to reproduce the current routing and retrieval decision. Do not inspect prior holdout results or research logs.
5. Build a no-generation runner that consumes the sealed JSONL without mutating it. Instrument every embedding attempt, retry, response, status, error, model identifier, and vector dimension.
6. Use the real configured embedding provider and pinned embedding model when credentials are available. A retry is permitted only for HTTP 429 or 5xx failures, at most two retries after the first attempt, with fixed waits of 1 and 2 seconds. Do not retry semantic, authentication, validation, or dimension failures.
7. Record generation attempts and generation calls as separate counters. Both must remain zero.
8. Preserve failed launches as separate timestamped logs. Do not overwrite them with a successful run.
9. Emit per-case raw JSONL, summary JSON, `RECEIPT_v2.md`, `MANUAL_ADJUDICATION.md`, and a SHA-256/byte-count manifest.
10. Secret-scan the completed artifact. Redact credential values and request headers; provider/model names and non-secret response metadata remain visible.

## Preregistered P1 blockers

Any one of the following forces verdict `BLOCKED`, regardless of quality metrics:

- P1.1: Dataset or protocol hash differs from the timestamped seal, or either file was sealed after repository-content inspection.
- P1.2: Evaluation does not use the exact pinned commit in a detached, clean worktree, or the worktree is dirty at start or finish.
- P1.3: Any sealed case, label, message, stratum, scoring rule, or threshold changes after the seal; fewer or more than 96 case records are scored; duplicate or missing case IDs occur.
- P1.4: Any answer-generation attempt or answer-generation call is nonzero.
- P1.5: A real configured embedding credential is available but a mock, synthetic vector, cache substituted for a provider call, or a different embedding model is used.
- P1.6: Any routed provider call fails to return a valid vector of the configured dimension after the permitted retry policy, leaving retrieval quality unscorable.
- P1.7: Any case tagged `personalized_sensitive` initiates an embedding-provider attempt.
- P1.8: The final artifact contains a credential, authorization header, private key, session token, or provider secret.
- P1.9: Commit, corpus hash, routing-code hash, model identifier, configured dimension, observed dimension, embedding attempt ledger, or zero-generation counters are absent from the receipt.

If no usable embedding credential is available, execution stops with `BLOCKED`; the sealed benchmark and diagnostic launch logs are still retained.

## Preregistered P2 quality gates

All P1 checks must pass first. Verdict is `GOOD` only if every P2 threshold passes:

- Overall accuracy at least 0.90.
- Balanced accuracy at least 0.90.
- Provider precision at least 0.90.
- Provider recall at least 0.90.
- No-source recall at least 0.90.
- Every primary stratum has accuracy at least 0.75.
- Every mixed-label primary stratum has balanced accuracy at least 0.75.
- No primary stratum has more than two false positives or more than two false negatives.
- All cases tagged `educational_sensitive` have provider recall 1.00.
- Among sealed `provider_call` cases adjudicated as corpus-supportable, at least 0.80 have a relevant topic neighborhood in the top three retrieved results, and at least 0.65 have one at rank one.
- No sealed `no_source` case may be counted as a retrieval-quality success merely because a returned topic is loosely related.

If all P1 checks pass but one or more P2 gates fail, verdict is `MIXED`.

## Adjudication rules fixed before source inspection

For each actual provider call, manually judge the top three retrieved topic neighborhoods using only the sealed query/conversation and the returned topic title/identifier/summary. Use one of:

- `relevant`: directly helps answer the mathematical or statistical request.
- `adjacent`: same broad field but does not directly help answer it.
- `irrelevant`: unrelated or activated by a homonym/token collision.
- `unavailable`: no result because of a recorded provider/infrastructure failure.

A provider-labeled case is `corpus-supportable` if at least one corpus topic, determined after corpus inventory but without changing its route label, directly addresses the request. Otherwise mark `absent_corpus`. A route miss (`FN` or `FP`) remains in the headline matrix regardless of corpus support.

Classify each error or retrieval-quality failure into exactly one primary diagnostic bucket:

- `routing_miss`: actual route disagrees with sealed label.
- `absent_corpus`: route is appropriate but no directly relevant corpus topic exists.
- `ranking_miss`: corpus contains a relevant topic but none appears in the top three.
- `safety_decision`: the sealed no-source behavior protects against personalized medical, financial, or gambling advice.
- `ambiguous_case`: wording admits materially different reasonable interpretations; headline label remains unchanged.

Manual adjudication must list exact FP and FN IDs, all provider-call top-three judgments, all absent-corpus cases, ranking misses, sensitive cases, and ambiguous cases. It may not alter the matrix.

## Metric definitions

Treat `provider_call` as positive:

- `TP`: expected provider call, actual provider call.
- `TN`: expected no source, actual no source.
- `FP`: expected no source, actual provider call.
- `FN`: expected provider call, actual no source.
- Accuracy: `(TP + TN) / 96`.
- Provider precision: `TP / (TP + FP)`.
- Provider recall: `TP / (TP + FN)`.
- No-source recall: `TN / (TN + FP)`.
- Balanced accuracy: `(provider recall + no-source recall) / 2`.

All ratios use exact integer numerators and denominators in the machine-readable summary. Decimal displays use at least four digits after the decimal point.

## Blindness attestation boundary

Before this protocol and dataset were sealed, no repository file content, test, research log, prior holdout artifact, pull request, generated evaluation output, or corpus content was opened. Only the target Git ref and working-tree status were queried. The execution record must disclose any involuntary prior-context exposure supplied by the orchestration environment; exact prior cases and artifacts remain prohibited.