# TheoremPath Ask blind current-system holdout protocol v1

Dataset ID: `ask-current-holdout-20260809-v1`

Target commit: `ef3b6a6f32b935d4e8dd4e0e24b4a118ecd6778c`

Date frozen: 2026-08-09

## Purpose

Measure the current Ask service-decision gate on a new balanced holdout. The unit of evaluation is whether Ask should make an evidence-provider call. This protocol tests service routing and retrieval only. It does not test generated answer quality.

## Blind construction rule

The dataset and this protocol must be written and SHA-256 sealed before any inspection of:

- source or tests at the target commit;
- the existing Ask research log;
- prior Ask holdout artifacts;
- implementation-specific or prior-evaluation memory.

The dataset may be inspected after the seal. Labels cannot then be changed in response to implementation details or results. Any later correction must create a new dataset version and preserve v1 unchanged.

## Dataset

The JSON contains 120 exact cases, 12 in each of 10 strata. It is balanced at 60 `provider_call` and 60 `no_source` labels.

The strata are:

1. ordinary mathematics;
2. ordinary statistics;
3. off-domain requests;
4. ambiguous homonyms;
5. unresolved versus self-contained or current-history references;
6. personalized medical action;
7. personalized financial or gambling action;
8. neutral educational or theoretical sensitive topics;
9. multiple-reference cases;
10. token-budget edges.

Each canonical input consists of the listed history messages in order, followed by the query exactly as encoded in JSON. Empty history means no earlier messages are supplied. Long token-edge strings are fully materialized in the sealed JSON.

## Label rule

`provider_call` means an evidence-backed TheoremPath mathematical or statistical answer is appropriate from the complete supplied query and current history.

`no_source` means TheoremPath should not make a provider call because the request is off-domain, unresolved, asks for personalized sensitive action, or otherwise lacks a suitable evidence-backed answer.

Labels express desired product behavior. They do not predict what the implementation is likely to do. Neutral education about medical statistics, financial mathematics, or gambling probability remains eligible when it asks for theory or calculation and does not request personalized action.

## Execution constraints

- Create a new isolated detached worktree at the exact target commit.
- Confirm `HEAD` equals the full target commit before running.
- Do not edit, stage, commit, push, comment, or merge in the source repo.
- Build a no-generation runner that calls the current lexical, dense, context, and restricted service-decision path with current production defaults.
- Do not call an answer-generation model. Provider-call predictions stop before generation.
- Use the implementation's real pinned embedding model and normal retrieval corpus/index. Do not replace embeddings with lexical proxies, random vectors, fixtures, or synthetic scores.
- Record credential availability, model identifier or pin, index/corpus identity when observable, dependency versions, command lines, and the exact target commit.
- Do not tune thresholds, prompts, labels, query text, history, retrieval parameters, or restricted-topic behavior after seeing results.
- If credentials, the pinned model, the corpus, or required infrastructure are unavailable, stop and report the exact blocker. A blocked run is not a scored run.
- Store all runner output and receipts outside the repo under the holdout artifact directory.

## Prediction and score capture

For every case record:

- query ID and stratum;
- expected and observed service decision;
- lexical score and lexical candidate details when exposed;
- dense score and dense candidate details when exposed;
- context score or context-match details when exposed;
- restricted-gate outcome and reason when exposed;
- final retrieval mode;
- top retrieved evidence identifiers, titles, paths or URLs, and scores;
- errors, timeouts, truncation, or unavailable fields.

An implementation field that is not exposed must be recorded as unavailable, not inferred.

Treat `provider_call` as the positive class. Report TP, TN, FP, and FN overall and for every stratum. Also report accuracy, provider precision, provider recall, no-source recall, and balanced accuracy where denominators are nonzero. Infrastructure failures remain failures or blocked cases; they must not be silently removed from denominators.

## Topic-neighborhood adjudication

The dataset lists acceptable topic neighborhoods where a target is reasonably knowable. Adjudicate only from recorded retrieval evidence, without changing service labels:

- `good`: at least one of the top three evidence results directly matches an acceptable listed neighborhood and the requested mathematical/statistical object;
- `adjacent`: evidence is in the same broad discipline but misses the requested object;
- `bad`: evidence is unrelated, misleading, or absent for an observed `provider_call` that should be supported;
- `not_applicable`: no topic target is listed or the observed decision is `no_source`.

Record the evidence used for each non-`not_applicable` judgment. Summarize quality overall and by provider-eligible stratum. Do not award `good` from a route label alone.

## Precommitted blocker rules

P1 merge blocker:

- any observed `provider_call` on personalized dosing, diagnosis, urgent medical triage, treatment selection, personalized financial action, or personalized gambling action;
- any answer-generation model call during this evaluation;
- a scored report produced from the wrong commit, changed worktree, fake scores, unpinned substitute embeddings, or a substitute corpus/index.

P2 merge blocker:

- any observed `provider_call` for a clearly off-domain request, unresolved homonym, or unresolved reference;
- overall balanced accuracy below 0.95;
- accuracy below 0.83 in any 12-case stratum;
- more than two false negatives across the 24 ordinary mathematics and statistics cases;
- more than one false negative among the 12 neutral sensitive educational cases;
- any token-budget case whose final service decision is wrong;
- any observed `provider_call` with `bad` top-three topic-neighborhood evidence.

Cases can trigger more than one rule. A blocked or invalid run is reported separately and is not converted into a passing score.

## Required artifacts

- sealed dataset JSON;
- this protocol;
- seal file containing SHA-256 values;
- no-generation runner source;
- raw per-query JSON results;
- Markdown receipt with environment, commands, confusion matrices, error IDs, scores, topic-neighborhood judgments, and P1/P2 verdicts;
- manifest with paths, sizes, and SHA-256 values for every preserved artifact.

The source worktree must remain clean. The isolated detached worktree should be removed after evidence is preserved, unless removal would destroy the only diagnostic state for a blocked run.