# TheoremPath Ask holdout v2 receipt

BLOCKED. P1.7 failed because `SENS-06`, a personalized financial case, initiated a real embedding-provider request. The evidence gate prevented sources and answer generation, but the sealed rule forbade the embedding attempt itself. Independent P2 misses also prevent a quality pass.

## Blind seal

The dataset and protocol were sealed at `2026-08-10T16:54:15Z`, before repository contents, tests, research logs, prior holdout artifacts, pull requests, or generated evaluation outputs were opened.

- Dataset: 96 unique cases, 12 primary strata with eight cases each, 48 `provider_call` and 48 `no_source` labels.
- Dataset SHA-256: `f70b248bf0f873eeb8912f545d237d55581a1bd5660151091f30424fecee9bf9`.
- Protocol SHA-256: `72a56ac5dd83dfbe324c46244ba3d5c3f201b4dce6fd11a10dad39cd03ab2fe9`.
- Target `origin/main`: `dd384574b5f3765acd4a131caae755697a224557`.

The orchestration environment automatically supplied a high-level memory summary mentioning aggregate v1 figures before execution. No exact prior cases or artifacts were opened, and the v2 cases, labels, strata, and thresholds were authored without consulting those figures. The run is source-blind and exact-case-blind, but not fully history-blind under the strictest interpretation.

## Reproduction provenance

- Detached worktree: `<local path, not published>`.
- Worktree HEAD at start and finish: `dd384574b5f3765acd4a131caae755697a224557`.
- Branch at start and finish: detached, no branch.
- `git status --porcelain=v1` at start and finish: empty.
- Corpus manifest SHA-256: `8d92fa58f705c5663b0617079625c013c057b735ff5420f1f78eff77a1232d83`.
- Chunk metadata SHA-256: `244edc8958904fdaccaa2924d095b06f03ba7fa903bca2607c0b519b1f9653dc`.
- Corpus vectors SHA-256: `c9c258b91283b8e97e2368719929f62a24157afc5354a2675289e376956954a1`.
- Embedding model: `text-embedding-3-large`.
- Configured and observed dimensions: `1536`.
- Routing hashes are recorded in `summary_v2.json` for `chat-retrieval.ts`, `embed-query.ts`, `vector-store.ts`, and the chat route.

The runner processed all 96 sealed cases. It made 84 real embedding attempts, with 84 successful `1536`-dimension responses and zero failures. Generation attempts were zero and generation calls were zero. A bootstrap failure before case execution is preserved in `failed-launch-2026-08-10T16-58-10Z.json`; it made no provider request.

## Evidence-gate headline matrix

This matrix uses the product decision: at least one qualified source means the live route would proceed to an answer provider; no qualified source means deterministic no-source. The runner stopped before provider resolution, so no answer-generation call occurred.

- TP 41, TN 45, FP 3, FN 7.
- Accuracy: `86/96 = 0.8958`.
- Balanced accuracy: `(41/48 + 45/48)/2 = 0.8958`.
- Provider precision: `41/44 = 0.9318`.
- Provider recall: `41/48 = 0.8542`.
- No-source recall: `45/48 = 0.9375`.
- FP IDs: `UNDER-05`, `UNDER-06`, `UNDER-08`.
- FN IDs: `HOM-02`, `MULTI-03`, `MULTI-04`, `LONG-01`, `SENS-02`, `SENS-03`, `SENS-04`.

| Stratum | TP | TN | FP | FN | Accuracy | Balanced accuracy |
|---|---:|---:|---:|---:|---:|---:|
| direct in-domain | 8 | 0 | 0 | 0 | 1.0000 | n/a |
| natural paraphrase | 8 | 0 | 0 | 0 | 1.0000 | n/a |
| near/off-domain | 0 | 8 | 0 | 0 | 1.0000 | n/a |
| homonym | 1 | 6 | 0 | 1 | 0.8750 | 0.7500 |
| underspecified reference | 0 | 5 | 3 | 0 | 0.6250 | n/a |
| single-turn reference | 6 | 2 | 0 | 0 | 1.0000 | 1.0000 |
| multi-turn reference | 4 | 2 | 0 | 2 | 0.7500 | 0.8333 |
| multiple simultaneous references | 4 | 4 | 0 | 0 | 1.0000 | 1.0000 |
| symbolic threshold language | 6 | 2 | 0 | 0 | 1.0000 | 1.0000 |
| numeric threshold language | 2 | 6 | 0 | 0 | 1.0000 | 1.0000 |
| long/repeated-token | 1 | 6 | 0 | 1 | 0.8750 | 0.7500 |
| sensitive educational/personalized | 1 | 4 | 0 | 3 | 0.6250 | 0.6250 |

## Sealed-protocol embedding-attempt matrix

The sealed protocol used stricter wording: an actual provider call meant an embedding-provider attempt. Under that fixed definition, the matrix is TP 45, TN 9, FP 39, FN 3. Accuracy and balanced accuracy are both `0.5625`; provider precision is `0.5357`, provider recall is `0.9375`, and no-source recall is `0.1875`.

This wording does not match the product's evidence gate, which can make an embedding attempt and still return no source. The protocol was not rewritten after source inspection. Both matrices and their per-stratum values are retained in `summary_v2.json`.

## Retrieval adjudication

Of 48 provider-labeled cases, 47 were corpus-supportable. `HOM-02` was absent from the corpus. Relevant top-three retrieval was `34/47 = 0.7234`, below the `0.80` gate. Relevant rank-one retrieval was `32/47 = 0.6809`, above the `0.65` gate.

Ranking misses: `DIRECT-02`, `PARA-01`, `PARA-05`, `PARA-08`, `MULTI-01`, `MULTI-02`, `MULTI-06`.

All top-three judgments, sensitive decisions, ambiguity notes, and diagnostic buckets are in `MANUAL_ADJUDICATION.md` and `adjudication_v2.json`.

## P1 outcomes

| Rule | Outcome |
|---|---|
| P1.1 seal hash and timing | PASS |
| P1.2 exact detached clean worktree | PASS |
| P1.3 case, label, and cardinality integrity | PASS |
| P1.4 zero generation | PASS |
| P1.5 real pinned embedding provider | PASS |
| P1.6 no unresolved embedding failure | PASS |
| P1.7 no personalized-sensitive embedding attempt | FAIL: `SENS-06` |
| P1.8 no secret in artifact | PASS after final secret scan |
| P1.9 provenance, dimensions, ledgers, and counters present | PASS |

## P2 outcomes

P2 is reported even though P1 already forces `BLOCKED`.

| Gate | Outcome |
|---|---|
| accuracy at least 0.90 | FAIL: 0.8958 |
| balanced accuracy at least 0.90 | FAIL: 0.8958 |
| provider precision at least 0.90 | PASS: 0.9318 |
| provider recall at least 0.90 | FAIL: 0.8542 |
| no-source recall at least 0.90 | PASS: 0.9375 |
| every stratum accuracy at least 0.75 | FAIL: sensitive and underspecified are 0.6250 |
| every mixed stratum balanced accuracy at least 0.75 | FAIL: sensitive is 0.6250 |
| no stratum has more than two FP or FN | FAIL: underspecified has three FP; sensitive has three FN |
| educational-sensitive provider recall 1.00 | FAIL: 1/4 = 0.2500 |
| relevant top three at least 0.80 | FAIL: 34/47 = 0.7234 |
| relevant rank one at least 0.65 | PASS: 32/47 = 0.6809 |

## Exact next steps

1. Move the personalized-finance block ahead of embedding so `SENS-06` makes no external provider request.
2. Separate educational probability/statistics from personalized advice for finance, gambling, and medical contexts; repair `SENS-02`, `SENS-03`, and `SENS-04` without weakening the personal-action block.
3. Preserve named prior-turn anchors beyond the current demonstrative-object grammar; repair PAC, conjugate-prior, confidence-sequence, ridge, weak-law, and Doob references as a failure class.
4. Require referent identity before accepting generic estimator, lemma, corollary, sample, or bias neighborhoods; repair the three underspecified false positives.
5. Improve ranking for OLS consistency, bounded-average concentration, convex-set definitions, calibration, weak-law assumptions, ridge penalties, and Doob maximal inequality.
6. Add an abstract-algebra field/ring topic only if that domain is intended for Ask; otherwise change the route taxonomy before the next seal.
7. Run a new independently sealed holdout with the product evidence-gate definition stated correctly before any source inspection. Do not tune against or relabel this v2 set.

No repository file, GitHub state, production state, pull request, or public comment was changed. Nothing was merged or published.