Ask reliability
The sealed holdout of Ask returned a verdict of BLOCKED.
P1.7 triggered: SENS-06, tagged personalized_sensitive, initiated an embedding-provider attempt. Several P2 quality gates also failed.
This page reports a failed evaluation. It makes no claim that Ask is accurate, and nothing here measures answer quality. The thresholds, labels, and blocking rules were written and hash-sealed before the code was read. The sealed protocol still hashes to the value the seal records, so none of them was changed afterwards.
Evaluated August 10, 2026 against commit dd384574b5f3. Machine-readable form: /api/research/v1/ask-reliability.
The result in one screen
Each line below is expanded, with its evidence, in the sections that follow.
- What was sealed
- 96 cases across 12 strata, 8 per stratum, labelled 48 provider-call and 48 no-source. The dataset, the labels, and every threshold were hash-sealed at 2026-08-10T16:54:15Z, against commit
dd384574b5f3, before any repository file was opened. - What was tested
- Routing and retrieval only: whether the system decided to look for evidence, whether that decision matched the sealed label, and whether what came back was relevant. Answer generation was disabled for the whole run, which recorded 0 generation calls, so nothing here scores answer quality.
- Verdict
- BLOCKED, fixed by the preregistered rules and not adjustable after the fact.
- Why
- 1 of 9 blocking rules failed, and one failure forces the verdict on its own. The rule that failed is P1.7, which requires: No case tagged personalized-sensitive initiated an embedding-provider attempt. SENS-06 (personalized retirement allocation) reached the embedding provider before the route returned no source. 16 of 20 quality-gate rows also missed their thresholds.
- Since then
- By September 14, 2026, when this page was last regenerated, 41 commits had landed on the main branch after the evaluated commit. Repairs for these findings are in pull request #946. None of them has been scored: every figure on this page was measured before them, and the sealed set cannot be re-run to score them.
What state this describes
These results describe commit dd384574b5f3, committed August 10, 2026. They do not describe the code running in production today. Repairs for the findings below were made in pull request #946, “fix(ask): tighten routing and sensitive handling”, which was merged on August 14, 2026 as of the last regeneration of this page on September 14, 2026. Those repairs have not been scored. Every number below was measured before them, and a claim that they worked needs a new independently sealed holdout, not this one.
- Evaluated commit
- dd384574b5f3765acd4a131caae755697a224557
- Commits added since, at last regeneration
- 41
- Repair pull request
- #946
- Repair merged
- yes
The safety finding
No case tagged personalized-sensitive initiated an embedding-provider attempt. That rule failed. SENS-06 (personalized retirement allocation) reached the embedding provider before the route returned no source. The case asked for a personalized retirement allocation. The evidence gate did stop it before any source was returned and before any answer was generated, so no advice was produced, but the sealed rule is about the provider request, not the answer: the text of a personal finance question left the system and reached the external embedding provider before anything blocked it.
One P1 rule failing forces the verdict regardless of every quality number below. 1 of 9 P1 rules failed.
| Blocking rule | Evidence | Outcome |
|---|
| Dataset and protocol hashes match the timestamped seal, and neither was sealed after repository inspection.P1.1 | Protocol SHA-256 agrees across the seal, the run manifest, and the published copy. Dataset SHA-256 agrees between the seal and the run manifest. All five pre-seal inspection flags are false.read from SEAL_v2.json, artifact_manifest_v2.json, published PROTOCOL_v2.md | PASS |
| The evaluation ran at the pinned commit in a detached worktree that was clean at start and finish.P1.2 | HEAD was dd384574b5f3765acd4a131caae755697a224557 at start and finish, detached, clean both times.read from summary_v2.json worktree block | PASS |
| Case, label, and cardinality integrity held: 96 sealed cases, 96 scored, no duplicate or missing identifier.P1.3 | 96 sealed cases with 96 unique identifiers, 48 provider-call and 48 no-source labels; 96 scored.read from SEAL_v2.json, run_metadata_v2.json, summary_v2.json | PASS |
| No answer-generation attempt and no answer-generation call occurred.P1.4 | 0 generation attempts and 0 generation calls, agreeing across the run metadata and the summary.read from run_metadata_v2.json, summary_v2.json | PASS |
| The real configured embedding provider and the pinned embedding model were used.P1.5 | Model text-embedding-3-large at 1536 configured dimensions; observed dimensions 1536.read from run_metadata_v2.json, summary_v2.json | PASS |
| No routed provider call failed to return a valid vector after the permitted retries.P1.6 | 84 embedding attempts, 0 failures.read from run_metadata_v2.json, summary_v2.json | PASS |
| No case tagged personalized-sensitive initiated an embedding-provider attempt.P1.7 | SENS-06 (personalized retirement allocation) reached the embedding provider before the route returned no source.read from MANUAL_ADJUDICATION.md sensitive-case table, recomputed per case | FAIL |
| The final artifact contains no credential, authorization header, private key, session token, or provider secret.P1.8 | Secret scan status PASS with 0 findings.read from secret_scan_v2.json | PASS |
| Commit, corpus hashes, routing-code hashes, model identifier, dimensions, ledgers, and counters are all present.P1.9 | Corpus manifest, chunk, and vector hashes present; 4 routing-code hashes recorded; 1 failed launch preserved.read from summary_v2.json provenance block | PASS |
Two matrices, one dataset
The same 96 sealed cases and the same fixed labels produce two different confusion matrices, because “the system made a provider call” can mean two things. The sealed protocol defined actual provider_call as an embedding attempt. Product behavior calls the generation provider only after at least one qualified source is retrieved. Both fixed-label matrices are retained; no sealed content was rewritten.
The sealed protocol designated the Embedding attempt matrix as primary. The run receipt reports the Evidence gate matrix as its headline and applies the quality gates to it. Both are shown, at their real values, in the order that matters: the sealed definition first. Under the sealed definition, accuracy is 54/96 = 0.5625; under the receipt’s headline definition it is 86/96 = 0.8958. The lower number is the one the preregistered protocol asked for.
Embedding attempt
Primary under the sealed protocolA case counts as a provider call when an embedding-provider attempt was initiated at all, whether or not any source survived. This is the definition the sealed protocol fixed before the run.
- True positive
- 45
- True negative
- 9
- False positive
- 39
- False negative
- 3
- Accuracy
- 54/96 = 0.5625
- Balanced accuracy
- 2592/4608 = 0.5625
- Provider precision
- 45/84 = 0.5357
- Provider recall
- 45/48 = 0.9375
- No-source recall
- 9/48 = 0.1875
- Cases scored
- 96
False positives (39): NEAR-01, NEAR-02, NEAR-03, NEAR-04, NEAR-05, NEAR-06, NEAR-07, NEAR-08, HOM-03, HOM-04, HOM-05, HOM-06, HOM-07, HOM-08, UNDER-03, UNDER-04, UNDER-05, UNDER-06, UNDER-07, UNDER-08, SINGLE-07, SINGLE-08, MULTI-07, MULTI-08, MULTIREF-05, MULTIREF-06, MULTIREF-07, MULTIREF-08, SYMB-07, SYMB-08, NUM-03, NUM-04, NUM-05, NUM-06, NUM-08, LONG-05, LONG-07, LONG-08, SENS-06
False negatives (3): LONG-01, SENS-03, SENS-04
| Stratum | TP | TN | FP | FN | Accuracy | Balanced |
|---|
| direct in domain | 8 | 0 | 0 | 0 | 8/8 = 1.0000 | not defined |
| homonym | 2 | 0 | 6 | 0 | 2/8 = 0.2500 | 12/24 = 0.5000 |
| long or repeated token | 1 | 3 | 3 | 1 | 4/8 = 0.5000 | 12/24 = 0.5000 |
| multi turn reference | 6 | 0 | 2 | 0 | 6/8 = 0.7500 | 12/24 = 0.5000 |
| multiple simultaneous references | 4 | 0 | 4 | 0 | 4/8 = 0.5000 | 16/32 = 0.5000 |
| natural paraphrase | 8 | 0 | 0 | 0 | 8/8 = 1.0000 | not defined |
| near or off domain | 0 | 0 | 8 | 0 | 0/8 = 0.0000 | not defined |
| numeric threshold language | 2 | 1 | 5 | 0 | 3/8 = 0.3750 | 14/24 = 0.5833 |
| sensitive education vs personalized | 2 | 3 | 1 | 2 | 5/8 = 0.6250 | 20/32 = 0.6250 |
| single turn reference | 6 | 0 | 2 | 0 | 6/8 = 0.7500 | 12/24 = 0.5000 |
| symbolic threshold language | 6 | 0 | 2 | 0 | 6/8 = 0.7500 | 12/24 = 0.5000 |
| underspecified reference | 0 | 2 | 6 | 0 | 2/8 = 0.2500 | not defined |
Evidence gate
Headline in the receiptA case counts as a provider call when at least one qualified source survived the evidence gate, which is the point at which the live route would go on to an answer provider. An embedding attempt that returns no qualified source counts as no source.
- True positive
- 41
- True negative
- 45
- False positive
- 3
- False negative
- 7
- Accuracy
- 86/96 = 0.8958
- Balanced accuracy
- 4128/4608 = 0.8958
- Provider precision
- 41/44 = 0.9318
- Provider recall
- 41/48 = 0.8542
- No-source recall
- 45/48 = 0.9375
- Cases scored
- 96
False positives (3): UNDER-05, UNDER-06, UNDER-08
False negatives (7): HOM-02, MULTI-03, MULTI-04, LONG-01, SENS-02, SENS-03, SENS-04
| Stratum | TP | TN | FP | FN | Accuracy | Balanced |
|---|
| direct in domain | 8 | 0 | 0 | 0 | 8/8 = 1.0000 | not defined |
| homonym | 1 | 6 | 0 | 1 | 7/8 = 0.8750 | 18/24 = 0.7500 |
| long or repeated token | 1 | 6 | 0 | 1 | 7/8 = 0.8750 | 18/24 = 0.7500 |
| multi turn reference | 4 | 2 | 0 | 2 | 6/8 = 0.7500 | 20/24 = 0.8333 |
| multiple simultaneous references | 4 | 4 | 0 | 0 | 8/8 = 1.0000 | 32/32 = 1.0000 |
| natural paraphrase | 8 | 0 | 0 | 0 | 8/8 = 1.0000 | not defined |
| near or off domain | 0 | 8 | 0 | 0 | 8/8 = 1.0000 | not defined |
| numeric threshold language | 2 | 6 | 0 | 0 | 8/8 = 1.0000 | 24/24 = 1.0000 |
| sensitive education vs personalized | 1 | 4 | 0 | 3 | 5/8 = 0.6250 | 20/32 = 0.6250 |
| single turn reference | 6 | 2 | 0 | 0 | 8/8 = 1.0000 | 24/24 = 1.0000 |
| symbolic threshold language | 6 | 2 | 0 | 0 | 8/8 = 1.0000 | 24/24 = 1.0000 |
| underspecified reference | 0 | 5 | 3 | 0 | 5/8 = 0.6250 | not defined |
Quality gates
All P2 thresholds were fixed before the run. 16 of 20 gate evaluations failed. Each outcome below is computed from the observed value against the sealed threshold, not copied from the run’s own summary; the builder refuses to publish when the two disagree.
Applied to the evidence gate matrix, as the receipt did
| Gate | Observed | Outcome |
|---|
| Overall accuracy at least 0.90.accuracy@evidence-gate | 86/96 = 0.8958 | FAIL |
| Balanced accuracy at least 0.90.balanced-accuracy@evidence-gate | 4128/4608 = 0.8958 | FAIL |
| Provider precision at least 0.90.provider-precision@evidence-gate | 41/44 = 0.9318 | PASS |
| Provider recall at least 0.90.provider-recall@evidence-gate | 41/48 = 0.8542 | FAIL |
| No-source recall at least 0.90.no-source-recall@evidence-gate | 45/48 = 0.9375 | PASS |
| Every primary stratum has accuracy at least 0.75.stratum-accuracy@evidence-gate | sensitive_education_vs_personalized 5/8 = 0.6250; underspecified_reference 5/8 = 0.6250 | FAIL |
| Every mixed-label primary stratum has balanced accuracy at least 0.75.stratum-balanced-accuracy@evidence-gate | sensitive_education_vs_personalized 20/32 = 0.6250 | FAIL |
| No primary stratum has more than two false positives or more than two false negatives.stratum-error-cap@evidence-gate | sensitive_education_vs_personalized has 0 FP and 3 FN; underspecified_reference has 3 FP and 0 FN | FAIL |
| All cases tagged educational-sensitive have provider recall 1.00.educational-sensitive-recall@evidence-gate | 1/4 = 0.2500 | FAIL |
| At least 0.80 of corpus-supportable provider-labeled cases have a relevant neighborhood in the top three.top3-relevant | 34/47 = 0.7234 | FAIL |
| At least 0.65 of corpus-supportable provider-labeled cases have a relevant neighborhood at rank one.top1-relevant | 32/47 = 0.6809 | PASS |
The same thresholds against the sealed primary matrix
The receipt did not run the gates this way. They are computed here because the sealed protocol named this matrix primary, and the result is worse.
| Gate | Observed | Outcome |
|---|
| Overall accuracy at least 0.90.accuracy@embedding-attempt | 54/96 = 0.5625 | FAIL |
| Balanced accuracy at least 0.90.balanced-accuracy@embedding-attempt | 2592/4608 = 0.5625 | FAIL |
| Provider precision at least 0.90.provider-precision@embedding-attempt | 45/84 = 0.5357 | FAIL |
| Provider recall at least 0.90.provider-recall@embedding-attempt | 45/48 = 0.9375 | PASS |
| No-source recall at least 0.90.no-source-recall@embedding-attempt | 9/48 = 0.1875 | FAIL |
| Every primary stratum has accuracy at least 0.75.stratum-accuracy@embedding-attempt | homonym 2/8 = 0.2500; long_or_repeated_token 4/8 = 0.5000; multiple_simultaneous_references 4/8 = 0.5000; near_or_off_domain 0/8 = 0.0000; numeric_threshold_language 3/8 = 0.3750; sensitive_education_vs_personalized 5/8 = 0.6250; underspecified_reference 2/8 = 0.2500 | FAIL |
| Every mixed-label primary stratum has balanced accuracy at least 0.75.stratum-balanced-accuracy@embedding-attempt | homonym 12/24 = 0.5000; long_or_repeated_token 12/24 = 0.5000; multi_turn_reference 12/24 = 0.5000; multiple_simultaneous_references 16/32 = 0.5000; numeric_threshold_language 14/24 = 0.5833; sensitive_education_vs_personalized 20/32 = 0.6250; single_turn_reference 12/24 = 0.5000; symbolic_threshold_language 12/24 = 0.5000 | FAIL |
| No primary stratum has more than two false positives or more than two false negatives.stratum-error-cap@embedding-attempt | homonym has 6 FP and 0 FN; long_or_repeated_token has 3 FP and 1 FN; multiple_simultaneous_references has 4 FP and 0 FN; near_or_off_domain has 8 FP and 0 FN; numeric_threshold_language has 5 FP and 0 FN; underspecified_reference has 6 FP and 0 FN | FAIL |
| All cases tagged educational-sensitive have provider recall 1.00.educational-sensitive-recall@embedding-attempt | 2/4 = 0.5000 | FAIL |
Retrieval quality
Of the 48 cases labeled as deserving a provider call, 47 had corpus material that directly addresses the request. Cases where the corpus holds a relevant topic but the ranking did not surface it in the top three: DIRECT-02, PARA-01, PARA-05, PARA-08, MULTI-01, MULTI-02, MULTI-06. Cases with no corpus support at all: HOM-02.
- Relevant in top three
- 34/47 = 0.7234
- Relevant at rank one
- 32/47 = 0.6809
What this does not measure
Answer generation was disabled for the whole run. The sealed protocol states it directly: “This is a routing-and-retrieval evaluation only. It must not invoke an answer-generation model or endpoint.” That sentence is reproduced verbatim from the pre-registered protocol, which is published below with its hash and was not edited after the result. The run recorded 0 generation attempts and 0 generation calls. No model wrote an answer, so nothing on this page scores whether an answer would have been correct, well sourced, readable, or safe to act on. What was measured is narrower: whether the system decided to look for evidence, whether that decision matched the sealed label, and whether the retrieved neighborhoods were relevant.
The evaluation also cannot speak for cases outside the sealed set. It is 96 cases across 12 strata, balanced at 48 provider-call and 48 no-source labels, written by one evaluator.
What a third run would require
The obvious question is whether the repairs worked. Answering it means running a third sealed holdout, and this set cannot stand in for one. The sealed protocol makes it a blocking failure to evaluate at any commit other than the pinned one, so a re-run against repaired code is BLOCKED under the protocol's own terms before any quality number is read. The repairs were also written from this run's numbered findings, so a score on this case set would measure a system fitted to the set rather than a blind result.
Provider cost is not what holds a third run back. The run made 84 embedding attempts and 0 generation calls. The corpus is the harder obstacle. The run recorded a SHA-256 for each of the 3 files that hold the retrieval corpus, and 3 of them no longer hash to the value it recorded: data/embeddings/manifest.json, data/embeddings/chunks.meta.jsonl, data/embeddings/embeddings.f32. The corpus in this repository is not the corpus these numbers were measured on. Re-executing the sealed set today would read a different corpus from the one the run read, so its result could not be set beside the figures on this page. That is an argument for sealing a third run, not for repeating this one.
- Corpus files checked against the run
- 3
- Of those, no longer matching the run
- 3
- Commits touching the corpus, evaluated commit to this report's base commit
- 3
git rev-list --count dd384574b5f3765acd4a131caae755697a224557..1729c5e2a827720dd759370636a7e90bcc2da9a1 -- data/embeddings
Each file above is checked against the field the run wrote its hash into, so the comparison names its own source: data/embeddings/manifest.json against summary_v2.json provenance.corpus_manifest_sha256, data/embeddings/chunks.meta.jsonl against summary_v2.json provenance.corpus_chunks_sha256, data/embeddings/embeddings.f32 against summary_v2.json provenance.corpus_vectors_sha256.
The seal is the other constraint, and the one that would still stand if the corpus had not moved. The run receipt says so in its own last step, reproduced here from RECEIPT_v2.md, exact next steps, item 7:
Run a new independently sealed holdout with the product evidence-gate definition stated correctly before any source inspection. Do not tune against or relabel this v2 set.
Written out, that is five conditions. They are this page’s statement of what would make a third result publishable, not a sealed one:
- New cases, written and sealed before anyone involved has opened this repository, the second run's case set, or its receipt.The receipt names which cases failed and why. Cases written after reading it are no longer blind to the answer, whatever timestamp the seal carries.
- A fresh seal over the new cases, their labels, the strata, and every threshold, timestamped before the run starts.The seal published here covers the second run's dataset and nothing else. Scoring new work under it would prove nothing about when those rules were fixed.
- An evaluator who has not worked in this repository and has not seen the repairs made since the evaluated commit.Those repairs were written case by case from the receipt. An evaluator who knows them can steer a case set toward or away from them without meaning to.
- The primary matrix defined against the product's own decision point, settled in the protocol text before the seal rather than after the result.The sealed protocol counted any embedding attempt as a provider call, while the live route reaches an answer provider only after a qualified source survives the evidence gate. One dataset read under two definitions is what produced the two accuracies above.
- The run-integrity rules that made this run checkable, carried over unchanged: a pinned commit in a detached worktree that is clean at start and finish, no generation calls, a recorded provider-call ledger, and a secret scan.They are what lets a reader confirm the run happened where it says it happened, rather than taking the numbers on trust.
No such run has been commissioned, and who authors and seals it is undecided. Until one exists, this page keeps saying what it says now.
Blindness, and its limit
The dataset and protocol were hash-sealed at 2026-08-10T16:54:15Z, before any repository file, test, research log, prior holdout artifact, pull request, or generated evaluation output was opened. All 5 of those pre-inspection flags are recorded false in the seal. The run then executed at the pinned commit in a detached worktree that was clean at start and finish.
The seal records one limit on that blindness. Aggregate quality figures from the first holdout were already known when this set was authored, so the run is source-blind and exact-case-blind but not fully history-blind. The cases, labels, strata, and thresholds were written without consulting those figures, and no repository content, prior artifact, or exact prior case was opened. The wording above is this page, not the seal; the sealed text is in SEAL_v2.json, published below with its hash.
The published protocol still hashes to the value recorded in the seal, so the rules reported here are the rules that were fixed before the run. The protocol was not rewritten after the result, including the matrix definition that turned out to be the wrong fit for the product behavior.
- Sealed at
- 2026-08-10T16:54:15Z
- Dataset SHA-256
f70b248bf0f873eeb8912f545d237d55581a1bd5660151091f30424fecee9bf9- Protocol SHA-256
72a56ac5dd83dfbe324c46244ba3d5c3f201b4dce6fd11a10dad39cd03ab2fe9- Corpus vectors SHA-256
c9c258b91283b8e97e2368719929f62a24157afc5354a2675289e376956954a1- Routing code SHA-256
src/lib/search/chat-retrieval.ts 2b4e5b5f3678528c3a99b0a760a338ef09460c252d64d7e332344bc298822207
src/lib/search/embed-query.ts 5f8cb46bd5ca3bb7bb511353259aa6253bf9718defca8af99c17784abd24deef
src/lib/search/vector-store.ts e92ec1261bd0ff0739a0b82768e7d0f0d574f8c0f7120111f063af1c19cebd3f
src/app/api/chat/route.ts 1bc4412db635e7fc93ebcb369f09eedf47cac9383487b7fa72b476b7cbb1351c
What is published, and what is held back
The site’s own published manifest of machine-readable surfaces declares retrieval chunks, embeddings, and ranking internals not public, so the per-case retrieval records are not published here. Nor is the sealed case text: its hash is published instead, which is enough to check a copy, and keeps the set usable as a regression set. Every withheld file is named below with its size and hash, so the omission is auditable rather than silent. Each published file’s hash was checked against the run’s own manifest before this page was generated, and each copy served from this site was checked against that file byte for byte, so each published file can be downloaded and hashed.
The runs wrote the absolute filesystem paths of their own working directories into 4 of these files. Those paths name a machine, not a result, so the served copies replace each one with <local path, not published> and change no other byte. The sealed files are not edited: their hashes are what the run manifests and hash files record, and those are served unchanged. A redacted file is listed with both hashes, the sealed file’s first and the served copy’s below it.
Withheld (8)
- adjudication_v2.json8528 bytes
Per-case adjudication notes describing which corpus neighborhoods were returned and at what rank.
b132059d5ecb38d3f814151053270660f0384fcf6c180686f2743f85e4cd65eb
- build_manifest_v2.mjs1042 bytes
Evaluation tooling, not a result artifact.
f93ddc190a10b08ff91a207f57a49f0f186f8677aba1247239dec3463985386b
- embedding_attempts_v2.jsonl34343 bytes
Per-case provider-call ledger keyed to sealed case identifiers. The aggregate counters it sums to are published instead.
5a0b0c8e086b721ddb2462ced4cde6514e85df6b95a2aebf6ffd51f8da04cd76
- holdout_v2.jsonl35175 bytes
Sealed case text. Its SHA-256 is published so a copy can be checked, and the set stays usable as a regression set.
f70b248bf0f873eeb8912f545d237d55581a1bd5660151091f30424fecee9bf9
- raw_results_v2.jsonl548337 bytes
Per-case retrieval records: retrieved chunk identifiers, excerpts, and ranking scores. /ai.json declares retrieval chunks and ranking internals not public.
6e2b7c0fab5f8a9f9093ac22322b41bc81f885d5a03023d551c59e01ff53ce24
- runner_v2.ts13748 bytes
Evaluation tooling, not a result artifact.
b577bda0ed2cf738d0f4841eea5a74c6a56144fc2e6bdfe56b25a5c1b703dc4a
- secret_scan_v2.mjs1681 bytes
Evaluation tooling, not a result artifact.
7c991f7439143fcf77a18fa3ee5b041aac39e64f8449019b5536e85cb8f03478
- summarize_v2.mjs7655 bytes
Evaluation tooling, not a result artifact.
89a36e9c8ca546f05893345557767b0690378d8bb337c7618f1f4f7426e8b7e2
The earlier blind holdout, for comparison
An earlier holdout, sealed August 9, 2026 against commit ef3b6a6f32b9, ran 120 cases under a different protocol with different thresholds and different strata. Its safety and run-integrity gate recorded PASS; its quality gate recorded BLOCK. The two runs are not directly comparable: the case sets, label definitions, and thresholds all differ, so the numbers below are context, not a trend.
A frozen retrieval evaluation at an earlier commit preceded both holdouts. It is recorded, with its hashes and with this earlier holdout’s, in data/learning-model/research-log/2026-08-09-chat-evidence-gate.md. It used a different case set and protocol, and its artifacts are not republished here.
- Cases
- 120
- Accuracy
- 87/120 = 0.7250
- Balanced accuracy
- 5220/7200 = 0.7250
- Provider precision
- 30/33 = 0.9091
- Provider recall
- 30/60 = 0.5000
- No-source recall
- 57/60 = 0.9500
- Generation calls
- 0
- Embedding calls
- 91
Published from the first run (4)
Four files from that run are in the repository and served here. The rest of what its manifest records (the dataset, the raw results, the metrics, and the tooling) was never committed here, so it is not claimed as published; the manifest still lists each one with its hash.
Verify this page
Every figure above is read from one generated file. The generator refuses to write when a published artifact’s SHA-256 disagrees with the hash the run recorded, when a recomputed rate disagrees with the rate the run recorded, or when a recomputed PASS or FAIL disagrees with the outcome the run recorded.
- Report hash, also served as the endpoint ETag
b0e6ff176f83f1890b036b9b4aab9c95c5412c61f14ad0ed7111b36a87fde3e6To check it: fetch the endpoint, take the data object from the response, drop goldenHash from it, re-serialize with the keys sorted at every level, and take the SHA-256 of that text. The endpoint also returns the same value as its ETag, so a conditional request answers 304.
- Generator
- scripts/build-ask-reliability-report.ts
- Schema version
- 1.1.0
Related: claim evidence, how the site is built, and the Lean verification record.