M2X-045: 30-question Phase 2 gate set over the tiron reference corpus

.gitignore

@@ -23,6 +23,15 @@ eval/labels/heldout/*
 !eval/labels/heldout/seal-manifest.json
 # the pre-M2X-030 path, kept ignored so an older checkout cannot leak a case through it
 eval/heldout/
+# the Phase 2 gate's answer key (M2X-045), sealed on the same terms as the held-out
+# labels. Questions are public -- the system has to be asked them -- but a builder who can
+# read the expected segments can tune retrieval until they come back, and a retrieval
+# metric tuned against its own answer key measures nothing.
+eval/rag/expected/*
+!eval/rag/expected/*.gpg
+!eval/rag/expected/seal-manifest.json
+!eval/rag/expected/.gitkeep
+!eval/rag/expected/README.md
 # labels in progress, before the dev/held-out split has been drawn. Every case is
 # plaintext here and ten of them are about to become the sealed set -- committing this
 # directory would put the held-out cases in git history, and no later encryption can

eval/rag/README.md

@@ -0,0 +1,121 @@
+# The Phase 2 gate instrument (M2X-045)
+
+Thirty questions — **20 single-meeting, 5 cross-meeting, 5 that must abstain** — written
+blind against the corpus, never against system output. The Friday gate (M2X-050) scores
+context precision ≥0.75, faithfulness ≥0.80 and citation accuracy ≥0.90 on this set;
+M2X-046 is the harness that computes them.
+
+## Two halves, held differently
+
+| | in git? | why |
+|---|---|---|
+| `questions.jsonl` | **yes**, plaintext | the system has to be asked the questions; hiding them would only stop the harness running |
+| `expected/<qid>.json` | **never** — git-ignored | a builder who can read the expected segments can tune retrieval until they come back |
+| `expected/<qid>.json.gpg` | yes | recoverability, so a fresh clone can score the gate |
+| `expected/seal-manifest.json` | yes | integrity — digests, so an edit is visible |
+
+Same tooling and the same reasoning as the held-out labels; see
+[`../labels/heldout/README.md`](../labels/heldout/README.md) for why both artefacts are
+needed and why the manifest, not the ciphertext, is the one that proves anything.
+
+```bash
+uv run python scripts/seal_heldout.py verify --dir eval/rag/expected   # no passphrase
+uv run python scripts/seal_heldout.py unseal --dir eval/rag/expected   # gate day
+uv run python scripts/validate_rag_questions.py                        # needs it unsealed
+```
+
+**The same caveat as the labels applies.** One operator writes the questions, the expected
+answers and the system under test, so the seal is a discipline rather than an enforced
+boundary — see [`../labels/README.md`](../labels/README.md) §"these labels are not
+independent". Unlike the extraction labels, though, these questions were written against
+*human-annotated reference transcripts nobody in this project authored*, so what a question
+asks about is at least not downstream of the system's own vocabulary.
+
+## The corpus is `eval/tiron/`, and that is a decision
+
+`data/` is git-ignored, so a fresh clone has no meeting transcripts. Every citation `m2x
+ask` has ever produced is a `§ heading` against this repository's own markdown — the
+`[meeting · speaker · mm:ss–mm:ss]` path, which is the entire product promise, has **never
+been exercised against a real meeting** (`docs/design/day4-ask.md` §"What is not verified
+here").
+
+The tiron reference transcripts are committed, carry human speaker turns and real
+timestamps, and already ground the Phase 1B labels. A gate written against them runs on
+every clone with no audio and no transcription provider, and exercises the meeting citation
+path on every question.
+
+Four meetings are covered — a mall aquarium project, a marketing meeting for outdoor sports
+equipment, a company retreat to Mexico, and a robotic-nurse feature with a contrarian Q&A.
+They share a speaker pool, which is deliberate: it makes the cross-meeting questions hard
+in the way cross-meeting questions are supposed to be hard.
+
+## Ground truth is turn ranges, never rendered citations
+
+An expected answer names `(meeting_id, first_turn, last_turn)` into the reference
+transcript.
+
+- **Not an `mm:ss` string** — that scores the renderer, not the retrieval.
+- **Not a chunk id** — that is a function of the chunking parameters and changes the moment
+  anyone tunes them, silently invalidating the whole set.
+
+Turn indices are positions in a committed file. A retrieved chunk records the segment range
+it covers, so citation accuracy is an **overlap** test between the two — which is the
+ticket's rule ("the cited segment id is among the ground-truth segments, or overlaps its
+time range"). Overlap rather than containment, because chunks are packed to a character
+budget and a correct citation routinely covers the answer plus its neighbours; demanding
+containment would fail a chunk for being the size the indexer chose.
+
+## How the questions were chosen
+
+Written by reading the four transcripts end to end and picking facts that are **stated**,
+not inferable. Recurring shapes, all deliberate:
+
+- **Answers split across interrupted turns.** `q05` — *"I'm gonna come back on Tuesday
+  with a"* / *(Sarah interrupts)* / *"follow up on this"*. A retrieval that returns only the
+  first turn has a day and no object.
+- **A claim and its qualifier in one span.** `q13` asks whether family can come; spouses
+  yes, children explicitly no, eight turns apart. Returning half is a confidently wrong
+  answer, which is worse than an abstention.
+- **Speaker distractors.** `q10` — Ron proposes Kim Kardashian, Rachel asks Sarah a
+  *different* Kim Kardashian question, and only then does Sarah answer. Matching on the
+  name alone lands on the wrong turn.
+
+### The must-abstain five are near misses on purpose
+
+Four of the five sit right on top of a strongly matching passage:
+
+| | why it is unanswerable |
+|---|---|
+| `q26` total aquarium cost | Rachel **asks for exactly this** and is never answered. Highly relevant context retrieves; the system still has to decline. |
+| `q27` name of the equipment company | The marketing meeting says "our products" throughout and never names the company. |
+| `q28` Nursing Bot's annual revenue | The company is named and described at length; the number is absent. |
+| `q29` which hotel the retreat is booked with | The retreat centre's studio, restaurant and pool are all described; it is never named. |
+| `q30` 2026 cricket world cup | The control — no lexical overlap with anything. If this one does not abstain, nothing else in the bucket is interpretable. |
+
+A must-abstain set of `q30`s would be trivially passed by the distance threshold alone and
+would measure nothing about the model's judgement. These four force the *model* to abstain
+on passages the retriever was right to return.
+
+## The re-read pass
+
+The ticket asks for a second pass hours later: *can each answerable question really be
+answered from the recorded segment?* Both halves were done.
+
+**Mechanical** — `scripts/validate_rag_questions.py`. Ids unique and paired across both
+halves, the 20/5/5 mix, answerable questions carrying both evidence and a gist, unanswerable
+ones carrying neither, cross-meeting questions genuinely spanning ≥2 meetings, and every
+cited turn resolving against the reference. Exits non-zero on any problem.
+
+**By reading** — every cited span printed back beside its question and checked to contain
+the answer. One correction resulted: `q10`'s span ended one turn past Sarah's reply.
+
+The judgement itself is recorded per question in the sealed `notes` field, so at the gate an
+adjudicator can see *why* a span was chosen and where the call was close.
+
+## Known limitation
+
+**`gist` is prose and is graded by a judge, not by string equality.** Two correct answers to
+the same question rarely share wording — the same finding that replaced token-set F1 with
+embedding cosine in M2X-036. That means the answer-correctness half of this set inherits
+whatever the judge model's biases are, and only citation accuracy and abstention are
+mechanically checkable. Read the gate numbers with that split in mind.

eval/rag/expected/seal-manifest.json

@@ -0,0 +1,158 @@
+{
+  "schema": 1,
+  "sealed_at": "2026-08-13",
+  "cipher": "AES256",
+  "case_count": 30,
+  "cases": [
+    {
+      "case_id": "q01",
+      "file": "q01.json",
+      "sha256": "003b54d21c4b0b6bbf530395d13e39555d0c4fc69270689945494610226e02a8"
+    },
+    {
+      "case_id": "q02",
+      "file": "q02.json",
+      "sha256": "ca5791a898534fd921a13d48ed6f7507a78e36ba62574446231e5d29c07924dc"
+    },
+    {
+      "case_id": "q03",
+      "file": "q03.json",
+      "sha256": "2924e9c8530759855128b6437ee1bcb35c40a8a5ea2ea4150dad6c3251d1ddca"
+    },
+    {
+      "case_id": "q04",
+      "file": "q04.json",
+      "sha256": "ae969b53081cede6e389256f3373e703545cab4980fb3062f4c57f38a1398c16"
+    },
+    {
+      "case_id": "q05",
+      "file": "q05.json",
+      "sha256": "4c4548a9fd2a3566f97be828ffeff90ab9120feddcc9eba800e7c9ca192bdfde"
+    },
+    {
+      "case_id": "q06",
+      "file": "q06.json",
+      "sha256": "30d3bce2c4334c2a569e05b7d369a6cad23738e8f29aef6c439599139b992e32"
+    },
+    {
+      "case_id": "q07",
+      "file": "q07.json",
+      "sha256": "269b445389eee860d4502e2e72a5c6613c1e85ea2e1500cf8799f4b908320521"
+    },
+    {
+      "case_id": "q08",
+      "file": "q08.json",
+      "sha256": "2f5910533e00dddc13e7d34c98bfab62406f6c2f2a7c2cf4247cc462c38311b0"
+    },
+    {
+      "case_id": "q09",
+      "file": "q09.json",
+      "sha256": "d09de7791c37dca654f8059abdb68f9494fea86eee78625bdaaafeb9a535e11b"
+    },
+    {
+      "case_id": "q10",
+      "file": "q10.json",
+      "sha256": "b0a88ee664f3d5f7a77f5598c0681f384a7add914bfec91ef5131d2fdfd00f9f"
+    },
+    {
+      "case_id": "q11",
+      "file": "q11.json",
+      "sha256": "95ddfccc1981419a9953bc4484adde81f5320f947b0ad9eda014afef15322ce3"
+    },
+    {
+      "case_id": "q12",
+      "file": "q12.json",
+      "sha256": "29773d15c0f6ca5a5c02ac43830547b20927f0bfe689163e5a4b2077f57bd8ed"
+    },
+    {
+      "case_id": "q13",
+      "file": "q13.json",
+      "sha256": "7e897183a0dcdba2b999b95eadf6e012cf594782c2339419e0122982ebabd3b5"
+    },
+    {
+      "case_id": "q14",
+      "file": "q14.json",
+      "sha256": "e359bb77c72e59c9430bcc066804c6acbf2464ee5645ebd9267763f8819c7336"
+    },
+    {
+      "case_id": "q15",
+      "file": "q15.json",
+      "sha256": "3ca1fb198e33c7ad109e9c03fd9d91eb078ffd788eed8eec1ccb5a878a9fc6db"
+    },
+    {
+      "case_id": "q16",
+      "file": "q16.json",
+      "sha256": "e8e682b5403373b8eb771f0b8bb3880fd35054df6f5014339d30cfebb0fe2641"
+    },
+    {
+      "case_id": "q17",
+      "file": "q17.json",
+      "sha256": "1d0e08e8659c2e4f8f95f128e27314a2f7be85b57fc847ef0f57fa1fedfd2c7c"
+    },
+    {
+      "case_id": "q18",
+      "file": "q18.json",
+      "sha256": "74a1c9c330854d1281bf7381262c6c9f5505469df86de342a687d4856e5f7e86"
+    },
+    {
+      "case_id": "q19",
+      "file": "q19.json",
+      "sha256": "713f521a15abf8dc89bc0fc1e296c8d95919508b13abf66166f28238556d49dc"
+    },
+    {
+      "case_id": "q20",
+      "file": "q20.json",
+      "sha256": "a29f59c2509a26ec503fe32901fb62e62529663de04b26a7773fc12d7aca85f1"
+    },
+    {
+      "case_id": "q21",
+      "file": "q21.json",
+      "sha256": "00dcacb09a03241e857dbe6f554c3ae6a94aff8e112c63b8f8d2cc85ac90da45"
+    },
+    {
+      "case_id": "q22",
+      "file": "q22.json",
+      "sha256": "00146cd6d216d1732ef4b6ec6e23b8fd09f755ae17300bc45ce14e82266c03f4"
+    },
+    {
+      "case_id": "q23",
+      "file": "q23.json",
+      "sha256": "0351f9425bfbe8e6e4fa20b8796788cfe8dbb34bfe486391cddcc358faf39828"
+    },
+    {
+      "case_id": "q24",
+      "file": "q24.json",
+      "sha256": "b4ecea86c4eebeb4f06b84c5db9605bdce21f19fb22eb76b4da0f1fda70907bb"
+    },
+    {
+      "case_id": "q25",
+      "file": "q25.json",
+      "sha256": "e9adc19ba1b4b5b58ed5791c9153bd799a202c178974d164dbb22fb53e759449"
+    },
+    {
+      "case_id": "q26",
+      "file": "q26.json",
+      "sha256": "a0610d73c5b9f610c111c4a7e77baf6c2f4a87901c859c754193f6cdb6b36950"
+    },
+    {
+      "case_id": "q27",
+      "file": "q27.json",
+      "sha256": "226d9de8209d77c6e6140e34b326c5cf450ecf8b28c9cf12df829765ac9fd186"
+    },
+    {
+      "case_id": "q28",
+      "file": "q28.json",
+      "sha256": "93d9f6ca1728f6952d45e74e9d3063c1490a5a7cfd3588e5c11907a5d79da100"
+    },
+    {
+      "case_id": "q29",
+      "file": "q29.json",
+      "sha256": "50f398b8819e440b237e2a78766165a6d43044eb439d94fe6c1173d5c0303713"
+    },
+    {
+      "case_id": "q30",
+      "file": "q30.json",
+      "sha256": "d1aa4ba18a17676ee872c2f2f1b773adfd33dd6936ebed99795186022cc1ee91"
+    }
+  ]
+}

eval/rag/questions.jsonl

@@ -0,0 +1,30 @@
+{"question_id": "q01", "kind": "single_meeting", "question": "What is holding up the aquarium according to the legal department?"}
+{"question_id": "q02", "kind": "single_meeting", "question": "What did the marketing department buy to draw shoppers up to the third floor?"}
+{"question_id": "q03", "kind": "single_meeting", "question": "Which role has the mall still not hired for the third floor?"}
+{"question_id": "q04", "kind": "single_meeting", "question": "What did Beth report about the numbers this week?"}
+{"question_id": "q05", "kind": "single_meeting", "question": "When did John say he would come back about the legal problem?"}
+{"question_id": "q06", "kind": "single_meeting", "question": "Which overseas market did Beth want to target, and through what channel?"}
+{"question_id": "q07", "kind": "single_meeting", "question": "Why did Sarah argue against focusing on Japan?"}
+{"question_id": "q08", "kind": "single_meeting", "question": "What did Sarah ask Donald to produce, and by when?"}
+{"question_id": "q09", "kind": "single_meeting", "question": "Which part of the US did Donald name as the best target, and why?"}
+{"question_id": "q10", "kind": "single_meeting", "question": "Which celebrity did Ron suggest as an influencer, and how did Sarah respond?"}
+{"question_id": "q11", "kind": "single_meeting", "question": "Where is the company retreat being held?"}
+{"question_id": "q12", "kind": "single_meeting", "question": "What did Sally ask to add to the retreat programme?"}
+{"question_id": "q13", "kind": "single_meeting", "question": "Can employees bring family to the retreat?"}
+{"question_id": "q14", "kind": "single_meeting", "question": "Where does the trip stop on the way to Mexico, and for how long?"}
+{"question_id": "q15", "kind": "single_meeting", "question": "What did Jenny say when Donald suggested visiting a theme park in Florida?"}
+{"question_id": "q16", "kind": "single_meeting", "question": "Who developed the robotic nurse and how tall is it?"}
+{"question_id": "q17", "kind": "single_meeting", "question": "What does the piece claim about the robotic nurse's availability compared with human nurses?"}
+{"question_id": "q18", "kind": "single_meeting", "question": "How does Nursing Bot Incorporated address the safety and privacy concerns?"}
+{"question_id": "q19", "kind": "single_meeting", "question": "What is the main objection raised in the contrarian Q&A about robots in healthcare?"}
+{"question_id": "q20", "kind": "single_meeting", "question": "What does the contrarian say about the cost savings argument for robots?"}
+{"question_id": "q21", "kind": "cross_meeting", "question": "What does Donald offer or ask for in the marketing meeting and in the retreat meeting?"}
+{"question_id": "q22", "kind": "cross_meeting", "question": "What food plans are discussed in the mall meeting and in the retreat meeting?"}
+{"question_id": "q23", "kind": "cross_meeting", "question": "What does Sarah ask the group to do in each of the two meetings she chairs?"}
+{"question_id": "q24", "kind": "cross_meeting", "question": "Which cost or budget concerns are raised across the mall meeting and the marketing meeting?"}
+{"question_id": "q25", "kind": "cross_meeting", "question": "What physical or outdoor activities come up in the marketing meeting and the retreat meeting?"}
+{"question_id": "q26", "kind": "unanswerable", "question": "How much did the third-floor aquarium project cost in total?"}
+{"question_id": "q27", "kind": "unanswerable", "question": "What is the name of the company whose sports equipment is being marketed?"}
+{"question_id": "q28", "kind": "unanswerable", "question": "What is Nursing Bot Incorporated's annual revenue?"}
+{"question_id": "q29", "kind": "unanswerable", "question": "Which hotel is the Mexico retreat booked with?"}
+{"question_id": "q30", "kind": "unanswerable", "question": "Who won the 2026 cricket world cup?"}