M2X-045: 30-question Phase 2 gate set over the tiron reference corpus

[yashpancholi09]

The Phase 2 gate instrument: 20 single-meeting / 5 cross-meeting / 5 must-abstain, written by reading four tiron reference transcripts end to end and picking facts that are stated, not inferable.

Stacked on feature/m2x-042-rag-design.

The corpus choice is the load-bearing decision

data/ is git-ignored, so a fresh clone has no meeting transcripts, and every citation m2x ask has ever produced is a § heading against this repo's own markdown. The [meeting · speaker · mm:ss–mm:ss] path — the entire product promise — has never been exercised against a real meeting.

eval/tiron/ is committed, carries human speaker turns and real timestamps, and already grounds the Phase 1B labels. A gate written against it runs on every clone with no audio and no transcription provider.

Four meetings, sharing a speaker pool deliberately — it makes the cross-meeting questions hard in the way cross-meeting questions are supposed to be hard.

Ground truth is turn ranges, never rendered citations

An expected answer names (meeting_id, first_turn, last_turn). Not an mm:ss string — that scores the renderer rather than the retrieval. Not a chunk id — that is a function of the chunking parameters and would silently invalidate the set the moment anyone tunes them.

Citation accuracy is therefore an overlap test, which is the ticket's rule. Overlap rather than containment: chunks are packed to a character budget and a correct citation routinely covers the answer plus its neighbours, so containment would fail a chunk for being the size the indexer chose.

The seal

Questions are committed plaintext because the system must be asked them. Expected answers are sealed with the M2X-041 tooling, because a builder who can read the expected segments can tune retrieval until they come back. A test asserts questions.jsonl never contains a gist or an evidence span, so the seal cannot become decorative by accident.

The must-abstain five are near misses on purpose

Four of five sit on top of a strongly matching passage. q26 asks the total aquarium cost — which a participant explicitly asks for in the meeting and is never answered, so retrieval returns highly relevant context and the system still has to decline.

A bucket made only of q30-style questions ("who won the 2026 cricket world cup") would be passed by the distance threshold alone and would measure nothing about the model's judgement.

The re-read pass, both ways

Mechanical — scripts/validate_rag_questions.py: ids unique and paired across both halves, the 20/5/5 mix, answerable questions carrying evidence and a gist, unanswerable ones carrying neither, cross-meeting questions genuinely spanning 2+ meetings, every cited turn resolving. Exits non-zero on any problem.

By reading — every cited span printed back beside its question. One correction resulted: q10's span ended one turn past the reply it was citing.

Known limitation

gist is prose graded by a judge, so answer correctness inherits the judge's biases. Only citation accuracy and abstention are mechanically checkable. Recorded in eval/rag/README.md.

585 tests green (19 new).