Account for every conformance fixture in both SPEC rosters, and gate them

[jeremy]

Three rosters in this repo claim to account for every conformance fixture. Two of them had drifted, and nothing checked either.

claim	where	state on origin/main before this PR
every fixture has an owning spec section	SPEC §19 Test Categories table	missing documents_write.json, uploads_write.json
every fixture maps to a primary section	SPEC Appendix D	missing documents_write.json, search.json

The convention was missed by the last three fixture-adding commits, in three different ways. dee221c85 (#601) added documents_write.json and updated neither table. b238e5ed4 (#683) added uploads_write.json with four Appendix D rows and no §19 row. #726 added search.json's §19 row and missed Appendix D. Two half-applications in opposite directions is much stronger evidence for a gate than either gap alone: it is not a person being careless, it is a rule nobody can see. CONTRIBUTING.md told contributors to add conformance tests and mentioned neither table — this PR fixes that too.

No adversary here, and asking who the attacker is would mislead: this is the accident-and-regression class, which is what a cheap merge-blocking check is legitimately for.

It reuses existing machinery

sync-doc-constants.rb already does table-completeness checking — @assertion-types wraps §19's assertion-type table in a block marker and set-compares it against conformance/schema.json. The two fixture rosters are the same shape one level out, so they become two more block kinds: no new script, no new Makefile target, no new CI step. make doc-constants-check already runs the gate and its self-test and is already in check-targets and spec-gates.

The two invariants differ because the artifacts differ. §19's table is a bijection, so all of it is asserted — one row per fixture, both directions, and category slug == basename with _ as - (verified across all 22 rows, zero violations). Appendix D's rows are curated summaries that deliberately bundle several cases (uploads_write.json legitimately has four), so it gets coverage only, and a self-test case pins that difference by asserting several rows for one fixture still passes.

Both tables also reject a row whose attribution cell is blank, and a §N that resolves to no ## §N. heading — the latter catching a reference that resolved when written and stopped resolving when a section was renumbered, which a reviewer of the same PR cannot see. A row with no section reference is still accepted: rejecting it needs a carve-out for live-my-surface.json's external-governance attribution, and the carve-out list is the part that grows.

Neither table is writable. --write only ever touched line spans, and a row here carries an attribution only the fixture's author can make.

Verification

• Both checks reject origin/main as it stood, and pass only after the first commit's rows land (REAL_EXIT=1, output quoted in that commit).
• Every guard was shown to fail its self-test when removed — one mutation per guard, restored by copy.
• Vacuity guarded on both sides, and it earns it: a gutted table is caught by the set comparison alone, but both sides empty at once exits 0 without the explicit guards. That is why they are stated rather than left to emerge, and there is a case for it.
• Each commit is green standalone (the markers ship with the gate that reads them, so the backfill commit does not reference a kind the inventory has not yet declared).
• make doc-constants-check green under LC_ALL=C.

Scope: #602 is deliberately not here

This PR originally also carried a source-text parser for the six conformance runners, to answer #602 ("is any fixture executed by nothing?") and to hold SPEC §19's Zero-Skip roster to it. That half has been split out. Its rule count grew across nine commits and four review rounds while the call sites it covers never moved — the state AGENTS.md describes as "reassess the instrument, not write a sixth selector".

The question is right; the instrument was wrong. Ground truth for "what does this runner skip" is runtime, not source text, and every runner already reports Passed/Failed/Skipped/Total — so passed + failed + skipped == mock case count, per runner is a single arithmetic identity that subsumes most of what the parser did by hand: nested fixtures, mode typos, empty fixtures, dropped cases, result.skipped, derived tables, spreads and percent-strings alike.

The parser work is preserved on conformance-skips-parser-archive and #602 stays open. Note for whoever picks it up: five runners print SKIP: <name>; TypeScript does not — it calls it.skip(), so that lane needs vitest's JSON reporter rather than a line match.

Closes nothing on its own; #602 remains open.

[github-actions]

Sensitive Change Detection (shadow mode)

This PR modifies control-plane files:

• .github/workflows/test.yml

Shadow mode — this check is informational only. When activated, changes to these paths will require approval from a maintainer.

[Copilot]

Pull request overview

Adds automated gates to keep conformance fixture documentation complete and ensure every mock case remains executable by at least one SDK runner.

Changes:

• Gates fixture rosters in SPEC.md against tracked fixtures.
• Adds runner skip extraction and fixture-execution validation.
• Adds self-tests, CI wiring, and contributor guidance.

Tip

If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
spec/doc-constants.json	Registers fixture roster markers.
SPEC.md	Completes and documents fixture rosters.
scripts/test-doc-constants.rb	Tests roster drift detection.
scripts/test-check-fixture-execution.rb	Tests execution-gate failure modes.
scripts/sync-doc-constants.rb	Implements fixture roster checks.
scripts/conformance_skips.rb	Extracts runner exclusions.
scripts/check-fixture-execution	Detects universally skipped cases.
Makefile	Adds execution gate targets.
CONTRIBUTING.md	Documents fixture roster requirements.
.github/workflows/test.yml	Runs new gates in CI.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1fbfb340dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Makefile:1209

• This count is inaccurate: conformance_skips.rb reads six name-keyed literal tables, one for each runner, plus the Kotlin and Swift tag branches.

# it stayed green in both. This reads the six runners' skip mechanisms — five
# literal tables plus Kotlin's and Swift's whole-case `link-header` tag branch,
# which the other four use to suppress one ASSERTION rather than the case — and

scripts/check-fixture-execution:209

• The roster comparison accepts stale skip keys. If a fixture case is renamed or deleted while its table entry and SPEC bullet remain, both sets still agree even though a default run can never report that skip; this recreates the “wrong extraction and stale roster agree” false-green described above. Validate every primary skip-table key against the loaded mock-case names (and add a self-test for an unknown key) before comparing the roster.

  errors = check_companion_tables(tables) + check_roster(ROOT, tables) +
           check_modes(cases) + check_execution(cases, exclusions)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (5)

scripts/sync-doc-constants.rb:765

• This only requires the category and filename cells, so a row with no owning section still satisfies the new bijection and lets doc-constants-check vouch for a fixture that no spec section owns. Require a present, non-empty third cell (and add that malformed-row case to the self-test).

    if cells.length < 2
      errors << "#{span.file}:#{line_no}: category row has #{cells.length} cell(s); the shape is " \
                "| Category | `file.json` | Owning Spec Section(s) |"
      next
    end

scripts/conformance_skips.rb:250

• Ending at the first newline after any balanced brackets can silently parse a valid multiline Swift declaration as empty. For example, let temporarySkips: [String: String] = followed by the dictionary on the next line sets entered from the type annotation, reaches depth zero before the initializer, and returns no keys. That can hide a newly added skip. Start delimiter tracking at the initializer expression (or otherwise distinguish annotation brackets), and cover this Swift form in the self-test.

      if char == "\n"
        return [strings, :closed] if entered && depth.zero?

scripts/sync-doc-constants.rb:827

• The Appendix D coverage check accepts any row whose first cell is a fixture filename; even | alpha.json | counts as mapping that fixture to a primary section. Validate the non-empty summary and primary-section cells before adding the file to covered, otherwise the gate can pass while the documented mapping is absent.

    if cells.empty?
      errors << "#{span.file}:#{line_no}: mapping row has no cells; the shape is " \
                "| `file.json` | Test name | Primary section |"
      next
    end

scripts/check-fixture-execution:119

• Using || treats an explicitly malformed JSON value such as "mode": false or null as if the field were omitted, so this validator accepts it as mock instead of reporting an unrecognized mode. Default only when the key is absent. Also, the comment's "mdoe": "mock" example is not an all-runner exclusion: because mode is absent, every runner defaults it to mock.

def check_modes(cases)
  cases.filter_map do |test_case|
    mode = test_case["mode"] || ConformanceSkips::DEFAULT_MODE
    next if ConformanceSkips::KNOWN_MODES.include?(mode)

scripts/conformance_skips.rb:451

• This || [] bypasses the non-array guard for falsey JSON values: "tags": false or null is silently treated as no tags, despite the stated fail-closed behavior. Default only when the member is absent so every explicitly malformed value reaches the type check; add a false/null case alongside the existing object case.

        cases.each do |test_case|
          case_tags = test_case["tags"] || []
          # A non-array `tags` would answer include? in whatever way its own
          # class defines — a Hash by key, a String by substring — and the wrong
          # answer here is silent under-exclusion, which reads as a case some
          # runner still executes. conformance-fixtures-check validates the
          # fixture schema, but a gate that credits malformed input because
          # another gate usually catches it is crediting what it did not read.
          unless case_tags.is_a?(Array)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/conformance_skips.rb:476

• This audit only runs for runners that already have a registered tag branch. Because Go, Python, Ruby, and TypeScript have empty tag_branches, reintroducing #573's whole-case link-header checks in those runners would not be observed; the gate would still think those four execute the case and would miss the all-six exclusion it is intended to prevent. The self-test also models those four exclusions as name-table entries rather than actual tag branches. Please inventory/classify tag inspections for all six runners and add a mutation that introduces a whole-case branch in one of these four runners.

      runner.tag_branches.group_by(&:file).each do |file, branches|
        accessor = branches.first.accessor
        mentions = read(root, file).lines.count { |line| line.include?(accessor) }
        next if mentions == branches.length

scripts/check-fixture-execution:154

• Using || treats an explicitly present false or null mode as if the field were omitted. Those are not recognized modes, but this validator reclassifies them as mock, allowing this gate to pass unless a separate schema check happens to run. Distinguish an absent key from a present invalid value, and add this case to the self-test.

    mode = test_case["mode"] || ConformanceSkips::DEFAULT_MODE

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 02d6b4ebe0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/conformance_skips.rb:217

• The anchor guard is not fail-closed when the declaration keeps its name but stops being a standalone literal. For example, SKIPS = SOME_SET leaves entered false, so scan continues into the following declaration (often the companion reason map) and can return those keys; SKIPS = [] + EXTRA_SKIPS returns only the literal portion. Either reshape can under-report actual exclusions and let an all-runner skip pass. Bound parsing to the single assignment and reject aliases/composed expressions, with a mutation test for this case.

    offset = lines[0, hits.first].sum(&:length) + assignment
    strings, state = scan(source, offset, table.comment)

scripts/sync-doc-constants.rb:693

• This splits escaped Markdown pipes too. A valid Appendix row whose Test name is A \| B and whose Primary section is blank is parsed as if B were the third cell, so the checker accepts coverage without an actual section mapping. Split only on unescaped pipes; a regression case with an escaped pipe in the Test name would pin the behavior.

  parts = line.split("|")

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 046b133b8c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

scripts/conformance_skips.rb:510

• This regex cannot parse a rostered case name containing an escaped quote: - "a \"quoted\" case" — ... is truncated to a \. The new escaped-quote self-test actually expects skipped but not rostered, so it confirms extraction from the runner while leaving such a legitimate skip impossible to roster; the gate can never become green unless the fixture is renamed. Parse the quoted literal with escape awareness and make that self-test pass end-to-end.

    end

    body = lines[(hits.first + 1)..].take_while { |line| !line.start_with?("## ", "---") }.join
    split = body.split(/^\*\*([A-Za-z]+)\*\*/)[1..].to_a.each_slice(2).to_a
    blocks = split.to_h

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5ee9b4f5dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jeremy]

Rebased onto main now that #742 has merged (the census half of this PR's original split), and took the two findings in the last round that describe real interactions.

Fixed in 865bacc:

• Block bodies were dropped from the pin scan along with line spans. The two are not alike: the writer rewrites line spans only, and the block checkers read nothing but the | rows, so a sentence parked inside a roster block survives both. That let an unmarked current-pin claim hide inside a table — the gate's own span bookkeeping concealing the claim class it exists to catch.
• Colliding category slugs. The table tallied files, so foo_bar.json and foo-bar.json both derive foo-bar, both rows pass the per-row slug rule, and the table quietly stops being the bijection its heading asserts. Now tallied on the derived slug.

Both self-test cases were shown to fail against the un-fixed gate first.

Declined, deliberately: the two Markdown-spelling findings (three-hyphen separator cells, backslash parity before a pipe). Both are true. But this is the eleventh comment on sync-doc-constants.rb, and that shape — "another Markdown spelling the parser does not recognise" — is what got the source-text parser withdrawn from this same PR after nine commits of rules covered exactly the call sites it started with. The failure mode is a colleague mistyping a roster row; the consequence is a table that renders wrong in SPEC.md, which a human reviewing that diff sees immediately. If one ever does slip through unnoticed, the answer is to stop hand-parsing GFM in Ruby, not to widen the regex again. Reasoning is in each thread so it does not get re-litigated next round.

make doc-constants-check green (13 marked spans, 22 fixtures). Full make check running on Linux.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

scripts/sync-doc-constants.rb:932

• Require exactly three cells here as well. For example, an unescaped pipe in a test summary shifts the real Primary section into a fourth cell; this code treats the preceding fragment as the section, ignores the actual §N, and still counts the fixture as covered. Rejecting extra columns makes malformed rows fail closed.

    if cells.length < 3

SPEC.md:2195

• This now documents a gate that is not present: the PR deliberately split check-fixture-execution out, the repository has no such script or Make target, and #602 remains open. The following roster-validation paragraph is likewise describing the removed gate. Restore the pending-gap wording so SPEC does not promise coverage CI cannot provide.

layer down. `make check-fixture-execution` (#602) is what detects it now: it
reads the six runners' skip mechanisms and fails when a mock case is excluded
everywhere. Maximum overlap today is 2 of 6, so that gate is green on arrival
and its self-test — which crafts the all-six state — is what proves it can say
no.

scripts/sync-doc-constants.rb:827

• Require exactly three cells, not merely at least three. With a raw pipe in this attribution, a four-cell row is accepted and only cells[2] is validated; a later invalid reference such as §99 is silently ignored even though the gate claims to validate every section reference. Escaped pipes still remain valid cell content.

This issue also appears on line 932 of the same file.

    if cells.length < 3

scripts/sync-doc-constants.rb:795

• This explanation is incorrect for the both-empty case covered by the new self-test: when both the tracked-fixture set and table rows are empty, the bidirectional set comparison passes, so this guard does add coverage rather than only improving the message. State that distinction explicitly; otherwise a maintainer could remove the guard based on this comment and reintroduce the vacuous pass.

# Both directions already fail loudly on their own — an empty table makes every
# fixture `missing`, an empty fixture list makes every row `extra` — so this
# buys no coverage. It buys the message: a regex that stopped matching would
# otherwise be reported as 22 individually plausible drift findings instead of
# the one thing that is actually wrong. House standard since

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 865bacccb6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jeremy]

Round two at 9d70731. One of these was a genuine miss on my part.

Codex P1 — SPEC promised a gate that does not exist. make check-fixture-execution was the source-text parser split out of this PR to conformance-skips-parser-archive; the prose describing it stayed behind. Verified: make -n check-fixture-execution exits "No rule to make target". That is precisely the defect class this PR family exists to prevent, sitting in the PR that adds the gates. Copilot flagged it in its suppressed block too. SPEC §19 now states what is true — the merged case census (#742) catches a case executed by no runner for a mechanical reason, and explicitly does not catch the deliberate all-six exclusion, because each census counts its own skip and stays green. #602 stays open with the reason recorded, and #736 now points at the same cross-runner manifest.

Copilot suppressed — roster_vacuity's comment invited deleting a live guard. It claimed the guard "buys no coverage". True when one side is empty; false when both are, where missing and extra are both empty, the comparison is trivially satisfied, and this guard is the only thing refusing the vacuous pass. A committed self-test covers exactly that case. Comment corrected to name the distinction and say which test dies if the guard goes.

Declined: roster tables inside code fences. Fenced rows are still parsed and still checked, so it cannot let a wrong roster pass — it produces a roster that renders as a code block, a display defect any reader sees. That makes it the third Markdown-spelling finding on this file after two were declined this round, which is the signal rather than the task. Reasoning is in the thread.

make doc-constants-check green; make check passed on Linux at 865bacc and is re-running for this head.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

CONTRIBUTING.md:238

• This says the convention “is the only place” it was documented, but before this PR the checklist mentioned neither roster, while after this PR the convention is also stated in SPEC.md and the gate policy. Rephrase this as a historical absence so the contributor guidance matches the PR’s rationale.

   `make doc-constants-check` fails until both exist — it is the only place
   this convention was written down, which is why three fixtures in a row
   landed with one row, the other, or neither.

scripts/sync-doc-constants.rb:949

• Rows with more than three cells are accepted. An unescaped pipe in a test summary (for example, | alpha.json | supports A | B | §1 |) shifts B into cells[2] and the real section into an ignored fourth cell; because non-§ attributions are intentionally allowed, the gate then passes even though the rendered Primary section is not the claimed section. Require exactly three parsed cells so malformed rows fail closed.

    if cells.length < 3
      errors << "#{span.file}:#{line_no}: mapping row has #{cells.length} cell(s); the shape is " \
                "| `file.json` | Test name | Primary section |"
      next

[jeremy]

@codex review

[jeremy]

Round three at 46bff5c, taking both of Copilot's suppressed comments.

Required exactly three cells per roster row. Copilot raised this three times across two rounds and I had been treating it as part of the declined Markdown-spelling class. That was the wrong call, and the distinction is worth stating: the declined findings ask the splitter to understand more Markdown (separator widths, backslash parity); this one asks it to refuse what it does not understand. That is the direction this file already argues for — "a row the parser cannot see is a row it silently vouches for" — and it closes the pipe class as a class rather than one spelling at a time.

The consequence was real, not cosmetic: a raw pipe in an attribution shifts the real section into a fourth cell and leaves the fragment before it in cells[2], where non-§ attributions are legitimately allowed. So the gate validated the wrong cell and a §99 in the actual section position was never checked — on a gate whose claim is that it validates every section reference. Both tables had it. Self-test cases added for each, shown to fail against < 3 first.

Worth noting this also subsumes the backslash-parity finding I declined earlier: a row whose pipes split unexpectedly now fails on its cell count instead of being silently mis-parsed. Loud-wrong beats quiet-wrong, which was the actual objection.

CONTRIBUTING.md claimed the checklist item "is the only place this convention was written down" — self-falsifying, since this PR also states it in SPEC §19 and the gate. Rephrased as the historical absence it describes.

make doc-constants-check green; make check passed on Linux at 9d70731 and CI was green there (26 runs, 0 failures). Re-running both for this head.

[jeremy]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 46bff5c37b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/sync-doc-constants.rb:741

• Valid Markdown table rows do not require a leading pipe, but this filter silently drops them. For example, adding gamma | gamma.json | §1 to this marked table renders as a roster row while the gate still passes, so an untracked or duplicate entry can evade the bidirectional check. Parse optional-leading-pipe rows, or explicitly reject any table row shape the gate does not support instead of filtering it out.

  numbered = span.lines.each_with_index
                 .map { |line, index| [span.line_no + index, line.strip] }
                 .select { |_, line| line.start_with?("|") }

scripts/sync-doc-constants.rb:391

• This changes prose to retain block-span bodies, but scan_file's return-contract comment above still says prose_lines excludes content inside any marked span. Update that contract to distinguish line spans from block spans so it describes the data callers now receive.

  spans.each do |s|
    next if s.block

    (s.line_no...(s.line_no + s.lines.length)).each { |n| covered << n }