Fixtures lack captured-provenance records: 82 of 125 across byte-integrity-covered directories

[lamemustafa]

What

AGENTS.md P1 states the rule without qualification:

Fixtures must be captured from a real instance, never authored by hand. A hand-written fixture encodes an assumption and then defends it.

A provenance check written against the four byte-integrity-covered fixture directories reports 82 of 125 fixtures carry no provenance record. This is not a single-PR problem; it is the repo-wide state.

How it was measured

scripts/check-fixture-provenance.mjs (added alongside the new deterministic CI gates) scans the fixture directories already covered by check-fixture-byte-integrity.mjs and requires each fixture to carry or be named in a provenance record. It currently runs as REPORTING, not blocking, precisely because it fails today.

Note the check is a documented, bounded heuristic — it looks for a provenance mention, not a proof of capture. A fixture with a provenance line claiming a live capture that never happened would still pass. Treat 82 as a floor on the problem, not an exact count of hand-authored fixtures.

Why it matters

This is the highest-severity defect class in the review corpus: across 880 findings, "hand-authored fixture standing in for a captured one" was 86% P1 — the highest P1 share of any class except privacy. AGENTS.md P1 exists because the read pipeline reached ~15,500 lines and 445 passing tests against a simulator this repo wrote for itself: "it verified that Bridge agreed with Bridge."

Known constraint

Some of these cannot be captured today. #45, #213 and #293 are each blocked on unavailable live-Tally environments. So closing this fully is gated on lab access, not only on effort — which is an argument for recording provenance honestly (including authored test input) rather than for leaving the field absent.

Acceptance

• Every fixture in the covered directories carries a provenance record stating how it was produced, using honest values — a mixed record such as {catalogue: captured, source_xml: authored test input, binding_dto: derived by the production parser and binder} is a valid and useful answer.
• Fixtures that are authored and cannot yet be captured are listed explicitly with the blocker.
• check-fixture-provenance.mjs is promoted from REPORTING to BLOCKING once the backlog is cleared.

Origin

Raised while dispositioning the P1 review thread on #331 (scripts/fixtures/source-draft-capture-bindings.json). That specific fixture predates #331 and #331 in fact adds a provenance block to it; the finding was accepted as follow-up against this issue rather than blocking the wrong-ledger binding fix in #329.

[lamemustafa]

Two properties of the gate that shape how this backlog should be cleared

Verified against scripts/check-fixture-provenance.mjs on master.

1. It matches the fixture's basename literally

check-fixture-provenance.mjs:151-158 takes each fixture's basename and does a substring match (with word-boundary-ish punctuation guards) across the Markdown under the fixtures tree. So a PROVENANCE.md that describes a fixture without naming the file counts as undocumented.

That is a real limitation to know before clearing 82 entries: a well-written provenance paragraph that says where a capture came from, but never spells out education-small-v1.json, will still be flagged. Found independently while documenting a new fixture in another lane — the paragraph was accurate and the gate rejected it.

2. A prose mention is accepted; only a table row is verified

This is the more important half. The script parses table rows of the form:

| name | bytes | sha256 |

into declaredHashes, and then checks the declared hash against the fixture's actual bytes. A prose mention satisfies the "documented" requirement but asserts nothing checkable.

So the two forms are not equivalent:

form	passes the gate	detects a silent swap
prose naming the file	yes	no
| name | bytes | sha256 | row	yes	yes

Recommended acceptance change

The acceptance criteria above should prefer the table form rather than any provenance mention. Clearing 82 fixtures with prose would satisfy this issue while leaving the thing the gate exists to catch — a fixture quietly replaced with different bytes — undetected.

That matters given why this issue exists: AGENTS.md P1 was written because ~15,500 lines and 445 passing tests ran against a simulator this repo wrote for itself. A provenance line that cannot be checked is the same category of evidence.

Worth noting the gate still cannot verify that a claim of live capture is true — a table row with a correct hash proves the bytes have not changed since they were recorded, not that they came from a real Tally. That part stays a human judgement, and is why the check is REPORTING rather than blocking today.

[lamemustafa]

The gate could not report this issue's own number, which is worth fixing before anyone tries to burn the backlog down.

check-fixture-provenance.mjs capped collection and display at the same MAX_REPORTED = 25, so failures.length could never exceed 25 and the reporting line's failures.length > shown.length branch — the one that prints "N more omitted" — was unreachable. The output read 25 problem(s) shown whether the real figure was 25 or 250. A backlog tracked by a number that saturates cannot be shown to be shrinking.

The count was already being kept (undocumented); it simply never reached the message.

#420 uncaps collection, keeps the display capped, and adds a denominator. Current output against origin/master:

fixture provenance: 82 problem(s), 82 of 126 fixtures with no provenance record
  — showing the first 25, 57 omitted

So this issue's 82 is confirmed, against a tree whose fixture total has moved since (126, not 125).

One correction worth recording, because it is the same mistake this issue exists to prevent. I first counted independently and got 60, and published that number to myself before checking it. It was wrong: I guessed the four fixture directories instead of reading the gate's list, and guessed one wrong. A confident, specific number derived from an input set nobody verified is exactly the hand-authored-fixture failure in another costume — the shape looks right, so nobody checks what produced it.

Not claiming the issue. The remaining work is the honest part and it does not scale by automation: each of the 82 needs a provenance line stating how those bytes were actually produced, and for fixtures I did not create I cannot know that without evidence. Writing "captured" to clear a gate would be strictly worse than the blank it replaces. The issue's own instruction — record honestly, including authored test input — is the right one, and it needs whoever has the history for each file.

What #420 gives that work is a number that moves when it succeeds.

[lamemustafa]

Correcting my comment above: 82 is not confirmed, and I had no business saying it was.

#413 — opened before my comment, which I had not looked for — shows the agent fixtures carry JSON sidecars beside each capture, recording the source read, observed date, source response bytes and SHA-256, the fixture's own bytes and SHA-256, the transformation list, a stated provenance limit and a sensitivity review. That is a stricter record than the Markdown table row this gate looks for, because it binds the fixture to the response rather than to a filename, and says what the fixture does not establish.

The gate reads only Markdown. So it counted every one of those as undocumented — and counted the sidecars themselves as undocumented, which is provenance records being asked for provenance. The real figure is 51, not 82.

So on this one question I produced two confident wrong numbers in an evening:

• 60, from guessing the four fixture directories instead of reading the gate's list.
• 82, from trusting the gate's own output without asking whether it could see the evidence format actually in use.

The second is the instructive one, and it is this issue's own thesis turned on the tool measuring it: a count is only as good as the population it ranges over and the evidence format it can recognise. A gate that cannot read JSON will report a repository with excellent JSON provenance as having none, and the number will look authoritative because a tool produced it.

I have closed my #420 in favour of #413, which fixes the counting defect I found and the recognition defect I missed.

What I would still flag for whoever finishes this: once #413 lands, the remaining figure is the honest one, and the part that does not scale by automation is unchanged — each remaining fixture needs a line stating how its bytes were actually produced, written by someone who knows. Writing "captured" to clear a gate is worse than the blank it replaces.

[lamemustafa]

A discriminator for the remaining set, with a positive control — it does not close anything, but it turns "undocumented" into two different problems.

The observation

Tally's real responses are CRLF-indented. Across every UTF-16LE fixture in the covered directories:

	n	CRLF present	CRLF absent
sidecar claims byte-exact (transformation: [])	13	13	0
sidecar records a transformation	2	2	0
no sidecar at all	20	17	3

13 of 13 known byte-exact captures carry CRLF, with no exceptions. That is the control, and it is what makes the absence meaningful rather than a guess.

Exactly three fixtures in the whole corpus lack it, and all three are among the four this issue's remaining set names:

• native-billallocations-wildcard.utf16le.xml
• native-entry-wildcard-allocations.utf16le.xml
• native-ledger-masters-duty-heads.utf16le.xml

The fourth, native-ledger-catalogue-renamed.utf16le.xml, does carry CRLF — so it is undocumented but looks captured, a different problem from the other three.

What this is and is not

It is correlational. A hand-authored fixture could include CRLF, and a captured one could lose it to a reformatting step that nobody recorded — which is itself a legitimate transformation entry rather than a disqualification. So this does not prove authorship either way.

What it does is split the backlog by what kind of answer is needed:

• 17 undocumented but CRLF-bearing — these look like captures whose record was never written. Someone with the history can likely write an honest sidecar.
• 3 undocumented and CRLF-free — these want a different question asked. Either they were authored, or they were derived from a capture through a step nobody recorded. authored test input may well be the honest value, and per this issue that is a valid one.

The three come from #311, whose commit message says the duty-head vocabulary was "backed by a live capture of all five spellings, not a hand-authored table". So the vocabulary was captured; whether these particular files are the capture bytes or a construction from them is what the sidecar should say, and I cannot tell from outside.

Reproducing it

Decode each *.utf16le.xml as UTF-16LE and test for \r\n. It costs seconds and is worth running before writing any provenance line, because it tells you which claim you are in a position to make.

I am not claiming this issue — recording the evidence so whoever does is not starting from 51 undifferentiated unknowns. Related: the CRLF property is also load-bearing for parser correctness, per tally-response-shape-defeats-a-current-tag-parser — a fixture without it cannot exercise the Event::Empty path that real responses produce.