Skip to content

test: the run counts itself, and the floor is one per suite - #450

Merged
firejune merged 1 commit into
mainfrom
test/the-run-counts-itself
Sep 6, 2026
Merged

test: the run counts itself, and the floor is one per suite#450
firejune merged 1 commit into
mainfrom
test/the-run-counts-itself

Conversation

@firejune

@firejune firejune commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Refs #439. Landed overnight; bun run typecheck clean, bun run lint clean, bun run selftest 559 PASS / 0 FAIL / 0 SKIP / 0 HOLE.

What the drift actually was

The issue reported one stale count (a suite's floor at 24 against 29 cases). Wrapping the calls put a number on the whole surface for the first time:

hand-written floor increments that were wrong 18 of 47
the summed floor stood at 435
against a run of 551
suites with no increment at all 1 (runLoopSeamSuite — its four cases had never counted toward the floor in any run)

Worst rows: path-slider 25/47, contour-mesh 8/30, check 3/19, packer 18/25, chainfit 12/18.

⚠️ A floor at 435 against a run of 551 means 116 cases could die and the run still prints green. That floor is the only thing standing between a vacuous run and a pass, and it had drifted a fifth of the way down without a sound.

The mechanism

console.log is wrapped for the duration of the suite phase; each suite call is bracketed and the case lines it printed are counted (PASS/FAIL at the two-space gutter; SKIP/INFO are seen and deliberately not counted; anything else faults). The summary asks the tally. tallyFaults() runs before the summary reads a single number off it.

🔒 The floor is one per suite, never on their sum. TY04 is the clause that makes that matter: a set whose total (20) is higher than the healthy set's (5) still faults, because one suite is dead.

Adding a control to a wrapped suite now needs no summary edit at all — measured, not asserted: planting one extra case moved the printed figure 34 → 35 with zero edits and the run stayed green. 34 of the 44 clause figures are on the tally. That is what #440/#441/#443/#444 were queued behind.

Gate — TY01TY08

TY01 is derived rather than tabled: it captures what reportCase actually prints and holds the scanner against it in both directions. TY02 holds the scanner against every gutter word this run printed — because a scanner that stops matching goes silent, not red.

Five mutants planted in the real file, run, and restored:

mutant what fired
a suite emptied the suite "…" reported that it ran and then printed no PASS or FAIL line at all
a suite call unwrapped 4 case line(s) were printed outside every tallied suite
reportCase prints OKAY the gutter fault, plus TY01/TY02 red
summary names a suite that did not run no suite ran under the key "…"
one wrapper over two suites the suite "…" opened 2 section header(s)

Verified independently before merge: emptying runSlotSuite on this branch exits 2, naming it twice — "reported that it ran and then printed no PASS or FAIL line at all" and "opened 0 section header(s); a suite opens exactly one". selftest.ts restored and checksum-matched.

Also measured: corpus parked → 508 PASS / 5 SKIP / 5 HOLE, exit 0 (the "did not run" branches behave); --cuts named-and-missing exits 2 and present-but-broken exits 1, unchanged.

🔒 The printed summary is byte-identical to main@0cd287b's apart from the one inserted clause for the new controls, and all 34 interpolated figures reproduce the hand-written number exactly.

The STOP condition fired, and nothing was edited around it

Four figures the tally derives disagree with what the summary states. They are left as literals, each marked in place with both numbers recorded beside them, and filed separately for a ruling — because three of them are a definition disagreement rather than staleness (some suites count their own positive control in their figure and some do not), and no single derivation reproduces the text. The fourth is stale under every reading.

Two more of the same family, reported not fixed: one suite of ten controls is described in the summary nowhere at all, and one corpus-dependent suite announces its absence nowhere in the section list.

Rejected approaches

  • Each suite returns its own count (~40 signature changes) — a counter a suite increments is still a restatement, and a case that forgets to increment drifts silently. The same defect one layer down.
  • Keep the stated number and compare it to the derived one (CUR08's pattern) — it keeps the hand-written number, so adding a control still costs a summary edit, which is the exact cost the four queued cards are waiting to be rid of.
  • A source scanner refusing literals in the summary region — cannot be green today, because four literals legitimately remain, so it would need a hand-kept exception table: the "✅ applied" antipattern this repository already has a judgment about.
  • Counting SKIP as substantive — rejected on measurement: all six SKIP sites mean "this did not run", so counting them would let a corpus-less run book five holes as coverage.
  • Listing HOLE in the gutter vocabulary — measured: HOLE only ever prints in the ten-space detail gutter, so it would have been a branch no control can reach.

Known limitation, stated plainly

This removes the stale count rather than gating the text. Nothing reads the printed summary, so a deliberate re-hardcode would not be caught. That is the price of the source scanner rejected above, and it is what a later card could still add once the four deltas are ruled on and the literals disappear.

The summary's figures were restated beside the suites rather than counted off
them, and the substantive floor was a hand-maintained sum. Both drifted, and
the drift was silent because a floor below the truth still fires.

Wrapping the calls put a number on it for the first time: 18 of the 47
hand-written increments were wrong, the summed floor stood at 435 against a run
of 551, and one suite had no increment at all, so its four cases had never
counted toward the floor in any run.

The run now brackets each suite call and counts the case lines that suite
printed. The summary asks the tally for its figures, the floor is one per suite
rather than one on their sum, and a suite that reports it ran and then measures
nothing exits 2 by name — even when the total grew around it.

Adding a control to a wrapped suite now needs no summary edit at all, measured
rather than asserted: planting one extra case moved the printed figure and the
run stayed green with no edit to the summary. 34 of the 44 clause figures are
on the tally.

TY01 is derived rather than tabled — it captures what reportCase actually
prints and holds the scanner against it two-sidedly — and TY02 holds the
scanner against every gutter word this run printed, because a scanner that
stops matching goes silent rather than red. Five mutants were planted in the
real file and run: a suite emptied, a suite call unwrapped, the case gutter
reworded, a summary naming a suite that did not run, and one wrapper over two
suites.

Four figures the tally derives disagree with what the summary states, so they
are left as literals with both numbers recorded beside them. Three are a
definition disagreement rather than staleness and need a ruling; the fourth is
stale under every reading. Filed separately.

Refs #439

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@firejune
firejune enabled auto-merge (squash) September 6, 2026 17:04
@firejune
firejune merged commit 3949c4a into main Sep 6, 2026
1 check passed
@firejune
firejune deleted the test/the-run-counts-itself branch September 6, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant