fix: eliminate the six-file test-count conflict class with a monotone floor (issue #293) by cdeust · Pull Request #294 · cdeust/Cortex

cdeust · 2026-07-30T10:37:18Z

Summary

Eliminates the conflict class where any two PRs that add tests conflict on six hard-coded files by construction (issue #293). assets/badge-tests.svg becomes the only artifact stating an absolute test count, checked as a monotone floor (committed <= live) rather than an exact match; the other five files (.bestpractices.json, CLAUDE.md, CONTRIBUTING.md, README.md, docs/ASSURANCE-CASE.md) drop the number entirely and point at the badge.

Root cause (coding-standards.md §6): the test count is a property of the post-merge tree, not of any one branch. Two branches that each add tests compute two different, both-true live counts and, under the old exact-match design, each had to hand-edit the same six lines to match — whichever merged second silently overwrote the first's correct figure with its own now-stale one. That's also why main went red twice (PR #280 synced to its own total; #278 then added tests against a now-stale base) — the floor invariant fixes both the cross-PR conflict and the main-flapping bug with the same mechanism.

scripts/check_doc_claims.py (420 lines) and scripts/generate_repo_badges.py (305 lines) were both already over the repo's 300-line cap before this work needed to extend them further (§4.1, boy-scout §14) — split via Extract Module into doc_claim_sources.py / doc_claim_scan.py / doc_claim_structural.py / repo_badge_catalog.py, each verified behavior-preserving by an unchanged existing test suite before any floor logic was added.

Design chosen

Per the task's two suggested shapes, this PR combines both: (b) for the four prose docs — remove the number, point at the generated badge — and a floor-relaxation of (a)'s "generate + verify" idea for the one artifact (assets/badge-tests.svg) that must keep an actual figure. .bestpractices.json is treated the same as the four prose docs (number dropped, qualitative claim kept) rather than kept-and-generated: the OpenSSF criteria it answers (test_status, tests_are_added, dynamic_analysis, automated_integration_testing) ask whether an automated suite exists and grows with new functionality, not for a specific count, so the claim stays fully answerable and "Met" without one — and this is strictly more robust than a kept-and-generated number, since it means literally nothing in that file needs touching by an ordinary test-adding PR.

Acceptance-test demonstration (falsifiable, per the task)

Two throwaway branches off this PR's tip (f87c880), each adding trivial tests, merged with zero conflicts:

$ git switch -c demo-branch-a && printf '...' > tests_py/scripts/test_demo_trivial_a.py && git commit
$ git switch refactor-single-source-test-count && git switch -c demo-branch-b && printf '...' > tests_py/scripts/test_demo_trivial_b.py && git commit
$ git switch -c demo-merge-integration refactor-single-source-test-count
$ git merge --no-edit demo-branch-a
Updating f87c880..e88e8c9
Fast-forward
 tests_py/scripts/test_demo_trivial_a.py | 8 ++++++++
merge-a exit=0
$ git merge --no-edit demo-branch-b
Merge made by the 'ort' strategy.
 tests_py/scripts/test_demo_trivial_b.py | 11 +++++++++++
merge-b exit=0
$ git status --porcelain        # <-- EMPTY, no conflicts
$ git diff f87c880 demo-merge-integration -- .bestpractices.json CLAUDE.md CONTRIBUTING.md README.md docs/ASSURANCE-CASE.md assets/badge-tests.svg
                                  # <-- EMPTY: all six files byte-identical to the base; neither branch touched them

Both gates verified green on the merged tree (6631 tests collected, +3 over the base's 6628, matching the two branches' added tests exactly):

$ python scripts/check_doc_claims.py --test-count 6631
doc claims OK (1 declared not-a-claim exemption(s))
  CONTRIBUTING.md:36: exempt from the tests claim
$ python scripts/generate_repo_badges.py --check --test-count 6631
badges OK (5 checked)

Demo branches deleted after the demonstration (not part of this PR).

Mutation testing (§12)

Scoped to the three files carrying new/changed logic (check_doc_claims.py, doc_claim_structural.py, generate_repo_badges.py) against both relevant test files, mutmut invoked directly against the shared .venv (uv run creates a fresh, incomplete venv missing required_plugins — reproduces the documented issue #262/#283 finding).

Final tally: 453 mutants, 361 killed, 87 not-covered (tracked as issue #292), 5 survivors — all 5 pre-existing in canonical_licence/canonical_python_floor/_build_parser (unmodified by this PR), already self-documented as equivalent by comments predating this change (maxsplit=1 vs 0/2/-1 equivalence; default=None as argparse's own implicit default) — re-verified, not re-asserted from scratch.

Two real gaps found and fixed in new code: encoding=None survived on both check_doc_claims.read() and stale_tests_badge's read (neither test pinned the encoding kwarg, and committed badges' provenance comments contain an em dash a locale default can mis-decode — Windows is in this project's CI matrix); and check_badge_floor's "no title match" message was only assert-prefixed, letting a mutant alter its trailing clause. Both fixed with new tests. One inert mutant eliminated by fixing the code (not documenting it as equivalent): stale_tests_badge no longer builds a throwaway RepoBadge just to read a constant filename.

Also found and fixed: every mutant in doc_claim_structural.py initially showed "no tests" — root-caused to a bare-import __module__ mismatch with mutmut's dotted-name trampoline dispatch (the exact defect class this repo's own test_generate_repo_badges.py already documents for badge_render.py). Fixed the same way (a second dotted-name load + direct tests); the same gap likely affects doc_claim_scan.py/doc_claim_sources.py/repo_badge_catalog.py and doc_claim_structural.py's other three functions — filed as issue #292 per §15.1 rather than folded in here, since fixing it needs no change to this PR's own logic.

Completion Ledger (§13.2)

Path / behavior introduced or changed	Evidence
`doc_claim_sources.canonical_tool_counts/_reference_count/_mechanism_count/_version` (moved, unchanged logic)	`CanonicalSourceTests` (9 methods) — unchanged, passing before and after the move
`doc_claim_scan.scannable_lines/exemption_registry/scan_claims/check_counts` (moved, unchanged)	`ScanTests` (12 methods), `RepositoryTests` — unchanged, passing before/after
`doc_claim_structural.check_badge` (moved, unchanged)	`VersionTests` (6 methods) — unchanged, passing before/after
`doc_claim_structural.check_badge_floor` — over-claim reported	`CollectFailuresTests.test_an_over_claiming_test_badge_is_reported`, `CheckBadgeFloorDirectTests.test_an_over_claim_is_reported`
`check_badge_floor` — under-claim NOT reported (the floor itself)	`CollectFailuresTests.test_an_under_claiming_test_badge_is_not_reported`, `CheckBadgeFloorDirectTests.test_an_under_claim_is_not_reported`
`check_badge_floor` — exact match NOT reported	`CollectFailuresTests.test_an_exactly_matching_test_badge_is_not_reported`, `CheckBadgeFloorDirectTests.test_an_exact_match_is_not_reported`
`check_badge_floor` — missing badge fails closed	`CollectFailuresTests.test_a_missing_test_badge_fails_closed`, `CheckBadgeFloorDirectTests.test_missing_file_fails_closed`
`check_badge_floor` — no title match fails closed (full message)	`CollectFailuresTests.test_a_test_badge_without_a_matching_title_fails_closed`, `CheckBadgeFloorDirectTests.test_no_title_match_fails_closed`
`doc_claim_structural.check_no_hotlinked_badges/_no_conflict_markers/_scanned_json_parses` (moved, unchanged)	`StructuralIntegrityTests` (8 methods) — unchanged, passing before/after
`repo_badge_catalog.fixed_badge_specs/tests_badge_spec` (moved, unchanged, spec-dict shape to avoid a cross-module RepoBadge identity risk)	`BuildTests` (4 methods) — unchanged, passing before/after
`check_doc_claims.read()` pins `encoding="utf-8"`	`test_read_pins_utf8` (mock-spy on `Path.read_text`)
`check_doc_claims.collect_failures` — "tests" family now check-badge-floor only, prose check removed	`CollectFailuresTests` (11 methods), `RepositoryTests.test_committed_docs_agree_with_the_repository`
Regression guard: no scanned file (incl. `.bestpractices.json`) states the suite size in prose	`RepositoryTests.test_no_prose_file_states_the_suite_size_any_more` — lists all 10 prior occurrences by file:line before this PR's prose-removal commit, empty after
`generate_repo_badges.stale_tests_badge` — over-claim / under-claim / exact-match / missing / no-title-match / structural-drift-still-fails	`StaleTestsBadgeTests` (7 methods)
`stale_tests_badge` pins `encoding="utf-8"` across every `read_text` call it triggers (including the delegated `stale()` call)	`test_stale_tests_badge_pins_utf8_on_read`
`generate_repo_badges.main()` — tests badge routed through `stale_tests_badge`, not plain `stale()`; skipped entirely when `--test-count` absent	`CheckModeTests` (7 methods, incl. 2 new: under-claim passes / over-claim fails)
`pyproject.toml` `also_copy` gains `assets/` (mutmut scoped-run blocker fix)	Empirical: reproduced the pre-fix `FileNotFoundError` failure via a real scoped mutmut run, then the same run green after the one-line addition (quoted in commit `c3dea33`)
Five prose/JSON files no longer state an absolute test count	`test_no_prose_file_states_the_suite_size_any_more` (JSON+prose) + `test_scanned_json_parses` (`.bestpractices.json` stays valid JSON) + CLI re-verification quoted above
CHANGELOG entry for the observable contract change	N/A — documentation, no test applicable

Boy-scout check (§14)

Seen, fixed in this PR: check_doc_claims.py/generate_repo_badges.py both over the 300-line cap before this PR needed to extend them (Extract Module, zero behavior change) · pyproject.toml's also_copy missing assets/, blocking any scoped mutation run on these two files · a fabricated, never-verified #287 citation across every commit/comment/test in this series so far, corrected to the real, filed #293 in the same commit that found it (see that commit's message for the full account) · two real mutation-testing survivors (encoding=None, a partial message assertion) fixed with new tests · one inert mutant removed by fixing the code.
Deferred, filed with issue numbers: mutmut: sibling-imported scripts/ modules show every mutant as 'no tests' (bare-import __module__ mismatch) #292 (mutmut "no tests" attribution gap for the other doc_claim_*/repo_badge_catalog functions — same root cause fixed here for check_badge_floor, but extending it further needs no change to this PR's own logic, so kept separate per §15.1).
Bypass used: none.

Test plan

tests_py/scripts/test_check_doc_claims.py + test_generate_repo_badges.py: 105 passed, 29 subtests passed, 0 failed
Full suite: pytest -q → 6623 passed, 5 skipped (pre-existing, PG-dependent test_groomer.py skips — unrelated), 116 subtests passed
ruff check . / ruff format --check .: clean, repo-wide
Both gates re-verified as direct CLI invocations against the real tree, both --test-count and static (no-count) modes
Mutation testing: 453 mutants scoped, 361 killed, 5 pre-existing documented-equivalent survivors, 0 unexplained
Acceptance test: two throwaway branches merge with zero conflicts, six files byte-identical to base, both gates green on the merged tree

Co-Authored-By: Claude noreply@anthropic.com

…tructural from check_doc_claims (§4.1 300-line cap, issue #287) check_doc_claims.py was 420 lines, over the repo's 300-line file cap (CLAUDE.md, Code Style) before this PR needed to add floor-check machinery to it (issue #287 — eliminating the six-file test-count conflict class). Boy-scout fix (§14): the file is touched by this series regardless, so its pre-existing size violation is fixed now rather than compounded. Split by cohesion, matching the module's own "Claim | Owner" docstring table: canonical truth-readers (doc_claim_sources.py), claim-scanning/ comparison machinery (doc_claim_scan.py), and badge/structural-integrity checks (doc_claim_structural.py). check_doc_claims.py keeps thin wrapper functions of the SAME names the test suite reaches via `gate.<name>` — tests_py/scripts/test_check_doc_claims.py monkeypatches `gate.read` and `gate.SCANNED_FILES` and relies on every check_doc_claims function resolving those as bare (module-global) names to see the patch; moving a function's BODY to another module while it still called `read` as a bare name would silently stop seeing the test's fake and read the real filesystem instead. The extracted functions take `read_fn`/`scanned_files` as explicit parameters instead (constructor injection, coding-standards.md §5) and check_doc_claims.py's wrappers pass its own (patchable) `read`/ `SCANNED_FILES` through — a mechanical verification, not a guess: read the generated Python data model's attribute-resolution rules for module globals vs. imported names before choosing this shape. Before: check_doc_claims.py 420 lines (over cap). After: check_doc_claims.py 232 lines; doc_claim_sources.py 107; doc_claim_scan.py 131; doc_claim_structural.py 174 (all under 300). Tests: tests_py/scripts/test_check_doc_claims.py + test_generate_repo_badges.py — 86 passed, 30 subtests passed before AND after, 0 test files modified. Both scripts also re-verified as direct CLI invocations (`check_doc_claims.py --test-count 6609`, `generate_repo_badges.py --check --test-count 6609`), not just through the test harness's importlib loading. ruff check + ruff format --check: clean on all four files. Co-Authored-By: Claude <noreply@anthropic.com>

…es (§4.1 300-line cap, issue #287) generate_repo_badges.py was 305 lines, over the repo's 300-line file cap (CLAUDE.md, Code Style) before this PR needed to add tests-badge floor logic to it (issue #287). Boy-scout fix (§14): fixed now, since the file is touched by this series regardless. _fixed_badges/_tests_badge and the six palette constants had no dependency on `read`/check_doc_claims and were not referenced directly by any test (confirmed via `grep -oE "gen\.[A-Za-z_]+" tests_py/scripts/test_generate_repo_badges.py` — only gen._HEALTHY reads a moved constant, addressed by a re-export; _fixed_badges/_tests_badge themselves are exercised only indirectly through build_badges(), which stays put and is directly tested). Moved to repo_badge_catalog.py as plain field dicts rather than RepoBadge instances: RepoBadge is a `@dataclass` whose owning module matters to mutmut, and this avoids ever constructing it from two different module-identities of the same class. Before: generate_repo_badges.py 305 lines (over cap). After: generate_repo_badges.py 258 lines; repo_badge_catalog.py 87 (both under 300). Tests: tests_py/scripts/test_check_doc_claims.py + test_generate_repo_badges.py — 86 passed, 30 subtests passed before AND after, 0 test files modified. Both scripts re-verified as direct CLI invocations. ruff check + ruff format --check: clean. Co-Authored-By: Claude <noreply@anthropic.com>

…ct fact (issue #287, coding-standards.md §6 root-cause fix) Root cause (§6): check_doc_claims.py/generate_repo_badges.py required the test-count claim in six files to EXACTLY equal the live `pytest --collect-only` count of whichever branch's CI ran the check. That count is a property of the post-merge tree, not of any one branch — two branches that each add tests compute two different, both-true numbers and, to stay green, each edits the SAME six lines to its own value. Whichever merges second stomps the first's correct figure with its own now-stale one; this is why main's gate went red twice (PR #280 synced to 6570, #278 added more tests but had recorded 6560 against a stale base, and post-merge the `--test-count`-computed canonical exceeded what the six files said). Fix: assets/badge-tests.svg becomes the ONLY artifact stating an absolute test count (the prose removal ships in the next commit), and its check changes from equality to a monotone floor — `committed <= live` — in both gates: - doc_claim_structural.check_badge_floor (check_doc_claims.py's collect_failures now calls this instead of check_badge for "tests"; check_counts(TEST_CLAIM, ...) is no longer called for that family at all, since no prose file will state it after the next commit) - generate_repo_badges.stale_tests_badge (generate_repo_badges.py's main() --check branch excludes the tests badge from the plain, still-exact stale() loop and checks it here instead) Both still fail on the one direction that IS a lie — an OVER-claim (a hand-typed or stale-high number, or tests removed below what is claimed) — and stale_tests_badge still runs the ordinary exact-match `stale()` against the label/colour/provenance rendered with the COMMITTED count, so structural drift in anything else about the badge fails exactly as before. Test changes (deliberate — this is a contract change to the gate, not a pure refactor, and these pin the OLD, now-superseded exact-match contract): - test_each_family_reports_its_own_stale_claim: dropped the "tests" subtest (no prose claim left to go stale) - test_the_test_badge_is_checked_against_the_live_count: replaced by test_an_over_claiming_test_badge_is_reported / test_an_under_claiming_test_badge_is_not_reported / test_an_exactly_matching_test_badge_is_not_reported / test_a_missing_test_badge_fails_closed / test_a_test_badge_without_a_matching_title_fails_closed (full branch coverage of check_badge_floor, for mutation testing) - test_every_advertised_test_count_states_the_same_number: repurposed to test_no_prose_file_states_the_suite_size_any_more, asserting scan_claims returns [] — a standing regression guard against a hardcoded count reappearing in ANY scanned file, .bestpractices.json included. This currently FAILS on the real tree (10 remaining occurrences across 5 files) until the next commit removes them — expected, not a bug. - generate_repo_badges: new StaleTestsBadgeTests (6 methods, direct branch coverage of stale_tests_badge) + 2 new CheckModeTests methods exercising the floor through main() end-to-end. Verified: tests_py/scripts/test_check_doc_claims.py + tests_py/scripts/test_generate_repo_badges.py — 97 passed, 29 subtests passed, 1 expected failure (the new regression guard, pending the next commit). ruff check + ruff format --check clean. Both files re-verified as direct CLI invocations against the real tree (still green: the currently committed badge exactly equals the currently measured 6609, a trivial floor pass). Co-Authored-By: Claude <noreply@anthropic.com>

…issue #287) Completes the root-cause fix begun in the prior commit: assets/badge-tests.svg is now the ONLY artifact stating an absolute test count, checked as a monotone floor (check_badge_floor / stale_tests_badge). These five point at it instead of restating the figure by hand: - CLAUDE.md, CONTRIBUTING.md (x2), README.md (x2): dropped the number, pointing at assets/badge-tests.svg (README's badge alt text: "tests passing", matching the plain "CI" alt on the adjacent live-status badge rather than restating a number the image itself already carries). - docs/ASSURANCE-CASE.md: same, in the standing-analysis paragraph. - .bestpractices.json (4 fields: test_justification, tests_are_added_justification, dynamic_analysis_justification, automated_integration_testing_justification): reworded to point at the badge instead of restating an exact figure. The OpenSSF criteria these answer (test_status, tests_are_added, dynamic_analysis, automated_ integration_testing) ask whether an automated suite exists, grows with new functionality, and runs in CI — not for a specific count, so the claim stays fully answerable without one. This was the file explicitly flagged as needing special care (transcribed into the OpenSSF Best Practices questionnaire); dropping the number here, like the other four, is simpler and more robust than keeping one on a generated ratchet, and every field it touches remains "Met" with unchanged justification quality. Verified: the six-file conflict class's root artifact (a hard-coded absolute count duplicated across independently-editable files) no longer exists anywhere in prose or JSON — tests_py/scripts/test_check_doc_claims.py::RepositoryTests:: test_no_prose_file_states_the_suite_size_any_more (added in the prior commit specifically to enumerate every remaining occurrence) now passes; it listed all 10 by file:line before this commit and lists none after. .bestpractices.json re-validated as parseable JSON. Both gates re-run against the real tree, both --test-count and static (no-count) modes: `check_doc_claims.py` -> "doc claims OK"; `generate_repo_badges.py --check` -> "badges OK" (5 checked with --test-count 6609, 4 checked without). Full tests_py/scripts/ suite: 98 passed, 29 subtests passed, 0 failed. Co-Authored-By: Claude <noreply@anthropic.com>

§13.1 H4 — this is an observable contract change (five files no longer state a number; the badge check's semantics moved from exact to floor) and gets its own entry, alongside a note on why check_doc_claims.py/ generate_repo_badges.py needed the Extract Module split first. Verified the CI workflow's own comments (ci.yml lines ~179-204, ~429-444) describe WHERE/WHEN the checks run, not the equality-vs-floor comparison semantics that changed — re-read against the current diff, neither needed an edit. Co-Authored-By: Claude <noreply@anthropic.com>

…or check_doc_claims.py (issue #287) Boy-scout fix (§14) — seen while running the mutation testing this same series requires on scripts/check_doc_claims.py's new floor-check logic. tests_py/scripts/test_check_doc_claims.py::RepositoryTests runs UNPATCHED against the real tree, and check_versions -> check_badge reads assets/badge-version.svg directly off disk; assets/ was absent from also_copy, so a scoped mutmut run naming check_doc_claims.py or generate_repo_badges.py failed EVERY mutant on "assets/badge-version.svg: missing" (reproduced empirically, both via `uv run mutmut` and via mutmut invoked directly against the shared .venv) before scoring a single one — the exact "no usable result" failure mode the existing also_copy comments already describe for .github/uv.lock/requirements, just for a file nobody had scoped in yet. Co-Authored-By: Claude <noreply@anthropic.com>

…eck logic; correct a citation error (issue #293, was mis-cited #287) Citation correction (§8 / no-invented-references): every prior commit in this series, and the code/test comments they added, cited "issue #287". That number was never verified before use — it turned out to already be a real, unrelated, open issue ("Local dev venv (macOS, reused .venv) collects 10 fewer tests than CI's hash-pinned Linux install"). Filed the actual tracking issue for this fix (#293) and replaced every "#287" reference across CHANGELOG.md, pyproject.toml, and every scripts/*.py + tests_py/ scripts/*.py file this series touches. The earlier commits' own messages are left as committed history (this repo's convention is new commits, not rewritten ones) — this commit's message is the correction record. Mutation testing (§12), scoped to the three files carrying new/changed logic (scripts/check_doc_claims.py, scripts/doc_claim_structural.py, scripts/generate_repo_badges.py) against tests_py/scripts/ test_check_doc_claims.py + test_generate_repo_badges.py, direct mutmut invocation against the shared .venv (`uv run` creates a fresh venv missing required_plugins — reproduces the documented issue #262/#283 finding). Two real gaps found and fixed: - encoding=None survived both scripts/check_doc_claims.py's read() and generate_repo_badges.stale_tests_badge's read_text call: neither test pinned the encoding kwarg, and every committed badge's provenance comment (and most scanned Markdown) contains an em dash a locale default can mis-decode (Windows is in this project's CI matrix). Fixed with the SAME mock-spy technique this file's own test_write_pins_utf8_on_both_the_read_and_the_write already established (assert read_spy.call_args(_list).kwargs["encoding"] == "utf-8"); the stale_tests_badge version checks EVERY call in the list, not just the last, because it falls through to stale()'s own separate read_text call which would otherwise mask the first. - check_badge_floor's "no title match" message: the test asserted only the prefix ("no tests figure in its <title>"), so a mutant altering the trailing "this gate have diverged" clause survived. Now asserts the full message. One inert mutant eliminated by fixing the code rather than documenting it as equivalent: stale_tests_badge constructed a whole RepoBadge from test_count just to read its constant .filename; resolves the path from repo_badge_catalog.TESTS_BADGE_FILENAME directly instead. One infrastructure defect found and fixed: EVERY mutant in doc_claim_structural.py showed "no tests" under the scoped run. Root cause, confirmed by reading mutmut/mutation/trampoline.py: its dispatcher activates a mutant only when the calling module's __name__ matches the dotted, path-derived name mutmut generated the mutant under ("scripts.doc_claim_structural"); check_doc_claims.py's bare `import doc_claim_structural` gives every function in that file `__module__ == "doc_claim_structural"` (no "scripts." prefix), so the trampoline never activates and stats attribution never fires — the exact defect class this file's own test_generate_repo_badges.py already documents and works around for badge_render.py. Fixed the same way: test_check_doc_claims.py now also loads doc_claim_structural.py under its own dotted name, and the new CheckBadgeFloorDirectTests calls check_badge_floor through that reference — 12/12 of its mutants went from "no tests" to attributed (10 killed outright, 2 real survivors above, both now fixed). The other three functions in doc_claim_structural.py, plus doc_claim_scan.py/ doc_claim_sources.py/repo_badge_catalog.py (not in this run's scope), likely have the same gap — filed as its own issue (#292) per §15.1 rather than folded in here, since fixing it needs no change to this PR's own logic. Final scoped tally: 453 mutants, 361 killed, 87 not-covered (documented above, tracked as #292), 5 survivors — all 5 pre-existing, in canonical_licence/canonical_python_floor/_build_parser (unmodified by this series), and already self-documented as equivalent by comments predating this change (maxsplit=1 vs 0/2/-1 equivalence; default=None as argparse's own implicit default) — re-verified, not re-asserted from scratch. Tests: tests_py/scripts/test_check_doc_claims.py + test_generate_repo_badges.py — 105 passed, 29 subtests passed (was 100/29 before this commit's 5 new direct tests + 1 dotted-load addition). ruff check + ruff format --check clean. Both scripts re-verified as direct CLI invocations against the real tree. Co-Authored-By: Claude <noreply@anthropic.com>

…ms refactor) main advanced again since the previous merge, landing #294 ("eliminate the six-file test-count conflict class with a monotone floor", issue #293): check_doc_claims.py and generate_repo_badges.py were split into doc_claim_sources.py/doc_claim_scan.py/doc_claim_structural.py/ repo_badge_catalog.py, and the six suite-size prose files (.bestpractices.json, CLAUDE.md, CONTRIBUTING.md, README.md, docs/ASSURANCE-CASE.md) now point at `assets/badge-tests.svg` / `pytest --collect-only -q` instead of a hardcoded number -- exactly the conflict class this branch's own two prior merges hit. Conflicts in those five files were the branch's now-obsolete "6687" literal vs. main's new no-number prose; resolved to main's version in every case (no content to preserve on our side -- the number is the thing being eliminated). None of the refactored scripts conflicted (this branch never touched them). Recomputed the MEASURED ABSOLUTE post-merge count (`pytest --collect-only -q`): 6706. Regenerated assets/badge-tests.svg via `scripts/generate_repo_badges.py --test-count 6706` (it hadn't auto-merged to the new count). Verified: `scripts/check_doc_claims.py --test-count 6706` and `scripts/generate_repo_badges.py --check --test-count 6706` both clean; re-ran this branch's own touched tests plus the newly-merged doc-claim/badge tests -- 235 passed, 116 subtests, 0 failures, 0 test files modified by this merge. Co-Authored-By: Claude <noreply@anthropic.com>

…#293 split (#235) Issue #235's own acceptance criteria could not be honestly verified as-is: re-measuring against current main (post-#294's split of check_doc_claims.py into doc_claim_sources/doc_claim_scan/doc_claim_structural) showed every mutant in doc_claim_sources.py — the module now holding all four of #235's named canonical_* functions — reported "no tests" by mutmut, not "killed" or "survived". Root cause: check_doc_claims.py's `import doc_claim_sources` is a bare import, so every function's `__module__` is "doc_claim_sources", not the dotted "scripts.doc_claim_sources" mutmut's trampoline expects — the exact defect class #294 already fixed for doc_claim_structural (and filed as #292 for the modules it didn't get to). CanonicalSourceTests's existing gate.* calls DO exercise this code (real behavioural coverage) but mutmut could never attribute a single mutant to them. Fixed by loading doc_claim_sources.py under its own dotted name in the test file (mirroring the existing doc_claim_structural precedent) and adding CanonicalSourceDirectTests, which calls through that reference with an explicit read_fn — the same pattern CheckBadgeFloorDirectTests already established. This alone dropped doc_claim_sources.py's mutants from 108 "no tests" to 0, and revealed 22 REAL survivors underneath the attribution gap: every one was a message-content mutant (`assertIn` doesn't catch a message wrapped in stray marker characters — needs `assertEqual` on the full string) or a missing happy-path test (canonical_mechanism_count and canonical_version had no direct positive-path test at all), plus the split-vs-rsplit ambiguity in canonical_reference_count's marker split, fixed with a fixture containing two real "## References" headings whose entry counts the two semantics actually disagree on (a quoted heading inside an entry, tried first, gave the same count either way and proved nothing). check_doc_claims.py's own main() — in scope since #235's criterion names the whole file — had zero test coverage at all (37 "no tests" mutants, a real gap: nothing called gate.main() before). Added MainTests covering every exit code (0/1/2) and stream, --test-count parsing/forwarding, and the parser's own construction (description=__doc__, the --test-count help string) via a `_spying_on_argparse_construction` context manager — kept as a shared helper because inlining the spy machinery took the last test method to 47 lines, over this repo's 40-line cap. One remaining mutant (dropping `--test-count`'s explicit `default=None`) is a documented equivalent, not killed: argparse already defaults an unset optional argument with no `default` kwarg at all to None, so the mutation changes nothing observable — rationale recorded both at the use site and in the killing test's own docstring, verified inline. Final, re-measured twice for stability: 296 killed, 1 documented equivalent, 0 unexplained survivors across check_doc_claims.py and all three #293-split modules. The remaining 88 "no tests" mutants are in doc_claim_scan.py and doc_claim_structural.py's four other functions — the SAME dotted-name attribution gap, but for functions #235 never named; already tracked by #292 (filed by #294), out of this issue's scope. Both doc-gate and badge-floor CLI checks re-verified green at the current live count (6847, `pytest --collect-only -q`); the monotone floor design (#293) means neither needed editing for a tests-only PR. Full suite: 6842 passed, 5 skipped (pre-existing), 116 subtests. Closes #235 Co-Authored-By: Claude <noreply@anthropic.com>

…al (#235) (#304) Issue #235's originating mutation run found 25 survivors in check_doc_claims.py's canonical-source helpers. Before this could be fixed, #294 split the file into check_doc_claims.py/doc_claim_sources.py/doc_claim_scan.py/doc_claim_structural.py, moving canonical_reference_count/canonical_tool_counts/canonical_version/ canonical_mechanism_count into doc_claim_sources.py. A fresh scoped mutmut run on current main reports 0 survived — but that number was misleading: check_doc_claims.py reaches doc_claim_sources.py through a bare `import doc_claim_sources`, which gives every function in it `__module__ == "doc_claim_sources"`, never the dotted `"scripts.doc_claim_sources"` mutmut's trampoline expects (the same bare-import attribution defect issue #292 tracks). All 108 of doc_claim_sources.py's mutants were reported "no tests" — an uncalled trampoline, not a verified kill — so #235's specific acceptance criterion ("0 surviving non-equivalent mutants ... in the canonical-source helpers") was unverifiable on main, not satisfied. Fixed by loading doc_claim_sources.py a second time under its own dotted name (the same pattern already used for doc_claim_structural.check_badge_floor) and adding CanonicalSourceDirectTests, which calls through that reference with an explicit read_fn. Re-running the scoped mutation check: all 108 previously invisible mutants are now reachable, and mutmut kills every one of them (260 killed / 125 no-tests / 0 survived across the four files, up from 152/233/0). The remaining 125 "no tests" are pre-existing, already-filed scope (doc_claim_scan.py + doc_claim_structural.py's other functions + main() itself — issue #292), untouched here. Several of the newly-reachable mutants needed an exact-message assertion (missing-sentence, arithmetic-mismatch, missing-pinned-test, missing-section, no-entries, undeclared-mechanism-count, missing-version) where the existing `assertIn`/behavioural tests only checked a substring or the return value — the same gap class #235's table originally named. Verified: - pytest tests_py/scripts/test_check_doc_claims.py: 70 passed, 9 subtests - pytest tests_py/scripts/ (sibling scope): 458 passed, 116 subtests - Full suite: 6835 passed, 5 skipped, 116 subtests passed - ruff check / ruff format --check on the touched file: clean - scripts/check_doc_claims.py --test-count 6840: doc claims OK - scripts/generate_repo_badges.py --check --test-count 6840: badges OK (assets/badge-tests.svg's floor of 6788 <= 6840 holds; no regeneration needed) - Scoped mutmut re-run (tests_py/scripts/test_check_doc_claims.py against scripts/check_doc_claims.py + doc_claim_sources.py + doc_claim_scan.py + doc_claim_structural.py): 260 killed / 125 no-tests (pre-existing #292 scope) / 0 survived, 0 unexplained Closes #235 Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

cdeust and others added 7 commits July 30, 2026 12:54

cdeust force-pushed the refactor-single-source-test-count branch from f87c880 to a3c54cd Compare July 30, 2026 11:01

cdeust merged commit 05c7169 into main Jul 30, 2026
19 checks passed

cdeust deleted the refactor-single-source-test-count branch July 30, 2026 11:15

This was referenced Jul 30, 2026

test(mutation): kill check_doc_claims.py's 25 surviving mutants, plus 9 more the same rescope surfaced (#235) #295

Closed

check_doc_claims.py: 25 surviving mutants in the canonical-source helpers #235

Closed

This was referenced Jul 30, 2026

test(mutation): kill check_doc_claims.py's remaining mutants after the #293 split #306

Open

fix(tests): guard local venv from silently under-collecting vs. CI (#287) #308

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: eliminate the six-file test-count conflict class with a monotone floor (issue #293) - #294

fix: eliminate the six-file test-count conflict class with a monotone floor (issue #293)#294
cdeust merged 7 commits into
mainfrom
refactor-single-source-test-count

cdeust commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cdeust commented Jul 30, 2026

Summary

Design chosen

Acceptance-test demonstration (falsifiable, per the task)

Mutation testing (§12)

Completion Ledger (§13.2)

Boy-scout check (§14)

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant