Skip to content

test(mutation): kill check_doc_claims.py's remaining mutants after the #293 split - #306

Merged
cdeust merged 1 commit into
mainfrom
fix-check-doc-claims-mutants-235
Jul 30, 2026
Merged

test(mutation): kill check_doc_claims.py's remaining mutants after the #293 split#306
cdeust merged 1 commit into
mainfrom
fix-check-doc-claims-mutants-235

Conversation

@cdeust

@cdeust cdeust commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #235.

Issue #235's acceptance criteria ("0 surviving non-equivalent mutants in
scripts/check_doc_claims.py") could not be honestly verified as-is against
current main. PR #294 (merged earlier the same day) split
check_doc_claims.py into doc_claim_sources.py / doc_claim_scan.py /
doc_claim_structural.py, moving #235's four named canonical_* functions
into doc_claim_sources.py. Re-measuring with a fresh scoped mutmut run
(scripts/mutation_check.sh tests_py/scripts/test_check_doc_claims.py scripts/check_doc_claims.py scripts/doc_claim_sources.py scripts/doc_claim_scan.py scripts/doc_claim_structural.py) showed every one
of doc_claim_sources.py's 108 mutants as no tests — not killed, not
survived. mutmut simply could not attribute a single mutant to the tests
that exercise this module.

Root cause: check_doc_claims.py's import doc_claim_sources is a bare
import, so every function's __module__ is "doc_claim_sources", not the
dotted "scripts.doc_claim_sources" mutmut's trampoline derives from the
file's path and expects. This is the exact defect class #294 already fixed
for doc_claim_structural.py (and filed as #292 for the modules it didn't
get to — doc_claim_scan.py and three of doc_claim_structural.py's four
other functions, none of them named by #235, left to #292 as originally
scoped). CanonicalSourceTests's existing gate.* calls DO exercise
doc_claim_sources.py's code — real behavioural coverage — mutmut just
couldn't see it.

Fix

Loaded doc_claim_sources.py under its own dotted name in the test file
(mirroring the existing doc_claim_structural precedent) and added
CanonicalSourceDirectTests, calling through that reference with an
explicit read_fn — the same pattern CheckBadgeFloorDirectTests already
established for check_badge_floor. This alone dropped
doc_claim_sources.py's mutants from 108 "no tests" to 0 attributable, and
revealed 22 real survivors underneath the attribution gap:

  • Every message-content mutant (mutmut's string-literal "wrap in XX"
    mutation) survived because the existing assertions used assertIn, which
    still matches a message wrapped in stray marker characters. Switched to
    assertEqual on the full raised message throughout the new direct tests.
  • canonical_mechanism_count and canonical_version had no direct
    happy-path test at all
    — added both.
  • canonical_reference_count's split(marker, 1) vs an rsplit/unlimited-
    split mutant needed a fixture where the two semantics actually disagree:
    a bibliography with two real "## References" headings, whose entry
    counts differ between split-on-first (2) and split-on-last (1). An
    earlier attempt (a heading merely quoted inside one entry) gave the same
    count either way and proved nothing — replaced.

check_doc_claims.py's own main() — in scope since #235's criterion names
the whole file — had zero test coverage (37 "no tests" mutants; nothing
called gate.main() before this PR). Added MainTests covering every exit
code (0/1/2) and stream, --test-count parsing/forwarding, and the parser's
own construction (description=__doc__, the --test-count help string) via
a _spying_on_argparse_construction context manager, extracted as a shared
helper because inlining the spy machinery took the last test method to 47
lines (over this repo's 40-line method cap).

One remaining mutant — dropping --test-count's explicit default=None
is a documented equivalent, not killed: argparse already defaults an
unset optional argument with no default kwarg at all to None
(ArgumentParser().add_argument("--x", type=int).default is None), so the
mutation changes nothing observable. Rationale recorded both at the use
site (scripts/check_doc_claims.py) and in the killing test's own
docstring (§12.1 — no silent "ignore").

Mutation testing (§12)

Scoped to check_doc_claims.py + all three #293-split modules, against
tests_py/scripts/test_check_doc_claims.py, mutmut invoked directly
against the shared .venv (uv run recreates an incomplete venv missing
required_plugins — issue #262/#283). Re-measured twice for stability
after the fix (identical both times):

final: 385 mutants — 384 killed, 0 no tests, 1 survived (documented equivalent)
$ python -m mutmut results | grep survived
    scripts.check_doc_claims.x_main__mutmut_8: survived   # documented equivalent, see above

Numbers updated post-rebase (see "Rebase reconciliation" below): the 88
"no tests" mutants this PR originally left in doc_claim_scan.py and
doc_claim_structural.py's other functions (#292 scope, untouched here)
are gone from the count because sibling PR #305 — merged to main
independently, closing #292 — already fixed that same dotted-name
attribution gap for those two modules before this branch rebased onto it.
Re-run against current main + this PR's tests: 0 "no tests" remain
anywhere in scope, only the one pre-known documented-equivalent survivor.

Rebase reconciliation (2026-07-30)

Rebased onto current main (e275362). One conflict, in
tests_py/scripts/test_check_doc_claims.py: sibling PR #304 (merged first,
also closing #235 under its own, narrower reading of the acceptance
criteria) added its own CanonicalSourceDirectTests covering the same four
doc_claim_sources.py functions, plus DocClaimScanDirectTests and
StructuralDirectTests for #292's remaining scope — none of which this
branch had. scripts/check_doc_claims.py's +5-line comment (the
default=None equivalence rationale) merged automatically with no
conflict.

Resolved by merging, not picking a side:

  • DocClaimScanDirectTests and StructuralDirectTests (main-only, mutmut: sibling-imported scripts/ modules show every mutant as 'no tests' (bare-import __module__ mismatch) #292
    scope): kept as-is — no equivalent in this branch.
  • MainTests and its _spying_on_argparse_construction helper
    (this-branch-only): kept as-is — no equivalent on main.
  • CanonicalSourceDirectTests (defined by BOTH sides, same class name,
    overlapping method names — a silent last-definition-wins shadow if left
    unmerged): merged into one class. Method-by-method, most pairs were
    behaviourally identical modulo fixture text (kept the version with the
    clearer docstring); one pair genuinely differed in strength —
    test_catalogue_drifting_from_the_pinned_registry_test_is_an_error used
    assertIn on main's copy vs. exact assertEqual on this branch's —
    kept the exact-match version (this branch's), since it's strictly
    stronger and this PR's own docstring already argues for exact-match
    throughout the class. No assertion was dropped; the two main-only tests
    (test_reference_count_is_the_number_of_entries_not_the_advertised_number,
    test_reference_entries_exclude_headings_and_separators) are preserved.
  • Confirmed no duplicate method names remain (ast-walked the file for
    same-class duplicate FunctionDef names — none) and no conflict markers
    remain anywhere in the tree.

Net effect measured against current main: test_check_doc_claims.py goes
from 1307 to 1485 lines (+178 net, after removing ~143 lines of duplicate
class body that a naive concatenation would have kept dead); no method
exceeds 40 lines. The file was already over the 300-line file cap on main
before this rebase (1307 lines) — per the coordinator's standing waiver,
this pre-existing §4.1 breach is tracked under #298 and is not a new issue;
growing it further here is the mandated result of preserving both PRs'
independently-reviewed test additions per the "keep both sets of assertions
unless subsumed" instruction, not discretionary scope creep. No method-size
(§4.2) debt was introduced or found.

Completion Ledger

Path / behavior Test
canonical_tool_counts happy path (direct) CanonicalSourceDirectTests.test_tool_counts_come_from_the_catalogue
canonical_tool_counts drift-vs-pinned-registry message (exact) test_catalogue_drifting_from_the_pinned_registry_test_is_an_error
canonical_tool_counts arithmetic-mismatch message (exact) test_inconsistent_arithmetic_in_the_catalogue_is_an_error
canonical_tool_counts missing-sentence message (exact) test_missing_standalone_total_sentence_is_an_error
canonical_tool_counts missing-pinned-test message (exact) test_missing_pinned_registry_test_is_an_error
canonical_reference_count split-on-FIRST-heading (not rsplit) test_the_first_references_heading_is_the_split_point
canonical_reference_count missing-section message (exact) test_bibliography_without_a_references_section_is_an_error
canonical_reference_count no-entries message (exact) test_a_references_section_with_no_entries_is_an_error
canonical_mechanism_count happy path (direct, was missing) test_mechanism_count_is_read_from_the_bibliography_header
canonical_mechanism_count undeclared message (exact) test_undeclared_mechanism_count_is_an_error
canonical_version happy path (direct, was missing) test_version_comes_from_pyproject
canonical_version missing-declaration message (exact) test_missing_version_declaration_is_an_error
main(): ClaimError abort, exit 2 MainTests.test_a_claim_error_aborts_to_stderr_with_exit_code_2
main(): failures report, exit 1 (exact stderr) test_failures_are_reported_to_stderr_with_exit_code_1
main(): success + exemption count, exit 0 test_success_reports_exemption_count_and_exit_code_0, test_success_with_no_exemptions_reports_a_zero_count
main(): --test-count parsed/forwarded test_the_test_count_flag_is_parsed_and_forwarded, test_without_the_flag_test_count_is_none
main(): parser description + help text (exact) test_the_parser_declares_the_modules_docstring_and_flag_help
default=None equivalence (argparse implicit default) documented at use site + in the test above's docstring

Test plan (post-rebase, this PR's actual pushed head 27b7fdd)

  • pytest tests_py/scripts/test_check_doc_claims.py — 110 passed, 14 subtests
  • pytest tests_py/scripts/ (sibling scope) — 545 passed, 121 subtests
  • ruff check . (whole repo, not just touched files) — clean, no other seen defects
  • ruff format --check on touched files — clean
  • python scripts/check_doc_claims.pydoc claims OK (1 declared not-a-claim exemption(s))
  • python scripts/generate_repo_badges.py --checkbadges OK (4 checked); no doc/badge edits needed (monotone floor, Any two PRs that add tests conflict on six files by construction (test count hard-coded) #293)
  • Scoped mutation (mutmut invoked directly against a symlinked shared .venv, not uv run's own incomplete sync — see above): 385 mutants, 384 killed, 0 no-tests, 1 documented equivalent, 0 unexplained survivors. Reproduced twice.
  • AST size gate on touched files: no function/method over 40 lines. tests_py/scripts/test_check_doc_claims.py is 1485 lines, over the 300-line §4.1 cap — pre-existing on main before this rebase (1307 lines already over cap); covered by the coordinator's standing waiver and tracked under Ratchet backlog: 229 file + 607 method size-cap breaches pre-date the new craftsmanship-checker gate (issue #297) #298, not a new issue (see "Rebase reconciliation").
  • No conflict markers anywhere in the tree; .bestpractices.json parses
  • pyproject.toml's transient [tool.mutmut] scoping restored to its committed baseline after every scoped mutation run (diff against origin/main...HEAD for this file: empty)
  • git diff origin/main...HEAD --stat: scripts/check_doc_claims.py (+5), tests_py/scripts/test_check_doc_claims.py (+196/-13) — matches the merged, deduplicated content described above

Co-Authored-By: Claude noreply@anthropic.com

…#293 split (#235)

Issue #235's own acceptance criteria could not be honestly verified as-is:
re-measuring against current main (post-#294's split of check_doc_claims.py
into doc_claim_sources/doc_claim_scan/doc_claim_structural) showed every
mutant in doc_claim_sources.py — the module now holding all four of #235's
named canonical_* functions — reported "no tests" by mutmut, not "killed" or
"survived". Root cause: check_doc_claims.py's `import doc_claim_sources` is
a bare import, so every function's `__module__` is "doc_claim_sources", not
the dotted "scripts.doc_claim_sources" mutmut's trampoline expects — the
exact defect class #294 already fixed for doc_claim_structural (and filed
as #292 for the modules it didn't get to). CanonicalSourceTests's existing
gate.* calls DO exercise this code (real behavioural coverage) but mutmut
could never attribute a single mutant to them.

Fixed by loading doc_claim_sources.py under its own dotted name in the test
file (mirroring the existing doc_claim_structural precedent) and adding
CanonicalSourceDirectTests, which calls through that reference with an
explicit read_fn — the same pattern CheckBadgeFloorDirectTests already
established. This alone dropped doc_claim_sources.py's mutants from 108
"no tests" to 0, and revealed 22 REAL survivors underneath the attribution
gap: every one was a message-content mutant (`assertIn` doesn't catch a
message wrapped in stray marker characters — needs `assertEqual` on the
full string) or a missing happy-path test (canonical_mechanism_count and
canonical_version had no direct positive-path test at all), plus the
split-vs-rsplit ambiguity in canonical_reference_count's marker split,
fixed with a fixture containing two real "## References" headings whose
entry counts the two semantics actually disagree on (a quoted heading
inside an entry, tried first, gave the same count either way and proved
nothing).

check_doc_claims.py's own main() — in scope since #235's criterion names
the whole file — had zero test coverage at all (37 "no tests" mutants, a
real gap: nothing called gate.main() before). Added MainTests covering
every exit code (0/1/2) and stream, --test-count parsing/forwarding, and
the parser's own construction (description=__doc__, the --test-count help
string) via a `_spying_on_argparse_construction` context manager — kept
as a shared helper because inlining the spy machinery took the last test
method to 47 lines, over this repo's 40-line cap.

One remaining mutant (dropping `--test-count`'s explicit `default=None`)
is a documented equivalent, not killed: argparse already defaults an unset
optional argument with no `default` kwarg at all to None, so the mutation
changes nothing observable — rationale recorded both at the use site and
in the killing test's own docstring, verified inline.

Final, re-measured twice for stability: 296 killed, 1 documented
equivalent, 0 unexplained survivors across check_doc_claims.py and all
three #293-split modules. The remaining 88 "no tests" mutants are in
doc_claim_scan.py and doc_claim_structural.py's four other functions —
the SAME dotted-name attribution gap, but for functions #235 never named;
already tracked by #292 (filed by #294), out of this issue's scope.

Both doc-gate and badge-floor CLI checks re-verified green at the current
live count (6847, `pytest --collect-only -q`); the monotone floor design
(#293) means neither needed editing for a tests-only PR. Full suite:
6842 passed, 5 skipped (pre-existing), 116 subtests.

Closes #235

Co-Authored-By: Claude <noreply@anthropic.com>
@cdeust
cdeust force-pushed the fix-check-doc-claims-mutants-235 branch from 3538625 to 27b7fdd Compare July 30, 2026 21:34
@cdeust
cdeust merged commit 3c180df into main Jul 30, 2026
19 checks passed
@cdeust
cdeust deleted the fix-check-doc-claims-mutants-235 branch July 30, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check_doc_claims.py: 25 surviving mutants in the canonical-source helpers

1 participant