test(mutation): kill check_doc_claims.py's remaining mutants after the #293 split - #306
Merged
Merged
Conversation
…#293 split (#235) Issue #235's own acceptance criteria could not be honestly verified as-is: re-measuring against current main (post-#294's split of check_doc_claims.py into doc_claim_sources/doc_claim_scan/doc_claim_structural) showed every mutant in doc_claim_sources.py — the module now holding all four of #235's named canonical_* functions — reported "no tests" by mutmut, not "killed" or "survived". Root cause: check_doc_claims.py's `import doc_claim_sources` is a bare import, so every function's `__module__` is "doc_claim_sources", not the dotted "scripts.doc_claim_sources" mutmut's trampoline expects — the exact defect class #294 already fixed for doc_claim_structural (and filed as #292 for the modules it didn't get to). CanonicalSourceTests's existing gate.* calls DO exercise this code (real behavioural coverage) but mutmut could never attribute a single mutant to them. Fixed by loading doc_claim_sources.py under its own dotted name in the test file (mirroring the existing doc_claim_structural precedent) and adding CanonicalSourceDirectTests, which calls through that reference with an explicit read_fn — the same pattern CheckBadgeFloorDirectTests already established. This alone dropped doc_claim_sources.py's mutants from 108 "no tests" to 0, and revealed 22 REAL survivors underneath the attribution gap: every one was a message-content mutant (`assertIn` doesn't catch a message wrapped in stray marker characters — needs `assertEqual` on the full string) or a missing happy-path test (canonical_mechanism_count and canonical_version had no direct positive-path test at all), plus the split-vs-rsplit ambiguity in canonical_reference_count's marker split, fixed with a fixture containing two real "## References" headings whose entry counts the two semantics actually disagree on (a quoted heading inside an entry, tried first, gave the same count either way and proved nothing). check_doc_claims.py's own main() — in scope since #235's criterion names the whole file — had zero test coverage at all (37 "no tests" mutants, a real gap: nothing called gate.main() before). Added MainTests covering every exit code (0/1/2) and stream, --test-count parsing/forwarding, and the parser's own construction (description=__doc__, the --test-count help string) via a `_spying_on_argparse_construction` context manager — kept as a shared helper because inlining the spy machinery took the last test method to 47 lines, over this repo's 40-line cap. One remaining mutant (dropping `--test-count`'s explicit `default=None`) is a documented equivalent, not killed: argparse already defaults an unset optional argument with no `default` kwarg at all to None, so the mutation changes nothing observable — rationale recorded both at the use site and in the killing test's own docstring, verified inline. Final, re-measured twice for stability: 296 killed, 1 documented equivalent, 0 unexplained survivors across check_doc_claims.py and all three #293-split modules. The remaining 88 "no tests" mutants are in doc_claim_scan.py and doc_claim_structural.py's four other functions — the SAME dotted-name attribution gap, but for functions #235 never named; already tracked by #292 (filed by #294), out of this issue's scope. Both doc-gate and badge-floor CLI checks re-verified green at the current live count (6847, `pytest --collect-only -q`); the monotone floor design (#293) means neither needed editing for a tests-only PR. Full suite: 6842 passed, 5 skipped (pre-existing), 116 subtests. Closes #235 Co-Authored-By: Claude <noreply@anthropic.com>
cdeust
force-pushed
the
fix-check-doc-claims-mutants-235
branch
from
July 30, 2026 21:34
3538625 to
27b7fdd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #235.
Issue #235's acceptance criteria ("0 surviving non-equivalent mutants in
scripts/check_doc_claims.py") could not be honestly verified as-is againstcurrent
main. PR #294 (merged earlier the same day) splitcheck_doc_claims.pyintodoc_claim_sources.py/doc_claim_scan.py/doc_claim_structural.py, moving #235's four namedcanonical_*functionsinto
doc_claim_sources.py. Re-measuring with a fresh scopedmutmutrun(
scripts/mutation_check.sh tests_py/scripts/test_check_doc_claims.py scripts/check_doc_claims.py scripts/doc_claim_sources.py scripts/doc_claim_scan.py scripts/doc_claim_structural.py) showed every oneof
doc_claim_sources.py's 108 mutants asno tests— notkilled, notsurvived. mutmut simply could not attribute a single mutant to the teststhat exercise this module.
Root cause:
check_doc_claims.py'simport doc_claim_sourcesis a bareimport, so every function's
__module__is"doc_claim_sources", not thedotted
"scripts.doc_claim_sources"mutmut's trampoline derives from thefile's path and expects. This is the exact defect class #294 already fixed
for
doc_claim_structural.py(and filed as #292 for the modules it didn'tget to —
doc_claim_scan.pyand three ofdoc_claim_structural.py's fourother functions, none of them named by #235, left to #292 as originally
scoped).
CanonicalSourceTests's existinggate.*calls DO exercisedoc_claim_sources.py's code — real behavioural coverage — mutmut justcouldn't see it.
Fix
Loaded
doc_claim_sources.pyunder its own dotted name in the test file(mirroring the existing
doc_claim_structuralprecedent) and addedCanonicalSourceDirectTests, calling through that reference with anexplicit
read_fn— the same patternCheckBadgeFloorDirectTestsalreadyestablished for
check_badge_floor. This alone droppeddoc_claim_sources.py's mutants from 108 "no tests" to 0 attributable, andrevealed 22 real survivors underneath the attribution gap:
XX"mutation) survived because the existing assertions used
assertIn, whichstill matches a message wrapped in stray marker characters. Switched to
assertEqualon the full raised message throughout the new direct tests.canonical_mechanism_countandcanonical_versionhad no directhappy-path test at all — added both.
canonical_reference_count'ssplit(marker, 1)vs anrsplit/unlimited-split mutant needed a fixture where the two semantics actually disagree:
a bibliography with two real
"## References"headings, whose entrycounts differ between split-on-first (2) and split-on-last (1). An
earlier attempt (a heading merely quoted inside one entry) gave the same
count either way and proved nothing — replaced.
check_doc_claims.py's ownmain()— in scope since #235's criterion namesthe whole file — had zero test coverage (37 "no tests" mutants; nothing
called
gate.main()before this PR). AddedMainTestscovering every exitcode (0/1/2) and stream,
--test-countparsing/forwarding, and the parser'sown construction (
description=__doc__, the--test-counthelp string) viaa
_spying_on_argparse_constructioncontext manager, extracted as a sharedhelper because inlining the spy machinery took the last test method to 47
lines (over this repo's 40-line method cap).
One remaining mutant — dropping
--test-count's explicitdefault=None—is a documented equivalent, not killed: argparse already defaults an
unset optional argument with no
defaultkwarg at all toNone(
ArgumentParser().add_argument("--x", type=int).default is None), so themutation changes nothing observable. Rationale recorded both at the use
site (
scripts/check_doc_claims.py) and in the killing test's owndocstring (§12.1 — no silent "ignore").
Mutation testing (§12)
Scoped to
check_doc_claims.py+ all three #293-split modules, againsttests_py/scripts/test_check_doc_claims.py, mutmut invoked directlyagainst the shared
.venv(uv runrecreates an incomplete venv missingrequired_plugins— issue #262/#283). Re-measured twice for stabilityafter the fix (identical both times):
Numbers updated post-rebase (see "Rebase reconciliation" below): the 88
"no tests" mutants this PR originally left in
doc_claim_scan.pyanddoc_claim_structural.py's other functions (#292 scope, untouched here)are gone from the count because sibling PR #305 — merged to
mainindependently, closing #292 — already fixed that same dotted-name
attribution gap for those two modules before this branch rebased onto it.
Re-run against current
main+ this PR's tests: 0 "no tests" remainanywhere in scope, only the one pre-known documented-equivalent survivor.
Rebase reconciliation (2026-07-30)
Rebased onto current
main(e275362). One conflict, intests_py/scripts/test_check_doc_claims.py: sibling PR #304 (merged first,also closing #235 under its own, narrower reading of the acceptance
criteria) added its own
CanonicalSourceDirectTestscovering the same fourdoc_claim_sources.pyfunctions, plusDocClaimScanDirectTestsandStructuralDirectTestsfor #292's remaining scope — none of which thisbranch had.
scripts/check_doc_claims.py's+5-line comment (thedefault=Noneequivalence rationale) merged automatically with noconflict.
Resolved by merging, not picking a side:
DocClaimScanDirectTestsandStructuralDirectTests(main-only, mutmut: sibling-imported scripts/ modules show every mutant as 'no tests' (bare-import __module__ mismatch) #292scope): kept as-is — no equivalent in this branch.
MainTestsand its_spying_on_argparse_constructionhelper(this-branch-only): kept as-is — no equivalent on
main.CanonicalSourceDirectTests(defined by BOTH sides, same class name,overlapping method names — a silent last-definition-wins shadow if left
unmerged): merged into one class. Method-by-method, most pairs were
behaviourally identical modulo fixture text (kept the version with the
clearer docstring); one pair genuinely differed in strength —
test_catalogue_drifting_from_the_pinned_registry_test_is_an_errorusedassertInonmain's copy vs. exactassertEqualon this branch's —kept the exact-match version (this branch's), since it's strictly
stronger and this PR's own docstring already argues for exact-match
throughout the class. No assertion was dropped; the two main-only tests
(
test_reference_count_is_the_number_of_entries_not_the_advertised_number,test_reference_entries_exclude_headings_and_separators) are preserved.ast-walked the file forsame-class duplicate
FunctionDefnames — none) and no conflict markersremain anywhere in the tree.
Net effect measured against current
main:test_check_doc_claims.pygoesfrom 1307 to 1485 lines (+178 net, after removing ~143 lines of duplicate
class body that a naive concatenation would have kept dead); no method
exceeds 40 lines. The file was already over the 300-line file cap on
mainbefore this rebase (1307 lines) — per the coordinator's standing waiver,
this pre-existing §4.1 breach is tracked under #298 and is not a new issue;
growing it further here is the mandated result of preserving both PRs'
independently-reviewed test additions per the "keep both sets of assertions
unless subsumed" instruction, not discretionary scope creep. No method-size
(§4.2) debt was introduced or found.
Completion Ledger
canonical_tool_countshappy path (direct)CanonicalSourceDirectTests.test_tool_counts_come_from_the_cataloguecanonical_tool_countsdrift-vs-pinned-registry message (exact)test_catalogue_drifting_from_the_pinned_registry_test_is_an_errorcanonical_tool_countsarithmetic-mismatch message (exact)test_inconsistent_arithmetic_in_the_catalogue_is_an_errorcanonical_tool_countsmissing-sentence message (exact)test_missing_standalone_total_sentence_is_an_errorcanonical_tool_countsmissing-pinned-test message (exact)test_missing_pinned_registry_test_is_an_errorcanonical_reference_countsplit-on-FIRST-heading (not rsplit)test_the_first_references_heading_is_the_split_pointcanonical_reference_countmissing-section message (exact)test_bibliography_without_a_references_section_is_an_errorcanonical_reference_countno-entries message (exact)test_a_references_section_with_no_entries_is_an_errorcanonical_mechanism_counthappy path (direct, was missing)test_mechanism_count_is_read_from_the_bibliography_headercanonical_mechanism_countundeclared message (exact)test_undeclared_mechanism_count_is_an_errorcanonical_versionhappy path (direct, was missing)test_version_comes_from_pyprojectcanonical_versionmissing-declaration message (exact)test_missing_version_declaration_is_an_errormain(): ClaimError abort, exit 2MainTests.test_a_claim_error_aborts_to_stderr_with_exit_code_2main(): failures report, exit 1 (exact stderr)test_failures_are_reported_to_stderr_with_exit_code_1main(): success + exemption count, exit 0test_success_reports_exemption_count_and_exit_code_0,test_success_with_no_exemptions_reports_a_zero_countmain():--test-countparsed/forwardedtest_the_test_count_flag_is_parsed_and_forwarded,test_without_the_flag_test_count_is_nonemain(): parser description + help text (exact)test_the_parser_declares_the_modules_docstring_and_flag_helpdefault=Noneequivalence (argparse implicit default)Test plan (post-rebase, this PR's actual pushed head
27b7fdd)pytest tests_py/scripts/test_check_doc_claims.py— 110 passed, 14 subtestspytest tests_py/scripts/(sibling scope) — 545 passed, 121 subtestsruff check .(whole repo, not just touched files) — clean, no other seen defectsruff format --checkon touched files — cleanpython scripts/check_doc_claims.py—doc claims OK (1 declared not-a-claim exemption(s))python scripts/generate_repo_badges.py --check—badges OK (4 checked); no doc/badge edits needed (monotone floor, Any two PRs that add tests conflict on six files by construction (test count hard-coded) #293).venv, notuv run's own incomplete sync — see above): 385 mutants, 384 killed, 0 no-tests, 1 documented equivalent, 0 unexplained survivors. Reproduced twice.tests_py/scripts/test_check_doc_claims.pyis 1485 lines, over the 300-line §4.1 cap — pre-existing onmainbefore this rebase (1307 lines already over cap); covered by the coordinator's standing waiver and tracked under Ratchet backlog: 229 file + 607 method size-cap breaches pre-date the new craftsmanship-checker gate (issue #297) #298, not a new issue (see "Rebase reconciliation")..bestpractices.jsonparsespyproject.toml's transient[tool.mutmut]scoping restored to its committed baseline after every scoped mutation run (diff againstorigin/main...HEADfor this file: empty)git diff origin/main...HEAD --stat:scripts/check_doc_claims.py(+5),tests_py/scripts/test_check_doc_claims.py(+196/-13) — matches the merged, deduplicated content described aboveCo-Authored-By: Claude noreply@anthropic.com