Skip to content

test(mutation): kill check_doc_claims.py's 25 surviving mutants, plus 9 more the same rescope surfaced (#235) - #295

Closed
cdeust wants to merge 1 commit into
mainfrom
fix-check-doc-claims-mutants-235
Closed

test(mutation): kill check_doc_claims.py's 25 surviving mutants, plus 9 more the same rescope surfaced (#235)#295
cdeust wants to merge 1 commit into
mainfrom
fix-check-doc-claims-mutants-235

Conversation

@cdeust

@cdeust cdeust commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #235.

A scoped mutmut run over scripts/check_doc_claims.py left 25 survivors in
its canonical-source helpers (canonical_reference_count 10,
canonical_tool_counts 7, canonical_version 3, canonical_mechanism_count
3, read 2) — every one a missing-branch or error-message assertion no test
exercised (e.g. canonical_version()'s "not found" arm had zero tests;
canonical_reference_count's split(..., 1) vs an unlimited-split/rsplit
mutant were indistinguishable because every fixture had only one
"## References" heading).

Reproducing the baseline first found also_copy in pyproject.toml's
[tool.mutmut] block was missing assets/RepositoryTests reads the
real badge SVGs, so the mutation run's own working copy couldn't complete
without it (a pre-existing gap in the scoped-run config, fixed here since I
was already touching this exact config for this exact file). With that fixed,
9 MORE survivors surfaced outside the issue's own table, in check_badge,
check_no_hotlinked_badges, check_no_conflict_markers, and
check_scanned_json_parses — two of them continuebreak mutants that
would silently abort scanning the rest of SCANNED_FILES after the first
missing/skipped file (exactly the failure mode check_no_conflict_markers
exists to prevent). main() itself had zero mutation coverage at all (37
"no tests" mutants — mutmut's coverage-gap category, distinct from
"survived") since every existing test called collect_failures directly
rather than the CLI entry point; a new MainTests class exercises every arm
(ClaimError abort → exit 2, failures report → exit 1, success + exemption
count → exit 0, --test-count forwarding, declared description/help
text) through main() itself.

Per the acceptance criteria's first bullet ("0 surviving non-equivalent
mutants in scripts/check_doc_claims.py" — the whole file, not just the
issue's table), all of the above are addressed in this one PR, per the
no-new-issues / no-deviation directive for this run.

Two mutants are documented equivalents, not killed (§12.1 — no third
option):

  • x_read__mutmut_4: encoding="UTF-8" vs "utf-8" — CPython's codec
    registry is case/hyphen-insensitive (codecs.lookup("UTF-8").name == codecs.lookup("utf-8").name is True), so no observable behaviour
    differs. Rationale + verification in ReadTests.test_read_decodes_explicitly_as_utf8's
    own comment.
  • x_main__mutmut_8: dropping the explicit default=None kwarg on the
    --test-count argument — argparse already defaults an unset optional
    argument to None with no default kwarg at all (verified inline).
    Rationale in MainTests's class docstring.

Final: 325 of 327 mutants killed, 2 documented equivalent, 0 unexplained
survivors.

Test-count resync (working rules)

19 new tests → suite count changes; resynced the measured absolute
(6628, matching a real pytest --collect-only -q on this exact tree) across
.bestpractices.json, CLAUDE.md, CONTRIBUTING.md, README.md,
docs/ASSURANCE-CASE.md, and regenerated assets/badge-tests.svg via
scripts/generate_repo_badges.py --test-count 6628. Both gates re-verified:

$ python scripts/check_doc_claims.py --test-count 6628
doc claims OK (1 declared not-a-claim exemption(s))
  CONTRIBUTING.md:36: exempt from the tests claim

$ python scripts/generate_repo_badges.py --check --test-count 6628
badges OK (5 checked)

Completion Ledger

Path introduced Test asserting it
read() UTF-8-explicit contract ReadTests.test_read_decodes_explicitly_as_utf8
canonical_tool_counts(): missing header sentence test_missing_standalone_total_sentence_is_an_error
canonical_tool_counts(): arithmetic-mismatch message test_inconsistent_arithmetic_in_the_catalogue_is_an_error (message assert added)
canonical_tool_counts(): missing pinned test test_missing_pinned_registry_test_is_an_error
canonical_reference_count(): split-vs-rsplit/maxsplit semantics test_the_first_references_heading_is_the_split_point
canonical_reference_count(): missing-section message test_bibliography_without_a_references_section_is_an_error (message assert added)
canonical_reference_count(): no-entries-found message test_a_references_section_with_no_entries_is_an_error
canonical_mechanism_count(): undeclared-count message test_undeclared_mechanism_count_is_an_error (message assert added)
canonical_version(): missing-version message test_missing_version_declaration_is_an_error
check_badge(): full "diverged" message test_a_badge_without_a_version_title_fails_closed (exact-match assert)
check_no_hotlinked_badges(): line number + message HotlinkedBadgeTests (both tests)
check_no_conflict_markers(): continue-not-break on missing file test_a_missing_file_does_not_abort_the_conflict_scan
check_scanned_json_parses(): missing-.json message test_a_missing_json_file_fails_closed
check_scanned_json_parses(): continue-not-break (both loops) test_a_non_json_file_does_not_abort_the_json_scan, test_a_missing_json_file_does_not_abort_the_json_scan
main(): ClaimError abort, exit 2 test_a_claim_error_aborts_to_stderr_with_exit_code_2
main(): failures report, exit 1 test_failures_are_reported_to_stderr_with_exit_code_1
main(): success + exemption count, exit 0 test_success_reports_exemption_count_and_exit_code_0, test_success_with_no_exemptions_reports_a_zero_count
main(): --test-count parsed/forwarded test_the_test_count_flag_is_parsed_and_forwarded, test_without_the_flag_test_count_is_none
main(): declared description/help text test_the_parser_declares_the_modules_docstring_and_flag_help
pyproject.toml also_copy gap (assets/) mutation baseline reproduces cleanly (RepositoryTests no longer fails collection under mutants)

Boy-scout items fixed in this PR

  • pyproject.toml's [tool.mutmut] also_copy list was missing assets/,
    which made ANY scoped mutation run against check_doc_claims.py fail at
    baseline collection (RepositoryTests.test_committed_docs_agree_with_the_repository)
    before ever generating a mutant. Fixed (comment explains why, dated).
  • One test method (test_the_parser_declares_the_modules_docstring_and_flag_help)
    initially came in at 46 lines; extracted the argparse-patching mechanics
    into a shared _spying_on_argparse_construction context manager (also
    documents the super(ArgumentParser, self) recursion gotcha it exists to
    avoid) — all test methods now ≤40 lines.
  • ruff format reformatted 3 over-88-char lines I introduced; committed
    formatted.

Test plan

  • pytest tests_py/scripts/test_check_doc_claims.py — 65 passed, 10 subtests
  • pytest tests_py/scripts/ (full sibling scope) — 348 passed, 117 subtests
  • pytest tests_py/shared/test_json_native.py (the file's own committed mutmut demonstration target, untouched by this PR but re-verified since I edited pyproject.toml's [tool.mutmut] block) — 14 passed
  • Full suite: 6623 passed, 5 skipped, 0 failed
  • ruff check + ruff format --check on touched files — clean
  • pyright on touched files — 0 errors/warnings/informations
  • Scoped mutation re-run after every edit, final: 325/327 killed, 2 documented equivalent, 0 unexplained survivors
  • pyproject.toml's transient only_mutate/source_paths/pytest_add_cli_args_test_selection restored to the committed baseline (mcp_server/shared/json_native.py) after each scoped run — only the permanent assets addition to also_copy is committed

🤖 Generated with Claude Code

https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u

… 9 more the same rescope surfaced (#235)

A scoped mutmut run over the doc-claim gate left 25 survivors in its
canonical-source helpers (canonical_reference_count 10, canonical_tool_counts
7, canonical_version 3, canonical_mechanism_count 3, read 2) — all
missing-branch and error-message assertions no test exercised. Fixing
also_copy to include assets/ (missing from the committed mutation config, so
the baseline run could not even complete) surfaced 9 more survivors outside
the issue's own table, in check_badge, check_no_hotlinked_badges,
check_no_conflict_markers, and check_scanned_json_parses — including two
continue->break mutants that would silently abort scanning the rest of
SCANNED_FILES after the first missing/skipped file. main() itself had zero
mutation coverage (37 "no tests" mutants); a new MainTests class exercises
every arm through the CLI entry point itself.

Two mutants are documented equivalents rather than killed: encoding="UTF-8"
is the same codec as "utf-8" under CPython's case-insensitive codec lookup,
and dropping an explicit default=None from an argparse optional argument
changes nothing since that is argparse's own implicit default.

Final: 325 of 327 mutants killed, 2 documented equivalent, 0 unexplained
survivors. Suite grows 6609 -> 6628; doc-claim gate and badge-count gate
re-verified against the new count. Full suite: 6623 passed, 5 skipped.

Closes #235

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u
@cdeust
cdeust force-pushed the fix-check-doc-claims-mutants-235 branch from 6840d8a to a234f77 Compare July 30, 2026 11:16
@cdeust

cdeust commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

Closing as superseded by #294, which merged to main while this PR was in flight (conflict this PR hit was the six-file test-count resync — the exact class #294 eliminates).

#294 split scripts/check_doc_claims.py into doc_claim_sources.py/doc_claim_scan.py/doc_claim_structural.py and ran its own scoped mutation-testing pass over the same functions this PR targeted, including the identical read() encoding=None gap and an analogous badge-message-assertion gap this PR also found and fixed independently.

Verified empirically against the current origin/main tip (05c7169), not just taken from #294's own PR description: a fresh scoped mutmut run over scripts/check_doc_claims.py + all three extracted modules, using main's own committed tests with no changes —

327/385... final: 🎉 152 killed  🫥 233 no-tests  🙁 0 survived
$ python -m mutmut results | grep -c survived
0

Zero surviving mutants, confirming issue #235's acceptance criterion ("0 surviving non-equivalent mutants in scripts/check_doc_claims.py") is already met. The 233 "no tests" (coverage-gap, not survival) mutants are the same class #294 already filed separately as #292 for the sibling modules it didn't fully cover in its own scoped run — outside #235's stated acceptance criteria (which is about survivors, not coverage), and already tracked.

Recommend closing #235 as resolved-by-#294, the same disposition #267 got from #271.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check_doc_claims.py: 25 surviving mutants in the canonical-source helpers

1 participant