From 73d404df782fbf30b057639e84c55214705de4d9 Mon Sep 17 00:00:00 2001 From: cdeust Date: Thu, 30 Jul 2026 09:26:48 +0200 Subject: [PATCH 1/2] fix(mutmut): re-verify survivors against full test selection before trusting them (#269) mutmut's per-mutant test attribution is recorded once, from a coverage trace of the FIRST test whose call reaches the mutated line. A module that builds a dispatch table eagerly at import time (memoizing the built closures, as ast_parser._EXTRACTORS does) is invisible to that attribution for every test after the first, so mutmut narrowed the per-mutant rerun to one irrelevant test and reported "survived" for all 50 mutants of ast_extractor_registry.py's _make_extractor/ build_extra_extractors even though the full 3-file test selection genuinely kills every one. scripts/mutation_recheck_survivors.py closes the gap generically: every mutant mutmut reports "survived" is re-run against the full declared test selection before the verdict is trusted. A mutant the full selection kills is reported RECOVERED, a distinct category, never silently folded into "killed" -- the false-survivor cause stays visible to the reader (acceptance criterion 2), not just absorbed. scripts/mutation_check.sh gained multi-file test-selection support, needed because these two functions require ast_parser.py's own repeatedly-called functions in the same run for mutmut's OWN forced-fail bootstrap check to succeed at all. Re-running the reproduction surfaced one genuine (non-false) survivor the current mutmut version generates that the issue's original reproduction did not: _make_extractor__mutmut_10 replaces the `source` argument to the calls-extractor with None, invisible to every existing per-language test because extract_calls_generic's hardcoded node-type list never matches any of the 7 languages' own call-expression node type. A new test closes it by asserting _make_extractor's own composition contract directly. Verified: 0 genuine survivors remain in ast_extractor_registry.py; json_native.py's existing scoped run is unchanged (same 5 documented equivalents); mutation_recheck_survivors.py itself carries 0 surviving mutants on its own committed scope. Resynced the 6 doc-count sites + badge to the measured 6593 (full suite green, exit 0). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u --- CHANGELOG.md | 1 + scripts/mutation_check.sh | 43 +- scripts/mutation_recheck_survivors.py | 219 +++++++++ .../core/test_ast_extractors_multilang.py | 37 ++ .../test_mutation_recheck_survivors.py | 449 ++++++++++++++++++ 5 files changed, 742 insertions(+), 7 deletions(-) create mode 100644 scripts/mutation_recheck_survivors.py create mode 100644 tests_py/scripts/test_mutation_recheck_survivors.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d30cffc0..083aa1e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,7 @@ adheres to [Semantic Versioning](https://semver.org/). - **The doc-claim gate's own vacuity guard could be held open by a number that was never a claim.** `TEST_CLAIM` matches any `N tests` phrase in a scanned file, so an incidental count — "12 tests skipped locally that CI runs", a true and dated measurement — was read as an advertisement of the suite size. That line is not yet on `main`: it arrives with #231, whose `Test (Python 3.12)` leg fails today on exactly `CONTRIBUTING.md:36: advertises 12 tests` while its `Lint` leg passes, because the static gate skips the test-count family when no `--test-count` is given. This change is therefore the build-first half of that pair — it lands the mechanism, and #231 declares the marker on its own line. Two defects followed, and the second is the dangerous one: the false positive *counted as a match*, so `check_counts`' vacuity guard (the thing that turns a reworded or deleted claim into a build failure rather than an unnoticed loss of coverage) stayed silent. Probed on the pre-fix code: a tree whose only `N tests` text was that incidental line returned a mismatch, not the vacuity message — every real `6268 tests` claim could have been deleted and the guard would not have fired. A line whose number counts something else now declares `[not-a-count-claim: