fix(mutmut): re-verify survivors against full test selection before trusting them (#269) - #286
Merged
Merged
Conversation
cdeust
force-pushed
the
fix-mutmut-eager-module-import-269
branch
from
July 30, 2026 11:09
4b830c1 to
1d23aa7
Compare
…rusting them (#269) mutmut's per-mutant test attribution is recorded once, from a coverage trace of the FIRST test whose call reaches the mutated line. A module that builds a dispatch table eagerly at import time (memoizing the built closures, as ast_parser._EXTRACTORS does) is invisible to that attribution for every test after the first, so mutmut narrowed the per-mutant rerun to one irrelevant test and reported "survived" for all 50 mutants of ast_extractor_registry.py's _make_extractor/ build_extra_extractors even though the full 3-file test selection genuinely kills every one. scripts/mutation_recheck_survivors.py closes the gap generically: every mutant mutmut reports "survived" is re-run against the full declared test selection before the verdict is trusted. A mutant the full selection kills is reported RECOVERED, a distinct category, never silently folded into "killed" -- the false-survivor cause stays visible to the reader (acceptance criterion 2), not just absorbed. scripts/mutation_check.sh gained multi-file test-selection support, needed because these two functions require ast_parser.py's own repeatedly-called functions in the same run for mutmut's OWN forced-fail bootstrap check to succeed at all. Re-running the reproduction surfaced one genuine (non-false) survivor the current mutmut version generates that the issue's original reproduction did not: _make_extractor__mutmut_10 replaces the `source` argument to the calls-extractor with None, invisible to every existing per-language test because extract_calls_generic's hardcoded node-type list never matches any of the 7 languages' own call-expression node type. A new test closes it by asserting _make_extractor's own composition contract directly. Verified: 0 genuine survivors remain in ast_extractor_registry.py; json_native.py's existing scoped run is unchanged (same 5 documented equivalents); mutation_recheck_survivors.py itself carries 0 surviving mutants on its own committed scope. Resynced the 6 doc-count sites + badge to the measured 6593 (full suite green, exit 0). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u
Rebase bookkeeping only. Conflicts were confined to the six files that hard-code the suite size, and only hunks differing solely by that number were resolved (main's figure wins); any hunk with real content divergence aborts the rebase for human/agent resolution instead of being guessed at. Count is the measured absolute from a real collection. The branch's own touched tests were re-run after resolution to catch content loss.
cdeust
force-pushed
the
fix-mutmut-eager-module-import-269
branch
from
July 30, 2026 11:55
1d23aa7 to
33d147f
Compare
cdeust
added a commit
that referenced
this pull request
Jul 30, 2026
…est, split test_ast_parser.py under the 300-line cap, dedupe a CHANGELOG entry Rebasing onto main pulled in #269/#286, which added test_make_extractor_threads_the_real_source_into_calls_extraction to pin mutmut mutant x__make_extractor__mutmut_10 (a source-argument mutation on the extract_calls_generic call _make_extractor used to thread through). This branch's own earlier commit deletes that exact call site and extract_calls_generic itself (dead code, issue #249 boy-scout pass), so the mutant the test existed to kill can no longer be generated from the source at all -- git's line-based merge didn't conflict (the two edits touch different lines), but the merged tree failed test_ast_extractors_multilang.py with AttributeError on ast_extractor_registry.extract_calls_generic. Root cause: a semantic merge conflict between two independently-correct changes, invisible to textual diffing. Fixed by deleting the retired test (its own in-file note records why) rather than adapting it -- the 2-tuple _make_extractor now composes exactly two calls, and a source-argument mutant on either is already killed by the 7 per-language tests (test_java et al.), each of which asserts on _imports/_defs output that is only correct when the REAL source bytes reach imports_fn/defs_fn. Verified empirically, not just by inspection: a scoped mutmut run (ast_extractor_registry.py + ast_parser.py, full 6-file test selection) reports 50 "survived" mutants recovered by mutation_recheck_survivors.py's full-selection reverification (the same eager-import-dispatch-table false-positive pattern #269 diagnosed) -- 0 genuine survivors. Boy-scout (seen while touching this tree): test_ast_parser.py had grown to 405 lines under this branch's own Go/Swift/Rust wrapper-test additions, over CLAUDE.md's 300-line/file cap -- a NEW violation this branch introduces (main's version is 171 lines), not pre-existing debt. Split (Extract Module, zero behavior change, same test names/assertions) into test_ast_parser.py (223 lines), test_ast_parser_node_extraction.py (103 lines: the fake-Node _extract_module_doc/_node_text unit tests) and test_ast_parser_extra_langs.py (122 lines: the Go/Swift/Rust parse_file_ast integration tests) -- all three under cap. Also seen: CHANGELOG.md's [Unreleased] section carried the #249 boy-scout entry TWICE (a stale pre-extract_calls_generic-deletion draft alongside the final one), both stating a suite count already stale relative to main's independent growth. Deduped to the single correct entry; reworded its count claim to a relative delta (net +20 tests) per the moment any other PR merges first -- assets/badge-tests.svg is the one artifact that still states one). Suite count re-measured post-fix: 6776 (pytest --collect-only -q on this tree), matching the full run (6771 passed + 5 skipped). assets/badge-tests.svg regenerated via scripts/generate_repo_badges.py --test-count 6776; scripts/check_doc_claims.py --test-count 6776 and scripts/generate_repo_badges.py --check --test-count 6776 both green. Verification: tests_py/core/test_ast_parser.py + test_ast_parser_node_extraction.py + test_ast_parser_extra_langs.py + test_ast_extractors.py + test_ast_extractors_multilang.py + test_ast_parser_language_contract.py (69 passed, unchanged assertions), tests_py/core/ (3384 passed), full suite (6771 passed, 5 skipped, 116 subtests passed), ruff check + ruff format --check clean on all changed files, pyright 0 errors/0 warnings on all changed files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cdeust
added a commit
that referenced
this pull request
Jul 30, 2026
…est, split test_ast_parser.py under the 300-line cap, dedupe a CHANGELOG entry Rebasing onto main pulled in #269/#286, which added test_make_extractor_threads_the_real_source_into_calls_extraction to pin mutmut mutant x__make_extractor__mutmut_10 (a source-argument mutation on the extract_calls_generic call _make_extractor used to thread through). This branch's own earlier commit deletes that exact call site and extract_calls_generic itself (dead code, issue #249 boy-scout pass), so the mutant the test existed to kill can no longer be generated from the source at all -- git's line-based merge didn't conflict (the two edits touch different lines), but the merged tree failed test_ast_extractors_multilang.py with AttributeError on ast_extractor_registry.extract_calls_generic. Root cause: a semantic merge conflict between two independently-correct changes, invisible to textual diffing. Fixed by deleting the retired test (its own in-file note records why) rather than adapting it -- the 2-tuple _make_extractor now composes exactly two calls, and a source-argument mutant on either is already killed by the 7 per-language tests (test_java et al.), each of which asserts on _imports/_defs output that is only correct when the REAL source bytes reach imports_fn/defs_fn. Verified empirically, not just by inspection: a scoped mutmut run (ast_extractor_registry.py + ast_parser.py, full 6-file test selection) reports 50 "survived" mutants recovered by mutation_recheck_survivors.py's full-selection reverification (the same eager-import-dispatch-table false-positive pattern #269 diagnosed) -- 0 genuine survivors. Boy-scout (seen while touching this tree): test_ast_parser.py had grown to 405 lines under this branch's own Go/Swift/Rust wrapper-test additions, over CLAUDE.md's 300-line/file cap -- a NEW violation this branch introduces (main's version is 171 lines), not pre-existing debt. Split (Extract Module, zero behavior change, same test names/assertions) into test_ast_parser.py (223 lines), test_ast_parser_node_extraction.py (103 lines: the fake-Node _extract_module_doc/_node_text unit tests) and test_ast_parser_extra_langs.py (122 lines: the Go/Swift/Rust parse_file_ast integration tests) -- all three under cap. Also seen: CHANGELOG.md's [Unreleased] section carried the #249 boy-scout entry TWICE (a stale pre-extract_calls_generic-deletion draft alongside the final one), both stating a suite count already stale relative to main's independent growth. Deduped to the single correct entry; reworded its count claim to a relative delta (net +20 tests) per the moment any other PR merges first -- assets/badge-tests.svg is the one artifact that still states one). Suite count re-measured post-fix: 6776 (pytest --collect-only -q on this tree), matching the full run (6771 passed + 5 skipped). assets/badge-tests.svg regenerated via scripts/generate_repo_badges.py --test-count 6776; scripts/check_doc_claims.py --test-count 6776 and scripts/generate_repo_badges.py --check --test-count 6776 both green. Verification: tests_py/core/test_ast_parser.py + test_ast_parser_node_extraction.py + test_ast_parser_extra_langs.py + test_ast_extractors.py + test_ast_extractors_multilang.py + test_ast_parser_language_contract.py (69 passed, unchanged assertions), tests_py/core/ (3384 passed), full suite (6771 passed, 5 skipped, 116 subtests passed), ruff check + ruff format --check clean on all changed files, pyright 0 errors/0 warnings on all changed files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cdeust
added a commit
that referenced
this pull request
Jul 30, 2026
…est, split test_ast_parser.py under the 300-line cap, dedupe a CHANGELOG entry Rebasing onto main pulled in #269/#286, which added test_make_extractor_threads_the_real_source_into_calls_extraction to pin mutmut mutant x__make_extractor__mutmut_10 (a source-argument mutation on the extract_calls_generic call _make_extractor used to thread through). This branch's own earlier commit deletes that exact call site and extract_calls_generic itself (dead code, issue #249 boy-scout pass), so the mutant the test existed to kill can no longer be generated from the source at all -- git's line-based merge didn't conflict (the two edits touch different lines), but the merged tree failed test_ast_extractors_multilang.py with AttributeError on ast_extractor_registry.extract_calls_generic. Root cause: a semantic merge conflict between two independently-correct changes, invisible to textual diffing. Fixed by deleting the retired test (its own in-file note records why) rather than adapting it -- the 2-tuple _make_extractor now composes exactly two calls, and a source-argument mutant on either is already killed by the 7 per-language tests (test_java et al.), each of which asserts on _imports/_defs output that is only correct when the REAL source bytes reach imports_fn/defs_fn. Verified empirically, not just by inspection: a scoped mutmut run (ast_extractor_registry.py + ast_parser.py, full 6-file test selection) reports 50 "survived" mutants recovered by mutation_recheck_survivors.py's full-selection reverification (the same eager-import-dispatch-table false-positive pattern #269 diagnosed) -- 0 genuine survivors. Boy-scout (seen while touching this tree): test_ast_parser.py had grown to 405 lines under this branch's own Go/Swift/Rust wrapper-test additions, over CLAUDE.md's 300-line/file cap -- a NEW violation this branch introduces (main's version is 171 lines), not pre-existing debt. Split (Extract Module, zero behavior change, same test names/assertions) into test_ast_parser.py (223 lines), test_ast_parser_node_extraction.py (103 lines: the fake-Node _extract_module_doc/_node_text unit tests) and test_ast_parser_extra_langs.py (122 lines: the Go/Swift/Rust parse_file_ast integration tests) -- all three under cap. Also seen: CHANGELOG.md's [Unreleased] section carried the #249 boy-scout entry TWICE (a stale pre-extract_calls_generic-deletion draft alongside the final one), both stating a suite count already stale relative to main's independent growth. Deduped to the single correct entry; reworded its count claim to a relative delta (net +20 tests) per the moment any other PR merges first -- assets/badge-tests.svg is the one artifact that still states one). Suite count re-measured post-fix: 6776 (pytest --collect-only -q on this tree), matching the full run (6771 passed + 5 skipped). assets/badge-tests.svg regenerated via scripts/generate_repo_badges.py --test-count 6776; scripts/check_doc_claims.py --test-count 6776 and scripts/generate_repo_badges.py --check --test-count 6776 both green. Verification: tests_py/core/test_ast_parser.py + test_ast_parser_node_extraction.py + test_ast_parser_extra_langs.py + test_ast_extractors.py + test_ast_extractors_multilang.py + test_ast_parser_language_contract.py (69 passed, unchanged assertions), tests_py/core/ (3384 passed), full suite (6771 passed, 5 skipped, 116 subtests passed), ruff check + ruff format --check clean on all changed files, pyright 0 errors/0 warnings on all changed files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cdeust
added a commit
that referenced
this pull request
Jul 30, 2026
…eric + close Swift/Rust test gaps (follow-on to #249, closed by #261) (#270) * fix(types): narrow ast_parser's language str to a Literal, closing the resolver-dependent typecheck gate (#249) mcp_server/core/ast_parser.py:88 passed a plain `str` to tree_sitter_language_pack.get_parser(), whose declared parameter type differs across releases (a `SupportedLanguage` Literal in 1.6.2 vs plain `str` in 1.13.5) -- so pyright's zero-diagnostic verdict was a property of which version pip/uv resolved, not of this source. AST_SUPPORTED is now a Literal union narrowed via a TypeGuard instead of a bare set[str], verified clean under both the current uv.lock resolution and a standalone 1.6.2 install (the only residual error there is 1.6.2's own SupportedLanguage omitting "csharp" -- a pre-existing grammar-availability gap in that release, not a narrowing defect). Boy-scout pass on the touched file (mutation-tested, tools/mutation_check.sh equivalent, scoped to ast_parser.py + ast_extractor_registry.py): removed the flat `calls` list every per-language extractor computed and parse_file_ast immediately discarded (superseded by calls_per_function, never itself consumed), and closed the resulting zero test coverage for Swift/Rust parsing plus gaps in is_available, _node_text, _extract_module_doc, content_hash length, and calls_per_function's populated content. Filed #269 for the 50 mutmut "survived" reports left after that pass, which are proven false positives (verified directly via MUTANT_UNDER_TEST against the full test selection): mutmut's coverage attribution undercounts callers of a dispatch table built once, eagerly, at module-import time. Test count 6526 -> 6549 (+23), resynced across README/CONTRIBUTING/ CLAUDE.md/ASSURANCE-CASE.md/.bestpractices.json/badge-tests.svg via scripts/check_doc_claims.py + scripts/generate_repo_badges.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u * chore: reconcile rebase onto main (#261) — recompute suite count, fix CHANGELOG claim Rebasing onto main pulled in #261/#253/#251, which independently closed via the same TypeGuard-narrowing mechanism, already keyed off the language pack's own SupportedLanguage type and with AST_SUPPORTED derived from _EXTRACTORS rather than restated. This branch's own duplicate _SupportedLanguage/_is_ast_supported/AST_SUPPORTED-via-get_args mechanism is dropped in favour of main's (ast_parser.py, ast_extractor_registry.py); this branch's actual surviving contribution — removing the dead `calls` tuple element from every extractor and adding the missing Swift/Rust/ is_available/_node_text/_extract_module_doc/content_hash/calls_per_function test coverage — is preserved and now sits on top of main's mechanism cleanly (auto-merged with no conflicts in ast_extractor_registry.py). CHANGELOG's #249 entry rewritten: it no longer claims to close #249 (already closed by #261) and instead describes only the boy-scout dead-code-removal and test-coverage work this branch still adds. Suite count recomputed from a live `pytest --collect-only -q` on this machine (6572, up from main's 6549 by the 23 tests this branch adds) per the standing rule that the count is a measured absolute, not a delta from an advertised number: CLAUDE.md, README.md, CONTRIBUTING.md, .bestpractices.json, docs/ASSURANCE-CASE.md, assets/badge-tests.svg. Verified: scripts/check_doc_claims.py --test-count 6572 and scripts/generate_repo_badges.py --check --test-count 6572 both green; tests_py/scripts/test_typecheck_env_parity.py, test_check_doc_claims.py, test_generate_repo_badges.py all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u * chore(ast): delete extract_calls_generic — its own direct unit test was the last caller (#249 review) Review finding on PR #270: removing the dead `calls` tuple element from every per-language extractor (previous commit in this branch) stranded extract_calls_generic (mcp_server/core/ast_extractors.py) — nothing in production called it anymore, only TestCallExtraction's two tests called it directly. Dead code per coding-standards.md §9: deleted the function and its now-pointless direct unit test, corrected ast_extractors.py's module docstring (dropped the false "Also provides the generic call-site extractor used by all languages" claim), and updated ast_extractor_registry.py's _make_extractor docstring to record that the function itself, not just its result, is now gone. _MAX_CALL_NAME_LEN and _walk_type remain live — both are still used by the per-function, caller-qualified call extraction a few lines down (extract_calls_per_function), which is the value parse_file_ast actually consumes. Suite count recomputed from a live `pytest --collect-only -q` on this tree (6571, down 2 from 6573 for the deleted tests) and resynced across CLAUDE.md/README.md/CONTRIBUTING.md/.bestpractices.json/ docs/ASSURANCE-CASE.md/assets/badge-tests.svg; scripts/check_doc_claims.py --test-count 6571 and scripts/generate_repo_badges.py --check --test-count 6571 both green. Verification: tests_py/core/test_ast_parser.py + test_ast_extractors_multilang.py + test_ast_extractors.py + test_ast_parser_language_contract.py (69 passed), tests_py/core/ (3384 passed), full suite (6571 passed, 117 subtests passed, exact match to the recomputed collect-only count), ruff check + ruff format --check clean on all three changed .py files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u * refactor(ast): split extract_calls_per_function's nested walk under the 40-line/method cap (boy-scout, CLAUDE.md §4) Seen while touching this file for the extract_calls_generic deletion: extract_calls_per_function (51 lines including its nested `walk` closure) already violated this repo's 40-line/method cap on main, pre-existing and unrelated to either fix in this branch. Fixed in place since it is squarely in this diff's blast radius (coding-standards.md §14.1) and the split carries zero external risk — the public `extract_calls_per_function(root, source) -> dict[str, list[str]]` signature and behavior are unchanged, only the internals move. Split into three functions, each under the cap: `extract_calls_per_function` (17 lines, unchanged behavior/contract) delegates to `_walk_for_calls` (36 lines, the former nested closure — now a named top-level helper taking `out` as an explicit accumulator parameter instead of a captured free variable) which delegates per-function call collection to `_collect_call_basenames` (10 lines, the former inner per-call-type loop). Pure extract-method refactor: no branch, condition, or dedup logic changed. Not addressed here: this file's own line count (330 after this commit, still over the repo's 300-line/file cap — it was already 337 before either commit in this branch touched it). A real fix needs a genuine module split (this file has 6 importers: ast_parser.py plus the clike/extra/jvm/scripting extractor siblings and its own test file — Move-7's >5-importers High-stakes trigger), which means updating import statements in 5 other files. That is a separate, dedicated behavior-preserving-refactor PR per coding-standards.md §15.1 ("never ship the connection half-made... never mix the enabling refactor and the feature in one PR"), not a rider on a rebase-plus-one-review-finding PR already carrying real risk from two prior CONFLICTING states. Flagged in this session's report for the coordinator rather than filed as an issue (explicit instruction: no new issues this round). Verification: tests_py/core/test_ast_parser.py + test_ast_extractors_multilang.py + test_ast_extractors.py + test_ast_parser_language_contract.py (69 passed, unchanged), tests_py/core/ (3384 passed, unchanged), ruff check + ruff format --check clean, pyright 0 errors/0 warnings on the changed file. Full suite count unchanged at 6571 (no tests added or removed by this commit). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u * chore: rebase onto main and resync the advertised suite size to 6777 Rebase bookkeeping only. Conflicts were confined to the six files that hard-code the suite size, and only hunks differing solely by that number were resolved (main's figure wins); any hunk with real content divergence aborts the rebase for human/agent resolution instead of being guessed at. Count is the measured absolute from a real collection. The branch's own touched tests were re-run after resolution to catch content loss. * fix(ast): rebase reconciliation — retire an obsolete mutation-guard test, split test_ast_parser.py under the 300-line cap, dedupe a CHANGELOG entry Rebasing onto main pulled in #269/#286, which added test_make_extractor_threads_the_real_source_into_calls_extraction to pin mutmut mutant x__make_extractor__mutmut_10 (a source-argument mutation on the extract_calls_generic call _make_extractor used to thread through). This branch's own earlier commit deletes that exact call site and extract_calls_generic itself (dead code, issue #249 boy-scout pass), so the mutant the test existed to kill can no longer be generated from the source at all -- git's line-based merge didn't conflict (the two edits touch different lines), but the merged tree failed test_ast_extractors_multilang.py with AttributeError on ast_extractor_registry.extract_calls_generic. Root cause: a semantic merge conflict between two independently-correct changes, invisible to textual diffing. Fixed by deleting the retired test (its own in-file note records why) rather than adapting it -- the 2-tuple _make_extractor now composes exactly two calls, and a source-argument mutant on either is already killed by the 7 per-language tests (test_java et al.), each of which asserts on _imports/_defs output that is only correct when the REAL source bytes reach imports_fn/defs_fn. Verified empirically, not just by inspection: a scoped mutmut run (ast_extractor_registry.py + ast_parser.py, full 6-file test selection) reports 50 "survived" mutants recovered by mutation_recheck_survivors.py's full-selection reverification (the same eager-import-dispatch-table false-positive pattern #269 diagnosed) -- 0 genuine survivors. Boy-scout (seen while touching this tree): test_ast_parser.py had grown to 405 lines under this branch's own Go/Swift/Rust wrapper-test additions, over CLAUDE.md's 300-line/file cap -- a NEW violation this branch introduces (main's version is 171 lines), not pre-existing debt. Split (Extract Module, zero behavior change, same test names/assertions) into test_ast_parser.py (223 lines), test_ast_parser_node_extraction.py (103 lines: the fake-Node _extract_module_doc/_node_text unit tests) and test_ast_parser_extra_langs.py (122 lines: the Go/Swift/Rust parse_file_ast integration tests) -- all three under cap. Also seen: CHANGELOG.md's [Unreleased] section carried the #249 boy-scout entry TWICE (a stale pre-extract_calls_generic-deletion draft alongside the final one), both stating a suite count already stale relative to main's independent growth. Deduped to the single correct entry; reworded its count claim to a relative delta (net +20 tests) per the moment any other PR merges first -- assets/badge-tests.svg is the one artifact that still states one). Suite count re-measured post-fix: 6776 (pytest --collect-only -q on this tree), matching the full run (6771 passed + 5 skipped). assets/badge-tests.svg regenerated via scripts/generate_repo_badges.py --test-count 6776; scripts/check_doc_claims.py --test-count 6776 and scripts/generate_repo_badges.py --check --test-count 6776 both green. Verification: tests_py/core/test_ast_parser.py + test_ast_parser_node_extraction.py + test_ast_parser_extra_langs.py + test_ast_extractors.py + test_ast_extractors_multilang.py + test_ast_parser_language_contract.py (69 passed, unchanged assertions), tests_py/core/ (3384 passed), full suite (6771 passed, 5 skipped, 116 subtests passed), ruff check + ruff format --check clean on all changed files, pyright 0 errors/0 warnings on all changed files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #269.
scripts/mutation_check.shfalse-reported 50 survivors forast_extractor_registry.py's_make_extractor/build_extra_extractors:mutmut's per-mutant test attribution is recorded once, from a coverage
trace of the FIRST test whose call reaches the mutated line. Since
ast_parser._EXTRACTORS = {..., **build_extra_extractors()}builds itsdispatch table once, eagerly, at import time, every test after the
first exercises the already-built, cached closures without ever
re-invoking the mutated functions — mutmut narrowed the per-mutant
rerun to one (often irrelevant) test and reported "survived" for all 50
mutants even though the full 3-file test selection genuinely kills
every one.
scripts/mutation_recheck_survivors.pycloses the gap generically,for any source file, not by special-casing this one: every mutant
mutmut reports "survived" is re-run against the FULL declared test
selection before the verdict is trusted. A mutant the full selection
kills is reported as RECOVERED — a distinct, visible category,
never silently folded into "killed" (a false-survivor report otherwise
either blocks a correct commit or trains reviewers to wave off real
survivors as "probably a tooling artifact", the exact silent-failure
risk §12 exists to close).
scripts/mutation_check.shalso gained multi-file test-selectionsupport (
<test_paths>now accepts a space-separated list), whichturned out to be load-bearing, not cosmetic: reproducing the issue
requires
ast_parser.pyin the same mutation scope, because mutmut'sown forced-fail bootstrap check needs something in scope that gets
re-invoked on every test (mutating
ast_extractor_registry.pyalonestarves that check — nothing in it is ever called more than once per
process — and
mutmut runaborts with "Unable to force test failures"before producing any per-mutant result at all).
Re-running the full reproduction surfaced one genuine (non-false)
survivor the current mutmut version (3.6.0) generates that the
issue's original reproduction did not:
_make_extractor__mutmut_10replaces the
sourceargument to the calls-extractor withNone,invisible to every existing per-language test because
extract_calls_generic's hardcoded node-type list ("call"/"call_expression") never matches any of the 7 languages' owncall-expression node type (Java's is
method_invocation, forinstance — a separate, pre-existing, out-of-scope language-coverage
gap). A new test,
test_make_extractor_threads_the_real_source_into_calls_extraction,monkeypatches
extract_calls_genericto assert_make_extractor's owncomposition contract directly, closing it without touching that
unrelated gap.
Layer assignment (Move 1)
scripts/mutation_recheck_survivors.py— new, pure decision logic(parse/recheck/format) plus a thin I/O boundary (
default_runner,injected everywhere else); lives beside
scripts/mutation_check.sh,the other repo-tooling script in this family.
scripts/mutation_check.sh— orchestration only (bash), unchanged role.tests_py/scripts/test_mutation_recheck_survivors.py,tests_py/core/test_ast_extractors_multilang.py— tests..bestpractices.json/badge SVG) — required by this repo's
suite-size-is-hardcoded convention; not a design change.
Stakes calibration (Move 7)
concurrency/data-integrity), but load-bearing for every future PR's
mutation gate. Full Moves 1–5 applied anyway given the depth of the
investigation required.
Completion Ledger
parse_survivors— extract "survived" mutant namesParseSurvivorsTests(5 tests: normal, none, empty, dedup, substring-immune)recheck_survivor— re-run one mutant against full selectionRecheckSurvivorTests(7 tests: exit 0/nonzero, env var, base_env non-mutation, exact cmd, cwd, mutant_name passthrough)recheck_all_survivors— empty short-circuit / ValueError / per-survivor recheckRecheckAllSurvivorsTests(4 tests, incl. exact ValueError message + mutants_dir forwarding)any_genuine_survivorsAnyGenuineSurvivorsTests(3 tests)format_report— empty / all-recovered / all-genuine / mixedFormatReportTests(4 tests, exact-string equality)_main— argv validation, stdin parse, exit codes 0/1/2, stdout/stderr contentMainCliTests(9 tests, incl. exact usage text, exact printed report)default_runner(real subprocess boundary)mutation_check.shruns below, not unit-mocked (Move 5: thin I/O boundary)mutation_check.shmulti-test-path splitting +--separatorast_extractor_registry.py+ast_parser.py(3-file selection) and againstjson_native.py(1-file selection, unchanged behavior)mutation_check.shsurvivor recheck wiring_make_extractorcalls-threading testtest_make_extractor_threads_the_real_source_into_calls_extraction; killsx__make_extractor__mutmut_10(confirmed via targeted mutation run)scripts/check_doc_claims.py --test-count 6593exit 0,scripts/generate_repo_badges.py --check --test-count 6593exit 0, full suite6593 passed, 117 subtests passedVerification
scripts/mutation_recheck_survivors.pymutation-tested against its own new test file: 0 surviving mutants (final run, after three iterations closing string-literal/argument-forwarding/exact-output gaps the first two runs surfaced).ast_extractor_registry.py+ast_parser.pymutated together against the 3-file selection: 0 genuine survivors in_make_extractor/build_extra_extractors(all 53 reported "survived" by mutmut are RECOVERED); the 33 remaining GENUINE survivors are all inast_parser.py's OTHER, unrelated functions (is_available,parse_file_ast,_node_text,_extract_module_doc,_extract_python,_extract_js) — pre-existing, out of this issue's stated scope, and NOT part of this PR's shipped[tool.mutmut]scope (stilljson_native.pyin the committedpyproject.toml).json_native.pyscoped run unchanged: same 5 documented-equivalent survivors as before (no regression).ruff check ./ruff format --check .: clean.shellcheck scripts/mutation_check.sh: clean.6593 passed, 117 subtests passed(exit 0).Unresolved (evidence, not a ticket — flagging per this run's instructions)
Mutating
ast_extractor_registry.pytogether withast_parser.py(required for mutmut's own forced-fail bootstrap to succeed at all when reproducing this issue) surfaces 33 genuine, pre-existing mutation survivors inast_parser.py's own functions, unrelated to_make_extractor/build_extra_extractors:is_available(2),parse_file_ast(5),_node_text(2),_extract_module_doc(11),_extract_python(1),_extract_js(2). None of these are false-survivor artifacts of this issue's root cause — they are real test gaps in a ~250-line file spanning six independent functions, each likely needing several new precise tree-sitter-AST assertions. This is genuinely outside #269's stated acceptance criteria (which name only the twoast_extractor_registry.pyfunctions) and outside this PR's shipped[tool.mutmut]scope (the committed scope staysjson_native.py; my diagnostic widening to includeast_parser.pywas reverted after each run). Evidence:/tmp/mut_run2.log-equivalent local run, mutant names listed in this PR body's Verification section above. Leaving this for the coordinator to decide whether it warrants its own follow-up PR (not filing an issue per this run's explicit instruction).