fix(custodian): close the vulture fail-open in the pre-push gate - #493
Open
ProtocolWarden wants to merge 5 commits into
Open
fix(custodian): close the vulture fail-open in the pre-push gate#493ProtocolWarden wants to merge 5 commits into
ProtocolWarden wants to merge 5 commits into
Conversation
The gate reported "0 findings, clean" while a Windows box running a newer Custodian reported hundreds. Windows was the correct side: the green gate was a FALSE CLEAN and had been for as long as the pin held. Three things lined up to hide it: 1. .custodian/config.yaml sets tools.vulture: true — the detector is meant to run. 2. pyproject.toml never declared vulture, so `uv pip install -e .[dev]` never installed it. The fleet venv has no vulture, and none is on PATH. 3. The pin d6ba8ab PREDATES Custodian 261bbb5 "fix(vulture): put paths before options, and stop reading a failed run as clean". On that pin the adapter built `vulture <src> --min-confidence=N <tests>`, which vulture's argparse rejects (exit 2, empty stdout) — and the empty output read as "no dead code". So even with vulture installed the pinned adapter could not emit a finding: the invocation was malformed and the failure was swallowed. The detector has never run. Bump the pin to 7a780b7 (origin/main, contains 261bbb5) and declare vulture==2.16 beside the existing ruff/ty pins. These must land together — after 261bbb5 a missing vulture fails LOUDLY, so bumping alone would red the gate on "vulture not found". Set tools.vulture_min_confidence: 80 explicitly. Custodian's adapter registry falls back to 60 while its own config loader documents 80 as the default; inheriting whichever wins is how this stays surprising. On this repo 60 yields 621 findings (essentially all UNUSED_METHOD heuristics), 80 yields 32, all at 100% confidence. Of those 32, 22 are names an external contract forces us to accept — the __exit__ protocol, pytest's pytest_sessionfinish hookspec, fixtures requested purely for a side effect, lambda stubs mirroring the callee they replace — plus two compat shims the source already documents as deliberate. Those go in a new .vulture_whitelist.py, which Custodian's adapter picks up automatically. It matches on bare NAME, not location, so it is kept minimal with a justification per entry. The remaining 10 are real and deliberately NOT whitelisted: * observer/cli.py x8 — --format, --skip-validation, --output, --filter-status, --signals-only, --input, --validate-after, --keep are declared as typer options and never read. `layers` and `full` in the same command ARE read, which is what makes these stand out rather than look like a vulture blind spot. `--format yaml` silently yields JSON. * pr_review_watcher/main.py:2508,2543 — `pending_checks` threaded through and never used. CONSEQUENCE: this turns the gate red on those 10 until they are triaged. That is the intended effect — it was green by accident. Whether each observer flag should be wired up or deleted is product work and is not guessed at here. Tracked in .console/backlog.md under Up Next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Clears the 10 genuine findings that were holding the pre-push gate red after the vulture fail-open was closed. `custodian-multi --fail-on-findings` now exits 0 under a custodian that actually runs vulture — it reported 621 before. Corrects the earlier claim that "`layers` and `full` in the same command ARE read, so parameter-usage detection is working". That was wrong. cmd_observe_and_validate's body reads ONLY `quiet`; `layers` and `full` are equally unread there and escaped the report because vulture matches on bare NAME and those names are used by other commands in the tree. The finding was bigger than 8 stray flags: FOUR commands (observe-and-validate, compare, import, cleanup) are stubs whose entire option lists are ignored, while --help and two user guides advertised them as working. Per the "implement or delete" bar: * The four stubs are documented as PLANNED (STAGE0_CLI_SPECIFICATION.md "Secondary Commands (Planned Future)"; both guides carry "not yet implemented" notes), so deleting the commands was wrong — but keeping parameters they discard was too. Each stub now takes --quiet only. The planned interface stays in the spec, which is where a design belongs; a half-declared signature that typer advertises in --help is not a spec, it is a promise the command breaks. Dropping `import`'s required input path is deliberate: accepting a file and discarding it is indistinguishable from importing it and failing. * `list --filter valid|invalid` deleted. It could never have worked — the listing walks snapshot directories and never loads or caches a validation status to filter on (its observed_at column is a literal "—"). Implementing it needs the caching layer the help text presumed. Also fixed in cmd_cleanup, and NOT one of the vulture findings: it exited EXIT_SUCCESS while deleting nothing, so a scheduled `cleanup --days 30` reported success and silently retained every snapshot, with no way for a caller to tell a working cleanup from a stub. Now exits non-zero. Same fail-open shape as the vulture bug — a green signal that means nothing — which is why it was worth fixing in place. The guide's two runnable cleanup examples are gone and both option tables are relabelled "Planned Options (not accepted today)". Removed `pending_checks` from _update_check_history and _should_escalate_ci_wait plus 16 call sites; neither body read it. The tests passed pending_checks=["audit"] in two places, implying behaviour that could not exist — those assertions were passing for the wrong reason. test_unimplemented_stubs_reject_planned_flags pins the intent: each stub must REJECT the planned flags rather than swallow them, so an ignored option cannot be re-added without a failing test. Nothing was added to .vulture_whitelist.py — every finding was resolved by removing dead code, not by suppressing the report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…vulture-fail-open
…roject The custodian-audit workflow hardcodes its own Custodian SHA, separate from pyproject's, and its comment requires the two be bumped together. This PR moved pyproject d6ba8ab -> 7a780b7 without it, so CI would have kept installing the old adapter and the vulture fail-open would have survived in the one place it matters most — the required gate. d6ba8ab predates Custodian 261bbb5, which fixed the adapter building `vulture <src> --min-confidence=N <tests>`, an argument order vulture's argparse rejects (exit 2, empty stdout) that was then read as "no dead code". That is why #492 observed "vulture was clean in CI" while vulture was installed and this repo in fact had 621 findings at the default confidence: every run failed and every failure was swallowed. It is the same vacuous-green mode the adjacent step already warns about for a missing ruff. Also drops the unpinned `pip install vulture`. vulture is a dev dependency now, so `.[dev]` pins it (2.16) beside ruff and ty — removing the moving part rather than relocating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…vulture-fail-open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gate was lying
The Custodian pre-push gate reported
0 findings, cleanon this repo, while a Windows box running a newer Custodian reported hundreds. Windows was the correct side. The green gate was a false clean, and had been for as long as the pin has been in place.Three things had to line up to hide it:
.custodian/config.yamlsetstools.vulture: true— the detector is meant to run.pyproject.tomlnever declaredvulturein the dev extra, souv pip install -e .[dev]never installed it. The fleet venv has no vulture and none is on PATH.d6ba8abpredates Custodian261bbb5— "fix(vulture): put paths before options, and stop reading a failed run as clean". On that pin the adapter builtvulture <src> --min-confidence=N <tests>, which vulture's argparse rejects (exit 2, empty stdout), and the empty output was read as "no dead code".So even had vulture been installed, the pinned adapter could not have produced a finding — the invocation itself was malformed and the failure was swallowed. The detector has never once run.
The fix
7a780b7(origin/main, contains261bbb5)vulture==2.16alongside the existingruff/typinsThese must land together: after
261bbb5a missing vulture fails loudly, so bumping the pin alone would red the gate on "vulture not found".Threshold set explicitly to
tools.vulture_min_confidence: 80. Custodian's adapter registry falls back to 60 while its own config loader documents 80 as the intended default — relying on whichever wins is how this stays surprising.--min-confidenceUNUSED_METHOD/attribute heuristics)What's whitelisted, and what deliberately isn't
Of the 32, 22 are names an external contract forces us to accept: the
__exit__protocol, pytest'spytest_sessionfinishhookspec, fixtures requested purely for a side effect, and lambda stubs that must mirror the callee they replace — plus two compat shims the source already documents as deliberate (max_rewrite_attemptscarries# noqa: ARG002 — kept for signature compat;queue_thresholdcarries# kept for config compat, not used in logic).Those go in a new
.vulture_whitelist.py, which Custodian's adapter picks up automatically when present. It matches on bare name, not location, so it's kept minimal with a justification per entry.The remaining 10 are real and are deliberately not whitelisted:
observer/cli.py×8 —--format,--skip-validation,--output,--filter-status,--signals-only,--input,--validate-after,--keepare declared astyper.Option(...)and never read in the body.layersandfullin the same command are read, which is what makes these stand out rather than look like a vulture blind spot. User-visible: passing--format yamltoday silently produces JSON.pr_review_watcher/main.py:2508,2543—pending_checksthreaded through two call sites and never used.Consequence
Merging turns the gate red until those 10 are triaged. Whether each observer flag should be wired up or deleted is product work and is not guessed at here — tracked in
.console/backlog.mdunder "Up Next".If you'd rather not take a red gate right now, hold this PR and I'll fix the 10 first; the fail-open has been in place a long time and a few more days won't change anything.
Separate finding, not fixable from this repo
5ef3f0f fix(adapters): make find_tool's venv-first preference work on Windowsexists only in the local Custodian checkout (branchclaude/reconcile-june-2026-08-03, upstream gone).origin/mainis at7a780b7, so that fix can't be pinned. Until it's pushed, a Windows Custodian run resolves linters off PATH rather than a venv — that produced 1222 phantom ruff findings earlier today (ruff 0.16's widened default rule set vs OC's pinned 0.15.13) until the local checkout picked the commit up mid-session. Also tracked in the backlog.Verification
The 10 are exactly the genuine set listed above. Every whitelisted name was inspected at its source location and classified before being added.
🤖 Generated with Claude Code