Skip to content

feat(pr-review): let the reviewer report omitted companion changes - #10

Merged
Troublor merged 1 commit into
mainfrom
feat/pr-review-omission-findings
Aug 3, 2026
Merged

feat(pr-review): let the reviewer report omitted companion changes#10
Troublor merged 1 commit into
mainfrom
feat/pr-review-omission-findings

Conversation

@Troublor

@Troublor Troublor commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Why

A real miss prompted this. megaeth-labs/dist-docs#52 is a release-notes PR that announces a hardfork, tells operators to update their genesis file, and never adds rex6Time to mainnet/genesis.json or testnet/genesis.json. Every prior hardfork release in that repo updated genesis in the same commit. The reviewer ran a full review with the pre-mortem enabled and returned ✅ Review clean.

It was not a judgment error. The reviewer was structurally incapable of reporting it:

  1. The compiler rejected the finding outright. compile_review raised FINDING_ANCHOR_INVALID for any finding whose path was not in full_pr_paths. That is not a demotion to body-only — PipelineError aborts the compile step, so a correct omission finding would have destroyed the entire review that carried it.
  2. The prompt never invited a look past the diff. Contract item 7 said to source anchors from the full PR diff; nothing said a full review may read the repository around it. A full review behaved like a diff review.
  3. The pre-mortem refuted omissions by construction. Its Evidence Verifier required that "the finding's anchor line still belongs to the current PR diff" — true of no omission, ever. And no rubric lens or pre-mortem dimension named the failure mode, so nothing generated the candidate in the first place.

What changed

review_pipeline.py — a finding may name a file the PR did not touch, provided that file exists in the checkout. It has no RIGHT side in the diff, so finding["inline"] is already False and the existing renderer puts it in the review body; no new output path. The anti-hallucination guard that rejecting unchanged paths used to provide is preserved as a file-existence check, and tracked_file_exists confines resolved paths to the workspace so a model-supplied ../ or absolute path cannot read outside it. compile_review takes an injectable repo_root for testability.

rubric.md — new lens 4, Completeness — what the diff leaves undone: derive the companion set from the stated intent rather than the diff, the recurring shapes (a value declared in one file and consumed in another, docs promising behavior the data lacks, an enum arm added at one site but not its siblings, a version that must agree across files), and git log on comparable earlier commits as the cheapest way to learn which files move together in this repo. Closes with the point that an unanchorable finding is not a weaker finding.

premortem.md — dimension 1 gains the stale-companion incident shape, where nothing in the diff itself looks wrong; the Evidence Verifier gets the matching exception to its anchor rule.

action.yml — contract item 1: the frozen scope bounds which changes may be raised, not what may be read. Item 7: how to anchor an omission, and not to downgrade it to an open question once verified. Strong-tier turn budget 36 → 44 to pay for the extra reading.

review-output.schema.jsonpath and line descriptions say how to anchor an omission.

debug_logs input (default off) — both analysis steps hardcoded show_full_output: false. Diagnosing this miss meant inferring from a job log that shows no tool calls at all; there was no way to tell which files the reviewer opened and which it never read. Turning this on is the difference between answering that question in a minute and guessing.

Testing

python3 -m unittest discover — 83 pass. New coverage: an omission finding on an unchanged-but-present file compiles to a body-only finding with verdict findings; a path absent from the checkout still fails loudly; a ../ path pointing at a real file outside the workspace is rejected.

Editing these files changes pipeline_version/rubric_version, so open PRs across the org get one full re-review on the next event rather than an incremental one. That is the intended behavior here — the new lens should apply to work already in flight.

🤖 Generated with Claude Code

A change can be correct on every line it touches and still ship broken
because a file it depends on was never updated. The reviewer could not
report that class of defect at all, for three compounding reasons:

- The compiler rejected any finding whose path was outside the PR's
  changed files, raising FINDING_ANCHOR_INVALID. That is not a demotion
  to body-only — it aborts the whole compile step, so a correct omission
  finding would have destroyed the review that carried it.
- The prompt told the model to source anchors from the full PR diff and
  said nothing about reading past it, so a full review behaved like a
  diff review.
- Neither the rubric nor the pre-mortem dimensions named the failure
  mode, and the pre-mortem's evidence verifier explicitly required every
  anchor to belong to the current diff — which refutes every omission
  finding by construction.

Now the compiler accepts a path that exists in the checkout even when the
PR did not touch it, and routes it into the review body, where findings
without an inline anchor already go. The existing anti-hallucination
guard is preserved as a file-existence check, with resolved paths
confined to the workspace so a `../` or absolute path cannot read out of
it. Rubric lens 4 asks what a complete version of the change touches and
how to check it against repository history; the pre-mortem gains the
stale-companion incident shape; and the contract states that the changed
files bound which changes may be raised, not what may be read.

The strong tier's turn budget goes 36 -> 44 to pay for that extra
reading.

Also adds a `debug_logs` input, default off. Both analysis steps hardcoded
`show_full_output: false`, which makes "why did the review miss this?"
unanswerable after the fact — the job log shows no tool calls, so there is
no way to tell which files the reviewer opened and which it never read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Troublor
Troublor merged commit 6aac66b into main Aug 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant