Skip to content

Check Changeset concludes success when it measured nothing, and no edited trigger ever re-reads a Clause-②: line added after the last push — a correct seat action leaves the level axis unjudged behind a green check #16776

Description

@os-zhuang

Split off #16361 by the triage seat. #16361's body is about the level rule's predicate scope (a PR-scoped declaration applied at package scope, producing a false red). This card is the two defects its filer added in comment 5560230929, which point the other way — at a green that judged nothing — and which the filer explicitly left for triage to place ("Whether that belongs on this card or its own is triage's call").

They are one card because defect B is the mechanism that makes defect A reachable in the ordinary course of carrier discipline.

All measurements below are the #15963 implementer's, taken by running the gate's own script offline against --event payloads, and the two live PRs of that round. ⚠️ Not re-driven by triage — see Boundaries.


Defect A — NOT MEASURED and PASSED are the same check-run conclusion

First, the credit, because it narrows the card: the gate is honest when blind. Measured, three states on commit 273247e56f:

event payload exit what the step log says
Clause-②: yes in the body, no carrier label 1 "…grades a package it grew patch" · declaration line: Clause-②: yes — fires on the body line alone
neither carrier nor body line 0 LEVEL AXIS: NOT MEASURED
(fixed head) body line, no carrier 0 ✓ LEVEL AXIS: … no package … is graded patch

⇒ it prints LEVEL AXIS: NOT MEASURED rather than inventing a green verdict. ⛔ So this is not "the gate lies", and the filer corrected their own earlier wording to say so.

The defect is one layer out. The check-run conclusion is success in both the "judged and passed" and the "had nothing to judge" cases. A reader who opens the step log can tell them apart. The merge queue, the required-check set, and every summary view read the conclusion, not the log. ⇒ at the only layer that gates a merge, a gate that measured nothing is indistinguishable from a gate that passed.

That is this repo's own stated failure class, applied to its own CI: a reading that cannot fail is indistinguishable from one that passed. Here the reading did not merely not-fail — it did not happen, and said so somewhere nobody with merge authority is reading.

Defect B — the durable declaration is never re-read until someone pushes

.github/workflows/pr-automation.yml triggers on [opened, synchronize, reopened, labeled, unlabeled], its Check Changeset job skips label events, and there is no edited trigger.

A PR-body edit that adds the Clause-②: line is never read by the gate until the next synchronize — i.e. until someone pushes a commit.

Measured live on both PRs of that round:

⭐ Why the two are one card: the failure needs no mistake

Compose them and the failure mode is reached by a seat doing only correct things, in a defensible order:

  1. at-tier review PASSes;
  2. the review seat strips needs:contract-review — correct carrier discipline;
  3. the seat adds the durable Clause-②: line to the PR body so the declaration outlives the carrier — exactly the remedy [finding] Check Changeset's clause-② level rule is PR-scoped but the fact it judges is package-scoped — a PR that widens package A is refused for grading package B patch, where B only received a comment #16361 proposes;
  4. nothing else is pushed.

Defect B means the gate never re-reads, so it still sees neither carrier nor line ⇒ LEVEL AXIS: NOT MEASURED; Defect A means that concludes success. The level rule — the maintainer ruling of 2026-09-04, decision batch #35, on #15294 — is simply never applied to that PR, and every merge-authority surface shows green.

⇒ ⭐ This also means #16361's proposed remedy does not work on its own. Requiring a body Clause-②: line is defeated by defect B whenever the line is added after the last push. The remedy needs either an edited trigger, or the line to be present from the PR's first push.

Directions (⛔ none chosen — recording what the measurements narrow to)

For A:

  • NOT MEASURED should not conclude success (neutral, or a failure requiring an explicit opt-out); or
  • it must be visible in the check-run's own output/summary, not only in step logs, so the conclusion carries the distinction to the surfaces that read it.

For B:

  • add an edited trigger (⚠️ note it fires on every body edit, including title/description churn — cost and noise want measuring); or
  • the filer's own cheapest reading: the Clause-②: line belongs in the PR body at creation time, written by the dev opening the PR rather than patched in later by the seat clearing the carrier. That also matches where the declaration comes from — the claim comment — rather than where it was being retrofitted.

No tolerance, no allowlist, no comment-only heuristic — carried over from #16361's body and it applies here too.

Priority reasoning (triage)

p2, not p1, and the capping fact comes from the round's own reviewer: all 69 packages sit in one fixed group, with 120 minor / 125 patch pending, so the released bump is set by the highest pending changeset regardless. ⇒ a level axis that goes unjudged on one PR does not, today, reliably ship a widened public surface inside a patch release.

Above p3 because the thing that goes unmeasured is a maintainer ruling, the surface that hides it is the one the merge queue reads, and reaching it requires no mistake by anyone.

Boundaries

Related

https://claude.ai/code/session_01SwJQDFKe8tVit3BXQ9EfR5

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions