From 9cf8013ae9e4d8c8dc56ab15dda8d0fdcedf25db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Tue, 11 Aug 2026 17:22:54 +0200 Subject: [PATCH 1/2] ci: unstarve the post-merge gates and alert when one goes dark (#7856) Ten heavy gates ran on `push: branches: [main]`. Fourteen workflows fired on every merge, ~29 jobs each time, against a repo that runs ~9 jobs concurrently. At 58 merges/day demand outran drain, the queue grew without bound, and the `main` runs -- the ones that gate nothing and that nobody watches -- aged out. `gc-ratchet` had 22 of its last 25 `main` runs queued (oldest 9h18m) and had not succeeded on `main` since 2026-08-09, across five collector-touching merges. The constraint is NOT macOS capacity, which was the natural reading. At the moment of measurement the queue held 45 `ubuntu-latest` jobs against 14 `macos-14`, and `zizmor` (ubuntu) was queued in the same second as `gc-ratchet` (macOS). Rebalancing pools cannot help; only cutting total demand can. Nor is it the `concurrency:` blocks, which are already correct and twice-repaired (#7205) -- they are left untouched. Post-merge arm becomes a staggered six-hourly sweep plus release tags. That removes 19 jobs from every merge: ~1,100 job-starts/day replaced by ~76, a 93% cut on this slice. Pull-request arms are unchanged, so every PR is still measured before it can merge, and no probe, threshold or baseline moved. `gate-freshness.yml` + `scripts/check_gate_freshness.py` close the hole that let this last two days: an empty result set is indistinguishable from a healthy one nobody checked. It fails when a gate has no successful post-merge `main` run inside its budget, and maintains one self-closing sticky issue. The checker is sabotage-tested, not merely exercised -- `--self-test` plants a stale gate, a gate with no successful run, and a gate whose only recent successes are `pull_request` runs, which is the exact shape that made `gc-root-dominance` look healthy while its `main` arm was dark. Its first live run already found two things outside the issue's scope: `llvm-inprocess` has been dark for 171.8h, and `security-audit` -- a required context, left on every merge -- is itself at 17.4h. --- .github/workflows/auto-opt-app-patterns.yml | 10 +- .github/workflows/eh-transport.yml | 20 +- .github/workflows/gate-freshness.yml | 81 ++++ .github/workflows/gc-moving-witnesses.yml | 10 +- .github/workflows/gc-native-roots.yml | 12 +- .github/workflows/gc-parse-churn-gate.yml | 10 +- .../workflows/gc-ptr-shape-off-witness.yml | 10 +- .github/workflows/gc-ratchet.yml | 29 +- .github/workflows/gc-root-dominance.yml | 13 +- .github/workflows/llvm-inprocess.yml | 20 +- .github/workflows/tls-budget.yml | 12 +- docs/src/SUMMARY.md | 1 + docs/src/testing/ci-gate-scheduling.md | 175 ++++++++ scripts/check_gate_freshness.py | 392 ++++++++++++++++++ scripts/gate_freshness.json | 80 ++++ 15 files changed, 863 insertions(+), 12 deletions(-) create mode 100644 .github/workflows/gate-freshness.yml create mode 100644 docs/src/testing/ci-gate-scheduling.md create mode 100755 scripts/check_gate_freshness.py create mode 100644 scripts/gate_freshness.json diff --git a/.github/workflows/auto-opt-app-patterns.yml b/.github/workflows/auto-opt-app-patterns.yml index 5e89f4777b..05dac73f1c 100644 --- a/.github/workflows/auto-opt-app-patterns.yml +++ b/.github/workflows/auto-opt-app-patterns.yml @@ -47,8 +47,16 @@ name: Auto-Optimize App Patterns on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "22 */6 * * *" push: - branches: [main] + tags: ["v*"] workflow_dispatch: permissions: diff --git a/.github/workflows/eh-transport.yml b/.github/workflows/eh-transport.yml index ad8d9f0087..07e0047706 100644 --- a/.github/workflows/eh-transport.yml +++ b/.github/workflows/eh-transport.yml @@ -28,8 +28,17 @@ name: eh-transport # `changes` job: a job skipped by `if:` still reports a check run. on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "32 */6 * * *" push: - branches: [main] + tags: ["v*"] + workflow_dispatch: concurrency: group: eh-transport-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} @@ -55,7 +64,14 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: | - if [ "${{ github.event_name }}" = "push" ]; then + # Anything that is not a pull request (the scheduled `main` sweep, a + # release tag, a manual dispatch) measures unconditionally: there is no + # PR file list to filter on, and #7856's starvation is precisely what + # happens when a post-merge arm quietly declines to run. Testing for + # `= "push"` here was correct only while `push: branches: [main]` was + # the post-merge trigger; under the schedule it would fall through to + # the PR branch, dereference an empty PR number and fail the step. + if [ "${{ github.event_name }}" != "pull_request" ]; then echo "relevant=true" >> "$GITHUB_OUTPUT" exit 0 fi diff --git a/.github/workflows/gate-freshness.yml b/.github/workflows/gate-freshness.yml new file mode 100644 index 0000000000..4b74dbc660 --- /dev/null +++ b/.github/workflows/gate-freshness.yml @@ -0,0 +1,81 @@ +name: Gate Freshness + +# Alerts when a post-merge gate stops producing successful `main` runs. +# +# THE FAILURE THIS EXISTS FOR (#7856): every heavy gate in this repo produced zero +# results on `main` for over two days. They were not failing and not cancelled -- +# they never reached a runner, because fourteen workflows enqueued ~29 jobs on every +# one of 58 daily merges against a repo that runs ~9 jobs at a time. Five +# collector-touching PRs merged inside that window and #7843's seven genuinely-red +# rows landed unseen. +# +# What made it last two days is the part worth engineering against: **an empty result +# set looks exactly like a healthy one that nobody has looked at.** Rescheduling the +# gates (docs/src/testing/ci-gate-scheduling.md) does not fix that on its own -- a +# cron that quietly stops firing fails in precisely the same way. This workflow is the +# detector for both, and it is the reason the next occurrence should be noticed in +# hours rather than days. +# +# DELIBERATELY NOT A REQUIRED CONTEXT. It reports on the health of other gates; it +# must never be able to block a PR. It is also deliberately cheap -- one short +# ubuntu job -- so that the alarm cannot be starved by the condition it is alarming +# about. + +on: + schedule: + # Every two hours, off the top of the hour (GitHub's scheduler is most contended + # at :00 and most likely to delay a run there). The tightest budget in + # scripts/gate_freshness.json is 12h, so this samples ~6x per budget window. + - cron: "5 */2 * * *" + # Validate the checker itself when it changes. Self-test only -- see the step + # guards below; a PR must not go red merely because `main`'s gates are behind. + pull_request: + paths: + - scripts/check_gate_freshness.py + - scripts/gate_freshness.json + - .github/workflows/gate-freshness.yml + workflow_dispatch: + +permissions: + contents: read + +concurrency: + # One sweep at a time. Unlike the gates this watches, coalescing is correct here: + # the freshness verdict is a function of "now", so a superseded run had nothing + # unique to say. PR runs supersede themselves; scheduled runs queue. + group: gate-freshness-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + gate-freshness: + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + # Listing workflow runs, and maintaining the single sticky alert issue. + actions: read + issues: write + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + + # Structural failure first, and unconditionally: prove the detector can still + # say no before trusting a green verdict from it. This plants a stale gate, a + # gate with no successful run at all, and -- the trap that made + # `gc-root-dominance` look healthy in #7856 -- a gate whose only recent + # successes are `pull_request` runs, then asserts the verdict for each. + # + # A green self-test means the detector works, not that nothing was tried. + - name: Self-test the freshness checker (can this gate still fail?) + run: python3 scripts/check_gate_freshness.py --self-test + + # The live check does not run on pull requests. A PR that merely touches this + # checker must not go red because `main`'s gates are behind -- that would make + # the alarm a merge blocker, which it is explicitly not. + - name: Check post-merge gate freshness + if: github.event_name != 'pull_request' + env: + GH_TOKEN: ${{ github.token }} + GITHUB_REPOSITORY: ${{ github.repository }} + run: python3 scripts/check_gate_freshness.py diff --git a/.github/workflows/gc-moving-witnesses.yml b/.github/workflows/gc-moving-witnesses.yml index 5d8113dc19..6c12b7a603 100644 --- a/.github/workflows/gc-moving-witnesses.yml +++ b/.github/workflows/gc-moving-witnesses.yml @@ -108,8 +108,16 @@ name: GC Moving Witnesses on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "12 */6 * * *" push: - branches: [main] + tags: ["v*"] workflow_dispatch: permissions: diff --git a/.github/workflows/gc-native-roots.yml b/.github/workflows/gc-native-roots.yml index a0c8bf679b..a652fbf2d7 100644 --- a/.github/workflows/gc-native-roots.yml +++ b/.github/workflows/gc-native-roots.yml @@ -111,8 +111,18 @@ on: # filter would mean the job never runs at all — CLAUDE.md's second way a gate # cannot fail. Cancellation is deliberately NOT set here: a `main` run that # gets cancelled by the next merge is the third way. + # + # And the FOURTH way is what `push: branches: [main]` turned out to be here: + # STARVED (#7856). Neither cancelled nor failing — simply never scheduled, + # because fourteen workflows enqueued ~29 jobs on every one of 58 daily merges + # against a repo that runs ~9 jobs at a time. The post-merge arm is now a + # staggered six-hourly sweep; the pull-request arm is unchanged, so every PR is + # still measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "37 */6 * * *" push: - branches: [main] + tags: ["v*"] pull_request: workflow_dispatch: diff --git a/.github/workflows/gc-parse-churn-gate.yml b/.github/workflows/gc-parse-churn-gate.yml index 03ab287d0f..1dcd7eb859 100644 --- a/.github/workflows/gc-parse-churn-gate.yml +++ b/.github/workflows/gc-parse-churn-gate.yml @@ -76,8 +76,16 @@ name: GC Parse-Churn Layout Gate on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "57 */6 * * *" push: - branches: [main] + tags: ["v*"] workflow_dispatch: permissions: diff --git a/.github/workflows/gc-ptr-shape-off-witness.yml b/.github/workflows/gc-ptr-shape-off-witness.yml index ec727b5654..d892fcd183 100644 --- a/.github/workflows/gc-ptr-shape-off-witness.yml +++ b/.github/workflows/gc-ptr-shape-off-witness.yml @@ -74,8 +74,16 @@ name: GC Ptr OFF-arm witness on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "47 */6 * * *" push: - branches: [main] + tags: ["v*"] workflow_dispatch: permissions: diff --git a/.github/workflows/gc-ratchet.yml b/.github/workflows/gc-ratchet.yml index 487fe24bf4..b4ec936d36 100644 --- a/.github/workflows/gc-ratchet.yml +++ b/.github/workflows/gc-ratchet.yml @@ -15,8 +15,35 @@ name: GC Ratchet on: pull_request: + # POST-MERGE ARM: a staggered six-hourly sweep of `main`, NOT one run per merge. + # + # `push: branches: [main]` is what took this gate dark for two days (#7856). + # Fourteen workflows fired on every merge, ~29 jobs each time, against a repo + # that runs ~9 jobs concurrently; at 58 merges/day demand outran drain and the + # queue grew without bound. On 2026-08-11 this workflow had 22 of its 25 most + # recent `main` runs sitting `queued` (oldest 9h18m) and its last successful + # `main` run was 2026-08-09 -- two days and five collector-touching merges + # earlier (#7799, #7809, #7812, #7834, #7839). #7843's seven genuinely-red rows + # landed inside that blind window. + # + # This is the THIRD variant of CLAUDE.md's "four ways a gate can be unable to + # fail": not `continue-on-error`, not missing from required contexts, not + # cancelled -- STARVED. A gate that runs four times a day and COMPLETES is + # worth more than one that runs on every merge and never does. + # + # It is NOT the `concurrency:` block below (already correct, twice-repaired -- + # see its comment) and it is NOT macOS capacity: at the moment of measurement + # the queue held 45 `ubuntu-latest` jobs against 14 `macos-14`. Rebalancing + # pools cannot help; only cutting total job demand can. + # + # ***DO NOT RESTORE `push: branches: [main]` HERE.*** Full measurement, the two + # #7856 claims that did not survive checking, and the cost this trades away: + # docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "7 */6 * * *" push: - branches: [main] + # Releases stay individually gated; tags are rare, so this costs nothing. + tags: ["v*"] workflow_dispatch: permissions: diff --git a/.github/workflows/gc-root-dominance.yml b/.github/workflows/gc-root-dominance.yml index 551c223cbd..d8b96be8cd 100644 --- a/.github/workflows/gc-root-dominance.yml +++ b/.github/workflows/gc-root-dominance.yml @@ -61,8 +61,19 @@ name: GC Root Dominance on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. Note this workflow runs `macos-14`, not `ubuntu-latest`; #7856's + # claim that it was unaffected was reading its PR arm, which drains because PR + # runs supersede each other, while its `main` arm queued like all the rest. + # ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "17 */6 * * *" push: - branches: [main] + tags: ["v*"] workflow_dispatch: permissions: diff --git a/.github/workflows/llvm-inprocess.yml b/.github/workflows/llvm-inprocess.yml index 2c572a7459..7eb5208bb6 100644 --- a/.github/workflows/llvm-inprocess.yml +++ b/.github/workflows/llvm-inprocess.yml @@ -16,8 +16,17 @@ name: llvm-inprocess # run (conclusion: skipped), which branch protection accepts. on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "42 */6 * * *" push: - branches: [main] + tags: ["v*"] + workflow_dispatch: concurrency: group: llvm-inprocess-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} @@ -46,7 +55,14 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: | - if [ "${{ github.event_name }}" = "push" ]; then + # Anything that is not a pull request (the scheduled `main` sweep, a + # release tag, a manual dispatch) measures unconditionally: there is no + # PR file list to filter on, and #7856's starvation is precisely what + # happens when a post-merge arm quietly declines to run. Testing for + # `= "push"` here was correct only while `push: branches: [main]` was + # the post-merge trigger; under the schedule it would fall through to + # the PR branch, dereference an empty PR number and fail the step. + if [ "${{ github.event_name }}" != "pull_request" ]; then echo "relevant=true" >> "$GITHUB_OUTPUT" exit 0 fi diff --git a/.github/workflows/tls-budget.yml b/.github/workflows/tls-budget.yml index ca79934df2..eb0be91407 100644 --- a/.github/workflows/tls-budget.yml +++ b/.github/workflows/tls-budget.yml @@ -54,8 +54,18 @@ name: TLS Budget on: pull_request: + # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. + # `push: branches: [main]` starved this gate and nine others for two days + # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job + # concurrency ceiling. The pull-request arm is unchanged -- every PR is still + # measured. The macOS arm below is irreducible (`_tlv_get_addr` is a Mach-O + # artefact), so moving it to Linux was never an option; cutting demand was. + # ***DO NOT RESTORE `push: branches: [main]`.*** + # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md + schedule: + - cron: "27 */6 * * *" push: - branches: [main] + tags: ["v*"] workflow_dispatch: permissions: diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 851d980de3..87d39db6fa 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -150,6 +150,7 @@ - [Test Registration (dark tests)](testing/test-registration.md) - [Geisterhand (UI Fuzzer)](testing/geisterhand.md) - [Node Compatibility Matrix](testing/node-compat-matrix.md) +- [CI Gate Scheduling](testing/ci-gate-scheduling.md) # CLI Reference diff --git a/docs/src/testing/ci-gate-scheduling.md b/docs/src/testing/ci-gate-scheduling.md new file mode 100644 index 0000000000..c9ab683ebb --- /dev/null +++ b/docs/src/testing/ci-gate-scheduling.md @@ -0,0 +1,175 @@ +# CI gate scheduling: why the heavy gates sweep `main` instead of gating every merge + +This page explains one deliberate choice: **the expensive post-merge gates run on a +staggered six-hourly schedule against `main`, not once per merge.** Their +pull-request arm is untouched — every PR is still measured before it can merge. + +If you are about to "fix" one of those workflows by putting `push: branches: [main]` +back, read this first. That trigger is what broke them. + +## The failure: starvation (#7856) + +On 2026-08-11, **every** heavy gate in this repo had produced zero results on `main` +for over two days. They were not failing and not cancelled. They never reached a +runner. + +The measurement that matters is not "macOS is slow". It is the ratio between how +many jobs a merge enqueues and how many the repo can run at once: + +| quantity | measured 2026-08-11 15:07 UTC | +|---|--:| +| workflow runs created that day (by 15:07) | **600** (392 `pull_request` + 208 `push`) | +| pushes to `main` per day | **10** (08-09) → **32** (08-10) → **58** (08-11) | +| workflows triggering on `push: main` | **14**, totalling ~**29 jobs per merge** | +| jobs running repo-wide, right then | **9** (6 `ubuntu-latest`, 3 `macos-14`) | +| runs queued repo-wide, right then | **100+** | +| `gc-ratchet` `main` runs queued, of its last 25 | **22**, oldest waiting **9h18m** | +| `gc-ratchet` last successful `main` run | **2026-08-09**, two days earlier | + +Reproduce the core of it with: + +```bash +# What is actually running, repo-wide, at the job level: +gh api "repos/PerryTS/perry/actions/runs?status=in_progress&per_page=100" \ + -q '.workflow_runs[].id' \ +| while read id; do + gh api "repos/PerryTS/perry/actions/runs/$id/jobs?per_page=100" \ + -q '.jobs[] | select(.status=="in_progress") | (.labels|join(","))' + done | sort | uniq -c + +# How deep the queue is, and on which pools: +gh api "repos/PerryTS/perry/actions/runs?per_page=100" -q '.workflow_runs[].id' \ +| while read id; do + gh api "repos/PerryTS/perry/actions/runs/$id/jobs?per_page=100" \ + -q '.jobs[] | select(.status=="queued") | (.labels|join(","))' + done | sort | uniq -c +``` + +Demand outran drain, the queue grew without bound, and the oldest entries — the +`main` runs, which are exactly the ones that gate nothing and therefore nobody is +watching — aged out. + +## What this is NOT + +**It is not the concurrency bug, and it is not macOS runner availability.** Both +were the obvious reading, and both are wrong. Getting this right matters, because +each wrong reading has a "fix" that would make things worse. + +**Not the concurrency block.** `gc-ratchet.yml`'s `concurrency:` comment records two +prior attempts (#7205): a shared group with unconditional `cancel-in-progress` +cancelled three consecutive `main` runs, and scoping `cancel-in-progress` to pull +requests did not fix it either, because GitHub allows at most one *pending* run per +group. Keying the group on `github.sha` for push events **did** fix cancellation. +The failure mode simply moved: runs stopped cancelling each other and started +queueing forever instead. **Those blocks are correct. Do not "fix" them again.** + +**Not macOS capacity.** This was the natural inference — the gates that went dark +are the macOS ones — but the job-level numbers refute it. At the moment of +measurement the queue held **45 `ubuntu-latest` jobs against 14 `macos-14` jobs**, +and `zizmor` (`ubuntu-latest`) was queued in the very same second as `gc-ratchet` +(`macos-14`). Ubuntu was starved harder in absolute terms. + +Two specific claims in #7856 do not survive checking, and are recorded here so the +next person does not re-derive them: + +- **`gc-root-dominance` does not run on `ubuntu-latest`.** It runs `macos-14`, in + two jobs. Its healthy-looking run count was pull-request runs, which drain because + they supersede each other; its `main` arm was queued like all the others. The + ubuntu-vs-macOS contrast that localised the problem to macOS was comparing a PR + arm against a `main` arm, not Linux against Darwin. +- **Moving `gc-ratchet` to Linux is not available as a remedy.** Its baseline is + captured under the `darwin-arm64` platform key and the checker *refuses* a + platform mismatch rather than comparing numbers that are not comparable. Moving it + would turn the gate red, not relieve it. `tls-budget`'s macOS arm is likewise + irreducible: `_tlv_get_addr` is a Mach-O artefact and the measurement does not + exist elsewhere. + +The consequence of both corrections is the same: **rebalancing pools cannot help.** +Only reducing total job demand can. That is what this change does. + +## The change + +For ten heavy gates, the post-merge arm became a staggered six-hourly sweep plus +release tags: + +```yaml +on: + pull_request: # unchanged — every PR is still measured + schedule: + - cron: "7 */6 * * *" # staggered; see the table below + push: + tags: ["v*"] # releases stay individually gated + workflow_dispatch: +``` + +Cron minutes are staggered so the ten do not re-create the thundering herd they were +meant to relieve, and none sits at `:00`, where GitHub's scheduler is most contended +and most likely to delay a run: + +| workflow | cron | runner of the heavy job | +|---|---|---| +| `gc-ratchet` | `7 */6 * * *` | `macos-14` | +| `gc-moving-witnesses` | `12 */6 * * *` | `ubuntu-latest` | +| `gc-root-dominance` | `17 */6 * * *` | `macos-14` (×2) | +| `auto-opt-app-patterns` | `22 */6 * * *` | `ubuntu-latest` | +| `tls-budget` | `27 */6 * * *` | `macos-14` | +| `eh-transport` | `32 */6 * * *` | `macos-15` | +| `gc-native-roots` | `37 */6 * * *` | matrix | +| `llvm-inprocess` | `42 */6 * * *` | `macos-15` | +| `gc-ptr-shape-off-witness` | `47 */6 * * *` | `ubuntu-latest` | +| `gc-parse-churn-gate` | `57 */6 * * *` | `ubuntu-latest` | + +That removes **19 jobs from every merge** (counting matrix expansion). At the cadence +measured above that is ~1,100 job-starts/day of demand replaced by ~76 — a **93% cut +on this slice**, which is what lets the remaining queue drain. + +### What was deliberately left alone + +- **`security-audit`** is a *required* status context and stays on every merge. +- **`zizmor`** and **`cache-warm`** are single cheap ubuntu jobs; `cache-warm` is + what makes everything else fast. +- **`container-tests`** (6 jobs on every merge, ~350 job-starts/day) is the + next-largest lever but is not a GC gate; left for a maintainer decision. +- **Every gate's actual content** — no probe, threshold, baseline, or matrix cell + was touched. This change alters *when* the post-merge arm runs, nothing else. +- **The in-job relevance filters** were left where they are. Hoisting them to + `on.pull_request.paths` would save PR-side slots, but a path-filtered workflow + reports *no* status rather than a passing one, which can wedge a required context. + Not worth the risk here. + +### The cost, stated plainly + +**Attribution latency.** A regression that slips past the PR arm used to be pinned +to one commit; now the next sweep names it against a window of commits. Each sweep +prints the SHA it tested, so the window is `previous sweep SHA .. this sweep SHA` — +bisect within that. In exchange the gate produces an answer at all, which for the +two days before this change it did not. Four completed runs a day beat 58 that never +start. + +## Staleness alerting + +The starvation was silent *by construction*: an empty result set looks exactly like +a healthy one nobody checked. Rescheduling the gates does not fix that — a cron that +silently stops firing fails the same way. + +`gate-freshness.yml` runs hourly on `ubuntu-latest` and calls +`scripts/check_gate_freshness.py`, which asks the Actions API for each gate's most +recent **successful** non-PR run on the default branch and fails when it is older +than that gate's budget in `scripts/gate_freshness.json`. On failure it opens — or +updates, never duplicates — a single sticky issue, and closes it once every gate is +fresh again. + +```bash +python3 scripts/check_gate_freshness.py --self-test # proves it can still fail +python3 scripts/check_gate_freshness.py --dry-run # real API, no issue writes +``` + +Budgets are the schedule interval plus headroom for a 90-minute job and a queue that +is still draining. A gate whose budget you have to keep raising is a gate that is +still starving; raise the *capacity* or lower the *demand* instead. + +**The checker is sabotage-tested, not merely exercised.** `--self-test` plants a +stale gate, a fresh gate, a gate with no successful run at all, and a gate whose only +recent success is a `pull_request` run (the exact shape that made `gc-root-dominance` +look healthy while its `main` arm was dark), and asserts the verdict for each. A +green `--self-test` means the detector works, not that nothing was tried. diff --git a/scripts/check_gate_freshness.py b/scripts/check_gate_freshness.py new file mode 100755 index 0000000000..92c76d4df2 --- /dev/null +++ b/scripts/check_gate_freshness.py @@ -0,0 +1,392 @@ +#!/usr/bin/env python3 +"""Fail when a post-merge gate has not produced a successful `main` run recently. + +#7856: every heavy gate in this repo produced zero results on `main` for over two +days. They were not failing and not cancelled -- they never reached a runner, because +fourteen workflows enqueued ~29 jobs on every one of 58 daily merges against a repo +that runs ~9 jobs at a time. Five collector-touching PRs merged inside that window and +#7843's seven genuinely-red rows landed unseen. + +The reason it lasted two days is the part worth engineering against: **the failure is +silent by construction.** An empty result set looks exactly like a healthy one that +nobody has looked at. Rescheduling the gates +(docs/src/testing/ci-gate-scheduling.md) does not fix that -- a cron that quietly +stops firing fails in precisely the same way. This script is the detector for both. + +For each gate in scripts/gate_freshness.json it asks the Actions API for the most +recent SUCCESSFUL run on the default branch, and fails when that run is older than the +gate's budget. + +Two things it deliberately does NOT count as evidence of health: + + * **pull_request runs.** PR runs supersede each other and therefore drain even when + the queue is saturated. Counting them is exactly what made `gc-root-dominance` + look healthy in #7856 while its `main` arm had been dark for two days. + * **runs that merely exist.** `queued`, `in_progress` and `cancelled` runs are not + results. A gate with 22 queued `main` runs and no successful one is a dark gate, + which is the whole point. + +Usage: + python3 scripts/check_gate_freshness.py --self-test # prove it can still fail + python3 scripts/check_gate_freshness.py --dry-run # real API, no issue writes + python3 scripts/check_gate_freshness.py # CI gate +""" + +from __future__ import annotations + +import argparse +import datetime as _dt +import json +import os +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable, Sequence + +REPO_ROOT = Path(__file__).resolve().parent.parent +MANIFEST = REPO_ROOT / "scripts" / "gate_freshness.json" + +# Marker in the sticky issue title so we update one issue forever instead of opening a +# new one every hour. Changing this string orphans the existing issue. +ISSUE_MARKER = "CI gate freshness alert" + +# Runs from these events count as evidence that the post-merge arm produced a result. +# `pull_request` is excluded on purpose -- see the module docstring. +POST_MERGE_EVENTS = frozenset({"push", "schedule", "workflow_dispatch"}) + + +def _utcnow() -> _dt.datetime: + return _dt.datetime.now(_dt.timezone.utc) + + +def _parse_ts(value: str) -> _dt.datetime: + return _dt.datetime.fromisoformat(value.replace("Z", "+00:00")) + + +@dataclass(frozen=True) +class Gate: + workflow: str + max_age_hours: float + why: str + + +@dataclass(frozen=True) +class Verdict: + gate: Gate + age_hours: float | None # None => no qualifying successful run at all + last_success: str | None + last_sha: str | None + + @property + def stale(self) -> bool: + return self.age_hours is None or self.age_hours > self.gate.max_age_hours + + @property + def detail(self) -> str: + if self.age_hours is None: + return "NO successful post-merge run in the sampled window" + return f"last success {self.age_hours:.1f}h ago (budget {self.gate.max_age_hours:g}h)" + + +def load_gates(manifest: Path = MANIFEST) -> tuple[list[Gate], str]: + data = json.loads(manifest.read_text()) + gates = [ + Gate( + workflow=g["workflow"], + max_age_hours=float(g["max_age_hours"]), + why=g.get("why", ""), + ) + for g in data["gates"] + ] + if not gates: + raise SystemExit("gate_freshness.json lists no gates; refusing to pass vacuously") + return gates, data.get("default_branch", "main") + + +def _gh_json(path: str) -> dict: + proc = subprocess.run( + ["gh", "api", "-H", "Accept: application/vnd.github+json", path], + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise RuntimeError(f"gh api {path} failed: {proc.stderr.strip()}") + return json.loads(proc.stdout) + + +def newest_post_merge_success( + repo: str, workflow: str, branch: str, fetch=_gh_json +) -> tuple[str, str] | None: + """Return (created_at, head_sha) of the newest successful post-merge run, or None. + + Asks for successful runs on the branch and then filters by event. The API's + `?event=` parameter takes a single value, so filtering client-side is what lets one + request cover push + schedule + workflow_dispatch. + """ + path = ( + f"repos/{repo}/actions/workflows/{workflow}/runs" + f"?branch={branch}&status=success&per_page=50" + ) + payload = fetch(path) + for run in payload.get("workflow_runs", []): + if run.get("event") in POST_MERGE_EVENTS: + return run["created_at"], run.get("head_sha", "") + return None + + +def evaluate( + gates: Sequence[Gate], repo: str, branch: str, now: _dt.datetime, fetch=_gh_json +) -> list[Verdict]: + verdicts: list[Verdict] = [] + for gate in gates: + try: + found = newest_post_merge_success(repo, gate.workflow, branch, fetch=fetch) + except RuntimeError as exc: + # A workflow file that no longer exists, or an API failure, must not be + # silently treated as "fresh". Report it as stale with no timestamp. + print(f"::warning::{gate.workflow}: {exc}", file=sys.stderr) + found = None + if found is None: + verdicts.append(Verdict(gate, None, None, None)) + continue + created, sha = found + age = (now - _parse_ts(created)).total_seconds() / 3600.0 + verdicts.append(Verdict(gate, age, created, sha)) + return verdicts + + +def render(verdicts: Iterable[Verdict]) -> str: + rows = list(verdicts) + width = max((len(v.gate.workflow) for v in rows), default=10) + lines = [f"{'workflow'.ljust(width)} status detail", f"{'-' * width} ------- ------"] + for v in rows: + status = "STALE" if v.stale else "ok" + lines.append(f"{v.gate.workflow.ljust(width)} {status:<7} {v.detail}") + return "\n".join(lines) + + +def issue_body(verdicts: Sequence[Verdict], branch: str) -> str: + stale = [v for v in verdicts if v.stale] + out = [ + f"The following post-merge gates have not produced a successful `{branch}` run " + "within their freshness budget.", + "", + "**This usually means they are starved, not broken** — queued behind a merge " + "cadence the runner pool cannot drain (#7856). Check the Actions queue depth " + "before assuming a code failure:", + "", + "```bash", + 'gh api "repos/PerryTS/perry/actions/runs?status=queued&per_page=100" \\', + " -q '.workflow_runs | length'", + "```", + "", + "| gate | detail | why it matters |", + "|---|---|---|", + ] + for v in stale: + out.append(f"| `{v.gate.workflow}` | {v.detail} | {v.gate.why} |") + out += [ + "", + "Scheduling rationale and the cost it trades away: " + "`docs/src/testing/ci-gate-scheduling.md`.", + "", + "_This issue is maintained automatically by `gate-freshness.yml`. It is updated " + "in place, never duplicated, and closes itself once every gate is fresh again._", + ] + return "\n".join(out) + + +# --------------------------------------------------------------------------- sticky issue + + +def _gh(args: list[str]) -> str: + proc = subprocess.run(["gh", *args], capture_output=True, text=True) + if proc.returncode != 0: + raise RuntimeError(f"gh {' '.join(args)} failed: {proc.stderr.strip()}") + return proc.stdout + + +def sync_issue(repo: str, verdicts: Sequence[Verdict], branch: str) -> None: + """Open, update, or close the single sticky alert issue. Never duplicates.""" + found = json.loads( + _gh( + [ + "issue", + "list", + "--repo", + repo, + "--state", + "open", + "--search", + f'"{ISSUE_MARKER}" in:title', + "--json", + "number,title", + "--limit", + "10", + ] + ) + ) + existing = next((i["number"] for i in found if ISSUE_MARKER in i["title"]), None) + stale = [v for v in verdicts if v.stale] + + if not stale: + if existing is not None: + _gh( + [ + "issue", + "close", + str(existing), + "--repo", + repo, + "--comment", + "All post-merge gates are fresh again; closing automatically.", + ] + ) + print(f"closed sticky issue #{existing} (all gates fresh)") + return + + title = f"{ISSUE_MARKER}: {len(stale)} gate(s) have no recent successful `{branch}` run" + body = issue_body(verdicts, branch) + if existing is None: + url = _gh( + ["issue", "create", "--repo", repo, "--title", title, "--body", body] + ).strip() + print(f"opened sticky issue {url}") + else: + _gh( + ["issue", "edit", str(existing), "--repo", repo, "--title", title, "--body", body] + ) + print(f"updated sticky issue #{existing}") + + +# --------------------------------------------------------------------------- self-test + + +def self_test() -> int: + """Prove the checker can still fail, and on which shapes. + + CLAUDE.md: a gate must assert its subject was live, not merely that nothing threw. + So this plants each failure shape one at a time and asserts the verdict, rather + than running the happy path and calling it green. + """ + now = _dt.datetime(2026, 8, 11, 15, 0, tzinfo=_dt.timezone.utc) + + def stamp(hours_ago: float) -> str: + return (now - _dt.timedelta(hours=hours_ago)).strftime("%Y-%m-%dT%H:%M:%SZ") + + fixtures = { + # fresh: 2h old scheduled success + "fresh.yml": [{"event": "schedule", "created_at": stamp(2), "head_sha": "aaa"}], + # stale: last success 3 days ago -- the #7856 shape + "stale.yml": [{"event": "push", "created_at": stamp(72), "head_sha": "bbb"}], + # no successful post-merge run at all + "never.yml": [], + # THE TRAP: plenty of recent successes, but all of them pull_request runs. + # This is what made gc-root-dominance look healthy while its main arm was dark. + "pronly.yml": [ + {"event": "pull_request", "created_at": stamp(0.5), "head_sha": "ccc"}, + {"event": "pull_request", "created_at": stamp(1.0), "head_sha": "ddd"}, + ], + # boundary: exactly at budget is NOT stale; just past it is. + "boundary.yml": [{"event": "schedule", "created_at": stamp(12), "head_sha": "eee"}], + } + + def fake_fetch(path: str) -> dict: + wf = path.split("/actions/workflows/")[1].split("/runs")[0] + return {"workflow_runs": fixtures[wf]} + + gates = [Gate(w, 12, "self-test") for w in fixtures] + verdicts = {v.gate.workflow: v for v in evaluate(gates, "o/r", "main", now, fetch=fake_fetch)} + + failures: list[str] = [] + + def expect(name: str, want_stale: bool, label: str) -> None: + got = verdicts[name].stale + if got != want_stale: + failures.append( + f"{label}: expected stale={want_stale} for {name}, got stale={got} " + f"({verdicts[name].detail})" + ) + + expect("fresh.yml", False, "a recent scheduled success is fresh") + expect("stale.yml", True, "a 3-day-old success is stale") + expect("never.yml", True, "no successful post-merge run at all is stale") + expect("pronly.yml", True, "pull_request runs alone do NOT count as fresh") + expect("boundary.yml", False, "exactly at budget is not yet stale") + + # The renderer must actually say STALE, or a red verdict could print as green. + table = render(verdicts.values()) + if "STALE" not in table: + failures.append("render() produced no STALE marker for a stale set") + + # And the exit path must be non-zero when anything is stale. + if _exit_code(list(verdicts.values())) == 0: + failures.append("exit code was 0 despite stale gates -- the gate cannot fail") + + # Sanity: an all-fresh set must exit 0, or the gate would be permanently red and + # would get muted, which is how gates die. + only_fresh = [verdicts["fresh.yml"], verdicts["boundary.yml"]] + if _exit_code(only_fresh) != 0: + failures.append("exit code was non-zero for an all-fresh set") + + print(table) + if failures: + print("\nSELF-TEST FAILED:", file=sys.stderr) + for f in failures: + print(f" - {f}", file=sys.stderr) + return 1 + print(f"\nself-test OK: {len(fixtures)} planted shapes, all verdicts as expected") + return 0 + + +def _exit_code(verdicts: Sequence[Verdict]) -> int: + return 1 if any(v.stale for v in verdicts) else 0 + + +# --------------------------------------------------------------------------- main + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--self-test", action="store_true", help="prove the checker can fail") + ap.add_argument("--dry-run", action="store_true", help="query the API but do not touch issues") + ap.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY", "PerryTS/perry")) + args = ap.parse_args(argv) + + if args.self_test: + return self_test() + + gates, branch = load_gates() + verdicts = evaluate(gates, args.repo, branch, _utcnow()) + + table = render(verdicts) + print(table) + + summary = os.environ.get("GITHUB_STEP_SUMMARY") + if summary: + with open(summary, "a") as fh: + fh.write(f"## Gate freshness (`{branch}`)\n\n```\n{table}\n```\n") + + stale = [v for v in verdicts if v.stale] + for v in stale: + print(f"::error::{v.gate.workflow} is stale: {v.detail}. {v.gate.why}") + + if not args.dry_run: + try: + sync_issue(args.repo, verdicts, branch) + except RuntimeError as exc: + # Failing to file the alert must not mask the alert itself. + print(f"::warning::could not sync the sticky issue: {exc}", file=sys.stderr) + + if stale: + print( + f"\n{len(stale)} of {len(verdicts)} gates have no recent successful " + f"`{branch}` run. See docs/src/testing/ci-gate-scheduling.md", + file=sys.stderr, + ) + return _exit_code(verdicts) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/gate_freshness.json b/scripts/gate_freshness.json new file mode 100644 index 0000000000..2d74aa934b --- /dev/null +++ b/scripts/gate_freshness.json @@ -0,0 +1,80 @@ +{ + "_comment": [ + "Freshness budgets for the post-merge gate sweeps. Checked by", + "scripts/check_gate_freshness.py, run hourly by .github/workflows/gate-freshness.yml.", + "", + "WHY THIS FILE EXISTS: #7856. Every heavy gate produced zero results on `main`", + "for over two days and nobody noticed, because an empty result set is", + "indistinguishable from a healthy one that nobody looked at. Rescheduling the", + "gates (docs/src/testing/ci-gate-scheduling.md) does not fix that on its own --", + "a cron that silently stops firing fails exactly the same way. This is the", + "detector for both.", + "", + "max_age_hours = schedule interval + headroom for a 90-minute job and a queue", + "that is still draining. A gate whose budget you keep having to RAISE is a gate", + "that is still starving: raise capacity or cut demand instead of the number.", + "", + "Entries are checked against the most recent SUCCESSFUL non-pull_request run on", + "the default branch. Pull-request runs are deliberately excluded -- counting them", + "is what made gc-root-dominance look healthy in #7856 while its `main` arm was", + "dark for two days." + ], + "default_branch": "main", + "gates": [ + { + "workflow": "gc-ratchet.yml", + "max_age_hours": 12, + "why": "Primary defence for collector retention/evacuation accounting. Went dark 2026-08-09..08-11 while #7799/#7809/#7812/#7834/#7839 merged; #7843's seven red rows landed in that window." + }, + { + "workflow": "gc-root-dominance.yml", + "max_age_hours": 12, + "why": "Static GC root-dominance checker over emitted LLVM IR. Its allowlist is empty, so every new hit is a red build -- worth nothing if the gate never runs." + }, + { + "workflow": "tls-budget.yml", + "max_age_hours": 12, + "why": "_tlv_get_addr call-count ratchet. macOS-only by nature; cannot be relieved by moving pools." + }, + { + "workflow": "gc-native-roots.yml", + "max_age_hours": 12, + "why": "RS4GC native-roots arm." + }, + { + "workflow": "gc-ptr-shape-off-witness.yml", + "max_age_hours": 12, + "why": "Exercises the Ptr OFF state. CLAUDE.md's kill-policy: an unexercised mode is a configuration nobody has verified." + }, + { + "workflow": "gc-parse-churn-gate.yml", + "max_age_hours": 12, + "why": "Parse-churn layout gate." + }, + { + "workflow": "gc-moving-witnesses.yml", + "max_age_hours": 12, + "why": "Asserts the copying minor actually moved objects -- the 'gate must assert its subject was live' arm (#7024/#7025)." + }, + { + "workflow": "auto-opt-app-patterns.yml", + "max_age_hours": 12, + "why": "Auto-optimize app-pattern regression sweep." + }, + { + "workflow": "eh-transport.yml", + "max_age_hours": 12, + "why": "Owned unwinder transport on arm64 macOS." + }, + { + "workflow": "llvm-inprocess.yml", + "max_age_hours": 12, + "why": "In-process LLVM backend arm; the promotion prerequisite." + }, + { + "workflow": "security-audit.yml", + "max_age_hours": 12, + "why": "Required status context, still on every merge. Included so that if IT starves too, this checker says so rather than staying silent." + } + ] +} From e41f57229928619b1baaed7b4c33dddcc82a7c97 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Tue, 11 Aug 2026 17:25:35 +0200 Subject: [PATCH 2/2] docs(changelog): fragment for #7860 --- changelog.d/7860-ci-gate-starvation.md | 63 ++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 changelog.d/7860-ci-gate-starvation.md diff --git a/changelog.d/7860-ci-gate-starvation.md b/changelog.d/7860-ci-gate-starvation.md new file mode 100644 index 0000000000..f98f8caee9 --- /dev/null +++ b/changelog.d/7860-ci-gate-starvation.md @@ -0,0 +1,63 @@ +### CI: the post-merge gates were starved, not broken — rescheduled, plus an alarm for the next time (#7856) + +Ten heavy gates ran on `push: branches: [main]`. Fourteen workflows fired on every +merge, ~29 jobs each time, against a repo that runs ~9 jobs concurrently. At the +cadence `main` reached (10 pushes/day on 2026-08-09 → 32 → **58** on 08-11) demand +outran drain, the queue grew without bound, and the entries that aged out were the +`main` runs — precisely the ones that gate nothing, so nobody watches them. +`gc-ratchet` had **22 of its last 25 `main` runs queued** (oldest 9h18m) and had not +succeeded on `main` since 2026-08-09, across five collector-touching merges (#7799, +#7809, #7812, #7834, #7839). #7843's seven genuinely-red rows landed inside that +blind window. + +This is a **third variant of CLAUDE.md's "four ways a gate can be unable to fail"**: +not `continue-on-error`, not missing from required contexts, not cancelled — +**starved**. + +**The constraint is total Actions concurrency, not macOS capacity**, which was the +natural reading and is wrong. At the moment of measurement the queue held **45 +`ubuntu-latest` jobs against 14 `macos-14`**, and `zizmor` (ubuntu) was queued in the +same second as `gc-ratchet` (macOS). Two claims in #7856 do not survive checking, and +both mattered: `gc-root-dominance` runs `macos-14`, not `ubuntu-latest` — its +healthy-looking run count was *pull-request* runs, which drain because they supersede +each other, while its `main` arm queued like the rest. So the ubuntu-vs-macOS contrast +that localised the problem to macOS was comparing a PR arm against a `main` arm. +**Rebalancing pools cannot help; only cutting total demand can.** (Moving `gc-ratchet` +to Linux was never available anyway: its baseline is keyed `darwin-arm64` and the +checker refuses a platform mismatch rather than comparing incomparable numbers.) + +The `concurrency:` blocks are **left untouched** — they are already correct and +twice-repaired (#7205). That fix worked; the failure mode simply moved from +*cancelled* to *never scheduled*. + +**Change.** The post-merge arm of ten gates (`gc-ratchet`, `gc-root-dominance`, +`tls-budget`, `gc-native-roots`, `gc-ptr-shape-off-witness`, `gc-parse-churn-gate`, +`gc-moving-witnesses`, `auto-opt-app-patterns`, `eh-transport`, `llvm-inprocess`) +becomes a staggered six-hourly sweep of `main` plus release tags. That removes **19 +jobs from every merge** — ~1,100 job-starts/day replaced by ~76, a **93% cut** on this +slice. Cron minutes are staggered and none sits at `:00`, so the ten do not re-create +the herd they were meant to relieve. Pull-request arms are unchanged, so every PR is +still measured before it can merge, and no probe, threshold, baseline or matrix cell +moved. `eh-transport` and `llvm-inprocess` also needed their relevance guard changed +from `= "push"` to `!= "pull_request"`; under a `schedule` event the old test fell +through to the PR branch and would dereference an empty PR number. + +**The cost, stated plainly:** attribution latency. A regression that slips past the PR +arm is now named against a window of commits rather than one (bisect `previous sweep +SHA .. this sweep SHA`). Four completed runs a day beat 58 that never start. + +**Alarm.** `gate-freshness.yml` + `scripts/check_gate_freshness.py` close the hole that +let this last two days: an empty result set is indistinguishable from a healthy one +nobody checked. It fails when a gate has no successful **post-merge** `main` run inside +its budget (`scripts/gate_freshness.json`) and maintains one self-closing sticky issue. +`pull_request` runs are deliberately not counted as evidence of health — counting them +is exactly what made `gc-root-dominance` look fine while it was dark. The checker is +**sabotage-tested, not merely exercised**: `--self-test` plants a stale gate, a gate +with no successful run at all, and a gate whose only recent successes are PR runs, and +asserts the verdict for each; three independent sabotages of the detector were each +caught with a specific message. Its first live run found two things outside the issue's +scope — **`llvm-inprocess` had been dark for 171.8h**, and **`security-audit`, a +required context left on every merge, was itself stale at 17.4h**. + +Full measurement, the refuted claims, and what was deliberately not changed: +`docs/src/testing/ci-gate-scheduling.md`.