ci: unstarve the post-merge gates and alert when one goes dark (#7856)

.github/workflows/auto-opt-app-patterns.yml

@@ -47,8 +47,16 @@ name: Auto-Optimize App Patterns
 
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "22 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

.github/workflows/eh-transport.yml

@@ -28,8 +28,17 @@ name: eh-transport
 # `changes` job: a job skipped by `if:` still reports a check run.
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "32 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
+  workflow_dispatch:
 
 concurrency:
   group: eh-transport-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }}
@@ -55,7 +64,14 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          if [ "${{ github.event_name }}" = "push" ]; then
+          # Anything that is not a pull request (the scheduled `main` sweep, a
+          # release tag, a manual dispatch) measures unconditionally: there is no
+          # PR file list to filter on, and #7856's starvation is precisely what
+          # happens when a post-merge arm quietly declines to run. Testing for
+          # `= "push"` here was correct only while `push: branches: [main]` was
+          # the post-merge trigger; under the schedule it would fall through to
+          # the PR branch, dereference an empty PR number and fail the step.
+          if [ "${{ github.event_name }}" != "pull_request" ]; then
             echo "relevant=true" >> "$GITHUB_OUTPUT"
             exit 0
           fi

.github/workflows/gate-freshness.yml

@@ -0,0 +1,81 @@
+name: Gate Freshness
+
+# Alerts when a post-merge gate stops producing successful `main` runs.
+#
+# THE FAILURE THIS EXISTS FOR (#7856): every heavy gate in this repo produced zero
+# results on `main` for over two days. They were not failing and not cancelled --
+# they never reached a runner, because fourteen workflows enqueued ~29 jobs on every
+# one of 58 daily merges against a repo that runs ~9 jobs at a time. Five
+# collector-touching PRs merged inside that window and #7843's seven genuinely-red
+# rows landed unseen.
+#
+# What made it last two days is the part worth engineering against: **an empty result
+# set looks exactly like a healthy one that nobody has looked at.** Rescheduling the
+# gates (docs/src/testing/ci-gate-scheduling.md) does not fix that on its own -- a
+# cron that quietly stops firing fails in precisely the same way. This workflow is the
+# detector for both, and it is the reason the next occurrence should be noticed in
+# hours rather than days.
+#
+# DELIBERATELY NOT A REQUIRED CONTEXT. It reports on the health of other gates; it
+# must never be able to block a PR. It is also deliberately cheap -- one short
+# ubuntu job -- so that the alarm cannot be starved by the condition it is alarming
+# about.
+
+on:
+  schedule:
+    # Every two hours, off the top of the hour (GitHub's scheduler is most contended
+    # at :00 and most likely to delay a run there). The tightest budget in
+    # scripts/gate_freshness.json is 12h, so this samples ~6x per budget window.
+    - cron: "5 */2 * * *"
+  # Validate the checker itself when it changes. Self-test only -- see the step
+  # guards below; a PR must not go red merely because `main`'s gates are behind.
+  pull_request:
+    paths:
+      - scripts/check_gate_freshness.py
+      - scripts/gate_freshness.json
+      - .github/workflows/gate-freshness.yml
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  # One sweep at a time. Unlike the gates this watches, coalescing is correct here:
+  # the freshness verdict is a function of "now", so a superseded run had nothing
+  # unique to say. PR runs supersede themselves; scheduled runs queue.
+  group: gate-freshness-${{ github.event_name }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  gate-freshness:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      # Listing workflow runs, and maintaining the single sticky alert issue.
+      actions: read
+      issues: write
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+
+      # Structural failure first, and unconditionally: prove the detector can still
+      # say no before trusting a green verdict from it. This plants a stale gate, a
+      # gate with no successful run at all, and -- the trap that made
+      # `gc-root-dominance` look healthy in #7856 -- a gate whose only recent
+      # successes are `pull_request` runs, then asserts the verdict for each.
+      #
+      # A green self-test means the detector works, not that nothing was tried.
+      - name: Self-test the freshness checker (can this gate still fail?)
+        run: python3 scripts/check_gate_freshness.py --self-test
+
+      # The live check does not run on pull requests. A PR that merely touches this
+      # checker must not go red because `main`'s gates are behind -- that would make
+      # the alarm a merge blocker, which it is explicitly not.
+      - name: Check post-merge gate freshness
+        if: github.event_name != 'pull_request'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+        run: python3 scripts/check_gate_freshness.py

.github/workflows/gc-moving-witnesses.yml

@@ -108,8 +108,16 @@ name: GC Moving Witnesses
 
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "12 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

.github/workflows/gc-native-roots.yml

@@ -111,8 +111,18 @@ on:
   # filter would mean the job never runs at all — CLAUDE.md's second way a gate
   # cannot fail. Cancellation is deliberately NOT set here: a `main` run that
   # gets cancelled by the next merge is the third way.
+  #
+  # And the FOURTH way is what `push: branches: [main]` turned out to be here:
+  # STARVED (#7856). Neither cancelled nor failing — simply never scheduled,
+  # because fourteen workflows enqueued ~29 jobs on every one of 58 daily merges
+  # against a repo that runs ~9 jobs at a time. The post-merge arm is now a
+  # staggered six-hourly sweep; the pull-request arm is unchanged, so every PR is
+  # still measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "37 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   pull_request:
   workflow_dispatch:
 

.github/workflows/gc-parse-churn-gate.yml

@@ -76,8 +76,16 @@ name: GC Parse-Churn Layout Gate
 
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "57 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

.github/workflows/gc-ptr-shape-off-witness.yml

@@ -74,8 +74,16 @@ name: GC Ptr<Shape> OFF-arm witness
 
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "47 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

.github/workflows/gc-ratchet.yml

@@ -15,8 +15,35 @@ name: GC Ratchet
 
 on:
   pull_request:
+  # POST-MERGE ARM: a staggered six-hourly sweep of `main`, NOT one run per merge.
+  #
+  # `push: branches: [main]` is what took this gate dark for two days (#7856).
+  # Fourteen workflows fired on every merge, ~29 jobs each time, against a repo
+  # that runs ~9 jobs concurrently; at 58 merges/day demand outran drain and the
+  # queue grew without bound. On 2026-08-11 this workflow had 22 of its 25 most
+  # recent `main` runs sitting `queued` (oldest 9h18m) and its last successful
+  # `main` run was 2026-08-09 -- two days and five collector-touching merges
+  # earlier (#7799, #7809, #7812, #7834, #7839). #7843's seven genuinely-red rows
+  # landed inside that blind window.
+  #
+  # This is the THIRD variant of CLAUDE.md's "four ways a gate can be unable to
+  # fail": not `continue-on-error`, not missing from required contexts, not
+  # cancelled -- STARVED. A gate that runs four times a day and COMPLETES is
+  # worth more than one that runs on every merge and never does.
+  #
+  # It is NOT the `concurrency:` block below (already correct, twice-repaired --
+  # see its comment) and it is NOT macOS capacity: at the moment of measurement
+  # the queue held 45 `ubuntu-latest` jobs against 14 `macos-14`. Rebalancing
+  # pools cannot help; only cutting total job demand can.
+  #
+  # ***DO NOT RESTORE `push: branches: [main]` HERE.*** Full measurement, the two
+  # #7856 claims that did not survive checking, and the cost this trades away:
+  # docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "7 */6 * * *"
   push:
-    branches: [main]
+    # Releases stay individually gated; tags are rare, so this costs nothing.
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

.github/workflows/gc-root-dominance.yml

@@ -61,8 +61,19 @@ name: GC Root Dominance
 
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. Note this workflow runs `macos-14`, not `ubuntu-latest`; #7856's
+  # claim that it was unaffected was reading its PR arm, which drains because PR
+  # runs supersede each other, while its `main` arm queued like all the rest.
+  # ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "17 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

.github/workflows/llvm-inprocess.yml

@@ -16,8 +16,17 @@ name: llvm-inprocess
 # run (conclusion: skipped), which branch protection accepts.
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "42 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
+  workflow_dispatch:
 
 concurrency:
   group: llvm-inprocess-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }}
@@ -46,7 +55,14 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          if [ "${{ github.event_name }}" = "push" ]; then
+          # Anything that is not a pull request (the scheduled `main` sweep, a
+          # release tag, a manual dispatch) measures unconditionally: there is no
+          # PR file list to filter on, and #7856's starvation is precisely what
+          # happens when a post-merge arm quietly declines to run. Testing for
+          # `= "push"` here was correct only while `push: branches: [main]` was
+          # the post-merge trigger; under the schedule it would fall through to
+          # the PR branch, dereference an empty PR number and fail the step.
+          if [ "${{ github.event_name }}" != "pull_request" ]; then
             echo "relevant=true" >> "$GITHUB_OUTPUT"
             exit 0
           fi

.github/workflows/tls-budget.yml

@@ -54,8 +54,18 @@ name: TLS Budget
 
 on:
   pull_request:
+  # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge.
+  # `push: branches: [main]` starved this gate and nine others for two days
+  # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job
+  # concurrency ceiling. The pull-request arm is unchanged -- every PR is still
+  # measured. The macOS arm below is irreducible (`_tlv_get_addr` is a Mach-O
+  # artefact), so moving it to Linux was never an option; cutting demand was.
+  # ***DO NOT RESTORE `push: branches: [main]`.***
+  # Rationale, measurement and cost: docs/src/testing/ci-gate-scheduling.md
+  schedule:
+    - cron: "27 */6 * * *"
   push:
-    branches: [main]
+    tags: ["v*"]
   workflow_dispatch:
 
 permissions:

changelog.d/7860-ci-gate-starvation.md

@@ -0,0 +1,63 @@
+### CI: the post-merge gates were starved, not broken — rescheduled, plus an alarm for the next time (#7856)
+
+Ten heavy gates ran on `push: branches: [main]`. Fourteen workflows fired on every
+merge, ~29 jobs each time, against a repo that runs ~9 jobs concurrently. At the
+cadence `main` reached (10 pushes/day on 2026-08-09 → 32 → **58** on 08-11) demand
+outran drain, the queue grew without bound, and the entries that aged out were the
+`main` runs — precisely the ones that gate nothing, so nobody watches them.
+`gc-ratchet` had **22 of its last 25 `main` runs queued** (oldest 9h18m) and had not
+succeeded on `main` since 2026-08-09, across five collector-touching merges (#7799,
+#7809, #7812, #7834, #7839). #7843's seven genuinely-red rows landed inside that
+blind window.
+
+This is a **third variant of CLAUDE.md's "four ways a gate can be unable to fail"**:
+not `continue-on-error`, not missing from required contexts, not cancelled —
+**starved**.
+
+**The constraint is total Actions concurrency, not macOS capacity**, which was the
+natural reading and is wrong. At the moment of measurement the queue held **45
+`ubuntu-latest` jobs against 14 `macos-14`**, and `zizmor` (ubuntu) was queued in the
+same second as `gc-ratchet` (macOS). Two claims in #7856 do not survive checking, and
+both mattered: `gc-root-dominance` runs `macos-14`, not `ubuntu-latest` — its
+healthy-looking run count was *pull-request* runs, which drain because they supersede
+each other, while its `main` arm queued like the rest. So the ubuntu-vs-macOS contrast
+that localised the problem to macOS was comparing a PR arm against a `main` arm.
+**Rebalancing pools cannot help; only cutting total demand can.** (Moving `gc-ratchet`
+to Linux was never available anyway: its baseline is keyed `darwin-arm64` and the
+checker refuses a platform mismatch rather than comparing incomparable numbers.)
+
+The `concurrency:` blocks are **left untouched** — they are already correct and
+twice-repaired (#7205). That fix worked; the failure mode simply moved from
+*cancelled* to *never scheduled*.
+
+**Change.** The post-merge arm of ten gates (`gc-ratchet`, `gc-root-dominance`,
+`tls-budget`, `gc-native-roots`, `gc-ptr-shape-off-witness`, `gc-parse-churn-gate`,
+`gc-moving-witnesses`, `auto-opt-app-patterns`, `eh-transport`, `llvm-inprocess`)
+becomes a staggered six-hourly sweep of `main` plus release tags. That removes **19
+jobs from every merge** — ~1,100 job-starts/day replaced by ~76, a **93% cut** on this
+slice. Cron minutes are staggered and none sits at `:00`, so the ten do not re-create
+the herd they were meant to relieve. Pull-request arms are unchanged, so every PR is
+still measured before it can merge, and no probe, threshold, baseline or matrix cell
+moved. `eh-transport` and `llvm-inprocess` also needed their relevance guard changed
+from `= "push"` to `!= "pull_request"`; under a `schedule` event the old test fell
+through to the PR branch and would dereference an empty PR number.
+
+**The cost, stated plainly:** attribution latency. A regression that slips past the PR
+arm is now named against a window of commits rather than one (bisect `previous sweep
+SHA .. this sweep SHA`). Four completed runs a day beat 58 that never start.
+
+**Alarm.** `gate-freshness.yml` + `scripts/check_gate_freshness.py` close the hole that
+let this last two days: an empty result set is indistinguishable from a healthy one
+nobody checked. It fails when a gate has no successful **post-merge** `main` run inside
+its budget (`scripts/gate_freshness.json`) and maintains one self-closing sticky issue.
+`pull_request` runs are deliberately not counted as evidence of health — counting them
+is exactly what made `gc-root-dominance` look fine while it was dark. The checker is
+**sabotage-tested, not merely exercised**: `--self-test` plants a stale gate, a gate
+with no successful run at all, and a gate whose only recent successes are PR runs, and
+asserts the verdict for each; three independent sabotages of the detector were each
+caught with a specific message. Its first live run found two things outside the issue's
+scope — **`llvm-inprocess` had been dark for 171.8h**, and **`security-audit`, a
+required context left on every merge, was itself stale at 17.4h**.
+
+Full measurement, the refuted claims, and what was deliberately not changed:
+`docs/src/testing/ci-gate-scheduling.md`.

docs/src/SUMMARY.md

@@ -150,6 +150,7 @@
 - [Test Registration (dark tests)](testing/test-registration.md)
 - [Geisterhand (UI Fuzzer)](testing/geisterhand.md)
 - [Node Compatibility Matrix](testing/node-compat-matrix.md)
+- [CI Gate Scheduling](testing/ci-gate-scheduling.md)
 
 # CLI Reference