From 74a4da828adcceeafef948e6f70fc374669fd96d Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Wed, 23 Sep 2026 01:20:20 -0500 Subject: [PATCH 1/6] =?UTF-8?q?ci:=20platform=20tiers=20=E2=80=94=20requir?= =?UTF-8?q?ed-checks.txt=20is=20tier=201,=20a=20gate=20pins=20main's=20col?= =?UTF-8?q?our=20to=20it=20(#1264)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README badge is the status of the whole ci.yml workflow on main, so any ci.yml job that can fail there colours it, required or not; main was red 2026-09-16 -> 2026-09-22 on a lane no PR had to pass (macos-15-intel, moved to nightly in #1265). - .github/required-checks.txt: ONE source of truth for tier 1 (the ruleset is synced from it). The 17 live contexts plus two: * `linux / clang` — tier 1 is "Linux gcc/clang" (#1264); it runs on every PR, and a half-required matrix leaves one leg able to colour main. * `build dev/ci image` — the required jobs that need it would be SKIPPED if it failed, and GitHub treats a skipped required check as satisfied. Until the ruleset is synced, `tools/ci_tier_check.sh --live` reports these two by name (expected). - tools/ci_tier_check.sh: every required name produced by exactly one job that reports on pull_request (trigger, no path filter, no event-gated job or ancestor); a required job's needs are required unless it is `if: always()`; every ci.yml main-lane job is required or a worker whose only consumer is a required `if: always()` aggregator that checks its result; every nightly job is read by a reporter that files an issue. Populations pinned exactly against independent counts (awk job ids, grep names). --selftest: 21 plants + a sanity start, each red through a NAMED check; PyYAML absent is exit 2, never a verdict. Wired into the `gate self-tests` job. - docs/CI.md: the stale "Required status checks" section (one required check) replaced by "Platform tiers": the model, the file, how to change it, what the gate enforces, every ci.yml job and every other workflow's check classified with the reason. - nightly.yml: the tracking-issue body no longer claims macos-15-intel runs on every push to main. - docs_claims_populations: PATHS|docs/CI.md 60 -> 63 (the new section cites three more resolved paths). Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/required-checks.txt | 37 ++ .github/workflows/ci.yml | 16 + .github/workflows/nightly.yml | 2 +- docs/CI.md | 198 ++++++---- tools/ci_tier_check.sh | 592 ++++++++++++++++++++++++++++++ tools/docs_claims_populations.txt | 6 +- 6 files changed, 774 insertions(+), 77 deletions(-) create mode 100644 .github/required-checks.txt create mode 100755 tools/ci_tier_check.sh diff --git a/.github/required-checks.txt b/.github/required-checks.txt new file mode 100644 index 00000000..95577eed --- /dev/null +++ b/.github/required-checks.txt @@ -0,0 +1,37 @@ +# TIER 1 (#1264) — the checks that block a merge to main. THE source of truth: +# the "Protection" ruleset (id 17713865) is synced FROM this file after merge, +# never edited by hand. tools/ci_tier_check.sh gates it against the workflows +# (every name is produced by exactly one job that runs on pull_request; every +# ci.yml job on the main lane is listed here or is a worker of an aggregator +# listed here); `tools/ci_tier_check.sh --live` diffs it against the ruleset. +# +# Format: one exact check-run name per line. A line starting with `#` is a +# comment (give the reason for a non-obvious entry). No trailing blanks. +# +# ci.yml +scope +# Every Linux leg runs inside this image; a failed prerequisite SKIPS the +# required jobs that need it, and GitHub counts a skipped required check as +# passing — so a required job's `needs` must be required too (#1264). +build dev/ci image +werror audit ([99i], cached) +gate self-tests (section plan + audit cache key) +linux / gcc +# The clang leg: -Werror at compile time + clang codegen (#1264: tier 1 is +# "Linux gcc/clang"). Core smoke on a PR, the full suite on main. +linux / clang +macos / macos-latest +extensions (http+model+gfx suite; embed/lsp/jit-smoke) +asan + ubsan (full suite) +db extension (postgres service) +jit differential (interpreter oracle, tape-replayed) +replay differential (same-binary tape fidelity) +freestanding profile (symbol gate + smoke) +tsan (concurrency race gate) +install.sh (interpreter + eigenlsp on PATH) +bench (instruction-count regression gate) +valgrind (memcheck smoke, JIT off) +# codeql.yml (workflow "CodeQL") +Analyze C +# pages.yml (workflow "Docs site") — the aggregator over the real emcc build +playground (real emcc wasm32 build) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 330207cd..e4cdb740 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -46,6 +46,11 @@ concurrency: # main lane (push to main) — the full matrix except macos-15-intel, with # [99i] owned by the one `werror audit` job instead of being repeated ten # times. The merge-er waits for this; contributors do not. +# TIERS (#1264): every job in this file is TIER 1 — listed in +# .github/required-checks.txt, or a worker of an aggregator listed there. +# A job that is neither must go to nightly.yml; tools/ci_tier_check.sh +# (the `gate self-tests` job) fails otherwise. Adding a job here means +# adding its check name to that file (the ruleset is synced from it). # nightly (.github/workflows/nightly.yml) — macos-15-intel (ONLY here, #1264) # and the full valgrind corpus, with a tracking issue on failure. # @@ -348,6 +353,17 @@ jobs: apt-get update && apt-get install -y --no-install-recommends python3-yaml fi python3 -c 'import yaml; print("yaml", yaml.__version__)' + # Platform tiers (#1264): every name in .github/required-checks.txt is + # produced by one job that reports on pull_request, and every job in + # THIS file that runs on a push to main is required or a worker of a + # required aggregator — so main's colour is decided by the merge gate. + # ~1 s + ~15 s of planted faults. Needs the PyYAML installed above; a + # missing loader is exit 2 (instrument error), never a pass. + - if: needs.scope.outputs.code == 'true' + name: Platform tiers — required-checks.txt vs the workflows (#1264) + run: | + bash tools/ci_tier_check.sh + bash tools/ci_tier_check.sh --selftest - if: needs.scope.outputs.code == 'true' name: Consumer-acceptance harness self-test (~7 min) run: bash tools/consumer_acceptance.sh --self-test diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index e6c371c1..600731f4 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -178,7 +178,7 @@ jobs: Run: $RUN_URL - These lanes moved off the PR path in #1160. \`macos-15-intel\` also runs on every push to \`main\`; the FULL valgrind corpus runs HERE ONLY (the PR lane and \`main\` both run the smoke spread), so a red \`valgrind-full\` is a finding nothing else will report. This thread exists so a nightly failure between main pushes is not silent." + These lanes moved off the PR path in #1160 and are tier 2 (#1264): they never colour \`main\`. \`macos-15-intel\` runs HERE ONLY; the FULL valgrind corpus runs HERE ONLY (the PR lane and \`main\` both run the smoke spread), so a red \`valgrind-full\` is a finding nothing else will report. This thread exists so a nightly failure between main pushes is not silent." if [ -n "$num" ]; then # Reopen first: a closed thread that starts failing again is the diff --git a/docs/CI.md b/docs/CI.md index af0d7a92..a7563281 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -925,8 +925,8 @@ tools/section_plan.sh --shards 3 --shard 2 # that shard's plan line EIGS_SUITE_SHARD=2/3 bash tests/run_all_tests.sh ``` -The aggregator `asan + ubsan (full suite)` — still the only ruleset-required -check, and still that name — does four things no shard can do for itself: it +The aggregator `asan + ubsan (full suite)` — a ruleset-required check (see +**Platform tiers**), and still that name — does four things no shard can do for itself: it requires every matrix leg green, re-runs `--shards 3 --check`, requires one **receipt** per shard carrying that shard's `PLAN: shard=k/3 …` line, and **sums the LeakSanitizer tallies and requires 0**. Splitting the job must not @@ -1032,79 +1032,127 @@ whose inputs really are the Makefile and the tracked scripts). A local does the suite's [99i]. `werror_cache_key.sh --selftest` reads both `ci.yml` and the audit script and fails if the split stops being used. -## Required status checks — what is actually required today - -Read off the live repo (`gh api repos/InauguralSystems/EigenScript/rulesets`, -2026-09-15), because round 1 of this change documented a list that does not -exist: - -- Classic branch protection on `main`: **not enabled** (`branches/main/protection` - returns 404, "Branch not protected"). -- Ruleset **"Protection"** (active, `~DEFAULT_BRANCH`) requires exactly **one** - status check: `asan + ubsan (full suite)`. -- Ruleset **"Main"** (active) targets `refs/heads/Main` — a branch with a - capital M that does not exist — and requires `Black`. It is inert. - -So `macos / macos-15-intel` was never in a required list, and nothing here -"must be removed" for the merge to work. What matters instead is the reverse: -**`asan + ubsan (full suite)` is the only gate the ruleset enforces**, and it -is an *aggregator* — it reports success only when both sanitizer workers -succeed (see below). That single rule keeps working unchanged under this -change. - -### The PR-lane job set, and which are aggregators - -On a pull request, `ci.yml` produces these checks: - -| Check | Kind | -|---|---| -| `scope` | gate; decides docs-only | -| `build dev/ci image` | prerequisite; every Linux leg runs inside it | -| `werror audit ([99i], cached)` | gate | -| `gate self-tests (section plan + audit cache key)` | gate | -| `linux / gcc` | the one full suite | -| `linux / clang` | build + derived core smoke | -| `macos / macos-latest` | full suite (code PRs only) | -| `extensions (http+model+gfx suite; embed/lsp/jit-smoke)` | **aggregator** over the four workers below | -| `extensions / http+model and ancillary checks` | worker | -| `extensions / gfx suite` | worker | -| `extensions / zlib suite` | worker | -| `extensions / net suite` | worker | -| `asan + ubsan (full suite)` | **aggregator** over the two workers below | -| `asan + ubsan / core and LSP` | worker | -| `asan + ubsan / HTTP and model suite` | worker | -| `db extension (postgres service)` | gate | -| `jit differential (interpreter oracle, tape-replayed)` | gate | -| `replay differential (same-binary tape fidelity)` | gate | -| `freestanding profile (symbol gate + smoke)` | gate | -| `valgrind (memcheck smoke, JIT off)` | gate (the smoke spread; the job prints its size) | -| `tsan (concurrency race gate)` | gate | -| `install.sh (interpreter + eigenlsp on PATH)` | gate | -| `bench (instruction-count regression gate)` | gate | -| `Analyze C` (workflow `CodeQL`) | gate, separate workflow | -| `playground (real emcc wasm32 build)` (workflow `Docs site`) | **aggregator** over the worker below; reports on every PR | -| `playground / build (real emcc, web/build.sh) + docs site` | worker | - -`macos / macos-15-intel` runs **only** in nightly (#1264): on the main lane it hit its -45-minute timeout on nearly every push, so main CI never finished green. - -An aggregator exists so that a *required* check name can survive the job being -split into parallel workers: it fails unless every worker succeeded, and it -treats `skipped`, `cancelled` and missing results as failure. A worker is not -separately required; it is required *through* its aggregator. - -### If the required set is ever widened - -The set worth requiring, if someone tightens the ruleset, is: `scope`, -`linux / gcc`, `extensions (…)`, `asan + ubsan (full suite)`, -`db extension (postgres service)`, `macos / macos-latest`, -`werror audit ([99i], cached)`, `gate self-tests (…)`, the two differentials, -`freestanding`, `tsan`, `install.sh`, `bench`, `valgrind` and `Analyze C`. -**Never** `macos / macos-15-intel`: it does not run on pull requests, and a -required check that never reports blocks the merge forever — the same trap the -`scope` job's comment in `ci.yml` describes. Add `playground (real emcc -wasm32 build)` — it reports on every pull request, success only when the -real emcc build of `web/build.sh` ran and passed, failure otherwise. +## Platform tiers — what blocks a merge, and what decides main's colour (#1264) + +Main CI did not finish green from 2026-09-16 to 2026-09-22 although every +required check passed on every merge: one lane that no pull request had to pass +(`macos / macos-15-intel`, main lane only) hit its timeout on nearly every +push. The README badge is the status of the whole `ci.yml` workflow on `main`, +so **any** `ci.yml` job that can fail there colours it, required or not. When +red is normal, a real regression is invisible. + +The fix copies Rust, CPython and Go: + +- **Tier 1** — the checks listed in `.github/required-checks.txt`. They block + a merge (the ruleset requires them), and they are the only things on the + main lane, so they are what decides `main`'s colour. +- **Tier 2** — slow and port lanes, in `.github/workflows/nightly.yml` + (today `macos-15-intel` and the full valgrind corpus). They never colour + `main`; a failure opens or appends to one tracking issue. + +### The source of truth: `.github/required-checks.txt` + +One exact check-run name per line; a line starting with `#` is a comment and +gives the reason for a non-obvious entry. The ruleset **"Protection"** +(`~DEFAULT_BRANCH`) is synced *from* this file — it is never edited by hand. +`bash tools/ci_tier_check.sh --live` diffs the file against the live ruleset +(read-only) and names every check on one side only. + +To change tier 1: edit the file in a PR (the gate below must stay green), merge, +and the orchestrator syncs the ruleset from the file; `--live` then prints OK. +Between the merge and the sync, `--live` reports the drift by name — that is +expected, and it is why CI does not run `--live`. + +### What `tools/ci_tier_check.sh` enforces + +It runs in the `gate self-tests` job on every code PR, followed by its +`--selftest` (planted faults, each required to go red through a *named* check). +It parses every workflow with PyYAML (a missing loader is exit 2, an instrument +error — never a pass) and fails when: + +- a required name is produced by **no** job (`[unproduced]`) or by more than one + (`[ambiguous]`); +- a required job may not report on a pull request (`[not-on-pr]`): its + workflow does not trigger on `pull_request` for `main`, the trigger is + path-filtered, or the job — or any job it transitively `needs` — has a + job-level `if:` that mentions the event. A required check that never reports + blocks every merge forever; +- a required job without `if: always()` needs a job that is not required + (`[need-not-required]`): a failed prerequisite **skips** its dependants, and + GitHub treats a skipped required check as satisfied; +- a `ci.yml` job on the main lane is neither required nor a covered worker + (`[uncovered]`), or only some of its matrix legs are required + (`[partial-matrix]`). This is the `macos-15-intel` shape, made red; +- a worker is not *covered* — its only consumer must be a required aggregator + (`[multi-consumer]` otherwise) that has `if: always()` (`[agg-not-always]`) + and compares `needs..result` to `success` in an unconditional step + (`[agg-unchecked]`), and the worker may not set `continue-on-error` + (`[worker-continue-on-error]`); +- a nightly job is not read by an `if: always()` reporter that checks its + result and files an issue (`[nightly-unreported]`); +- a population does not match its independent count (`[count-mismatch]`: the + YAML loader's `ci.yml` job count against an awk count of the keys under + `jobs:`, the parsed required names against a `grep` count), or is empty + (`[vacuous]`). + +It prints the classification of every job, so the tables below are its output +in prose, not a second list to keep in sync. + +### Every `ci.yml` job, classified + +| Job (check name) | Tier | Why | +|---|---|---| +| `scope` | 1 | decides docs-only; every job needs it | +| `build dev/ci image` | 1 | every Linux leg runs inside it; required because the jobs that need it are required (a failed image would *skip* them) — added by #1264 | +| `werror audit ([99i], cached)` | 1 | gate | +| `gate self-tests (section plan + audit cache key)` | 1 | gate; runs this checker | +| `linux / gcc` | 1 | the one full suite on a PR | +| `linux / clang` | 1 | clang `-Werror` build + core smoke on a PR, full suite on main — added by #1264 ("Linux gcc/clang"); a half-required matrix is red | +| `macos / macos-latest` | 1 | the one macOS leg on the PR lane | +| `extensions (http+model+gfx suite; embed/lsp/jit-smoke)` | 1 | **aggregator** | +| `extensions / http+model and ancillary checks`, `/ gfx suite`, `/ zlib suite`, `/ net suite` | 1, via the aggregator | workers | +| `asan + ubsan (full suite)` | 1 | **aggregator**; also re-derives shard coverage and sums the leak tally | +| `asan + ubsan / core and LSP (shard k/3)`, `asan + ubsan / HTTP and model suite` | 1, via the aggregator | workers | +| `db extension (postgres service)` | 1 | gate | +| `jit differential (…)`, `replay differential (…)` | 1 | gates | +| `freestanding profile (symbol gate + smoke)` | 1 | gate | +| `valgrind (memcheck smoke, JIT off)` | 1 | the smoke spread (the full corpus is tier 2) | +| `tsan (concurrency race gate)` | 1 | gate | +| `install.sh (interpreter + eigenlsp on PATH)` | 1 | gate | +| `bench (instruction-count regression gate)` | 1 | gate | +| `nightly / macos-15-intel full suite` | **2** (`nightly.yml`) | port lane, slow: hit its timeout on nearly every main push (#1265) | +| `nightly / valgrind (full corpus, JIT off)` | **2** (`nightly.yml`) | slow; the PR and main lanes run the smoke spread | + +No `ci.yml` job moved to nightly in #1264 beyond `macos-15-intel` (#1265): +every other job already runs on pull requests, so each was made tier 1 rather +than demoted. + +### Checks from other workflows + +They do not affect the `ci.yml` badge. Each is required or advisory by the same +file: + +| Check (workflow) | Tier | Why | +|---|---|---| +| `Analyze C` (`codeql.yml`) | 1 | runs on every PR to `main`, no path filter | +| `playground (real emcc wasm32 build)` (`pages.yml`) | 1 | **aggregator** over `playground / build (…)`; reports on every PR | +| `playground / build (real emcc, web/build.sh) + docs site` (`pages.yml`) | advisory | the worker; required through the aggregator | +| `deploy` (`pages.yml`) | advisory | push-only (`if: github.event_name != 'pull_request'`) — could never be required | +| `codspeed (simulation)` (`codspeed.yml`), and the CodSpeed app's `CodSpeed Performance Analysis` | advisory | path-filtered (`paths-ignore: '**.md'`), so it does not report on docs-only PRs; the instruction-count gate that blocks is `bench` | +| `build` (`docker.yml`) | advisory | push and tags only | +| `Scorecard analysis` (`scorecard.yml`) | advisory | push/schedule only; a posture score, not a correctness gate | +| `Analyze (python)`, `Analyze (javascript-typescript)` | advisory | CodeQL *default setup* (a GitHub app, not a workflow file) over the repo's non-C code; the C analysis that blocks is `Analyze C` | +| `issue-triage / …` (`issue-triage.yml`), `release.yml` jobs | advisory | not triggered by PRs or pushes to `main` | + +### The two traps the tier set has to respect + +- **A required check that never reports blocks every merge forever.** That is + why `macos-15-intel` could never have been made required, why the `scope` + job's runtime legs report success in seconds on a docs-only PR instead of + being skipped, and why `[not-on-pr]` exists. +- **A skipped required check is satisfied.** That is why the aggregators run + `if: always()` and treat `skipped`, `cancelled` and a missing result as + failure, and why a required job's prerequisites must be required too. ## The risk this accepts diff --git a/tools/ci_tier_check.sh b/tools/ci_tier_check.sh new file mode 100755 index 00000000..e8299be7 --- /dev/null +++ b/tools/ci_tier_check.sh @@ -0,0 +1,592 @@ +#!/usr/bin/env bash +# tools/ci_tier_check.sh — platform tiers (#1264): the main lane's colour means +# something. +# +# THE MODEL (copied from Rust / CPython / Go, owner decision on #1264): +# TIER 1 — the checks in .github/required-checks.txt. They block a merge +# (the "Protection" ruleset is synced FROM that file) and they are +# what decides whether `main` is green. +# TIER 2 — slow or port lanes (.github/workflows/nightly.yml). They never +# colour main; a failure opens or appends to a tracking issue. +# +# WHY A GATE. Main CI was red from 2026-09-16 to 2026-09-22 while every +# required check passed: ONE lane that no PR had to pass (macos-15-intel, on +# the main lane only) timed out on nearly every push. The README badge is the +# status of the whole `ci.yml` workflow on main, so ANY ci.yml job that can +# fail there colours it — required or not. The invariant that makes red mean +# "a merge broke something" is therefore structural, and this gate pins it: +# +# (1) every name in required-checks.txt is produced by EXACTLY ONE job +# across .github/workflows/, and that job reports on pull_request: +# its workflow triggers on pull_request for main with no path filter, +# and neither it nor any job it transitively `needs` is gated on the +# event (a required check that never reports blocks merges forever); +# (2) a required job without `if: always()` needs only REQUIRED jobs — a +# failed prerequisite SKIPS its dependants, and GitHub treats a +# `skipped` required check as satisfied; +# (3) every ci.yml job on the main lane is exactly one of +# (a) required — ALL of its matrix legs (a half-required matrix is red); +# (b) a worker whose ONLY consumer is a required aggregator that has +# `if: always()` and, in an unconditional step, compares this +# worker's `needs..result` against `success` (so a failed, +# cancelled or skipped worker fails the aggregator); the worker +# may not set `continue-on-error`; +# anything else must move to nightly.yml (tier 2); +# (4) every nightly.yml job is consumed by an `if: always()` reporter that +# checks its result and files an issue (`gh issue`) — a silent nightly +# is not a tier, it is a deletion. +# +# POPULATIONS (mechanical-gates §5/§121/§122). Both are DECLARED sets, so both +# are pinned exactly, each against a count taken by a different mechanism: +# * ci.yml job ids: the YAML loader's count == an awk count of the two-space +# keys under `jobs:` > 0; +# * required names: the parsed count == `grep -cv` of non-comment lines > 0. +# +# WHAT IT DOES NOT PROVE (residuals): +# * that the LIVE ruleset matches the file — that is `--live` (read-only +# `gh api`; needs a token that can read rulesets). CI does not run it: the +# ruleset is synced after merge, so a PR that edits the file is expected to +# differ from the live ruleset until then; +# * `if:` expressions are not evaluated. A job-level `if:` that mentions the +# event (github.event_name / github.event.* / github.ref*) on a required +# job or its ancestors is RED because it cannot be proven to report on a +# PR; a ci.yml job is taken off the main lane only by the exact +# `if: github.event_name == 'pull_request'`; +# * step-level `if:`s inside a required job are not judged (a required job +# whose steps all skip on a docs-only PR reports success by design — +# that is the `scope` job's contract); +# * checks from GitHub apps outside .github/workflows/ (CodeQL default +# setup's `Analyze (python)`, CodSpeed's app check) are invisible here; a +# required name produced only by an app would read as [unproduced]. +# +# Usage: +# tools/ci_tier_check.sh check the tree (exit 0 OK, 1 violation, +# 2 instrument error: no PyYAML, a file +# that does not load — never a verdict) +# tools/ci_tier_check.sh --selftest plant each violation class in a copy of +# the workflows and require the NAMED +# check to go red +# tools/ci_tier_check.sh --live diff required-checks.txt against the +# live ruleset (read-only) +# +# Env (the selftest points these at its copy): CI_TIER_WF_DIR, +# CI_TIER_REQUIRED, CI_TIER_MAIN (ci.yml), CI_TIER_NIGHTLY (nightly.yml). + +set -u +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +SELF="$ROOT/tools/ci_tier_check.sh" +WF_DIR="${CI_TIER_WF_DIR:-$ROOT/.github/workflows}" +REQ_FILE="${CI_TIER_REQUIRED:-$ROOT/.github/required-checks.txt}" +MAIN_WF="${CI_TIER_MAIN:-ci.yml}" +NIGHTLY_WF="${CI_TIER_NIGHTLY:-nightly.yml}" +RULESET_ID="${CI_TIER_RULESET_ID:-17713865}" +REPO="${CI_TIER_REPO:-InauguralSystems/EigenScript}" + +check() { + local main="$WF_DIR/$MAIN_WF" + [ -f "$main" ] || { echo "ci-tier: INSTRUMENT ERROR — $main does not exist; nothing was checked"; return 2; } + [ -f "$REQ_FILE" ] || { echo "ci-tier: INSTRUMENT ERROR — $REQ_FILE does not exist; nothing was checked"; return 2; } + local awk_jobs req_lines + # Independent count of ci.yml job ids: two-space keys under `jobs:` until + # the next top-level key. + awk_jobs=$(awk ' + /^jobs:[[:space:]]*(#.*)?$/ { j = 1; next } + j && /^[^[:space:]#]/ { j = 0 } + j && /^ [A-Za-z0-9_-]+:[[:space:]]*(#.*)?$/ { n++ } + END { print n + 0 }' "$main") + req_lines=$(grep -cvE '^(#|$)' "$REQ_FILE") + CT_WF_DIR="$WF_DIR" CT_REQ="$REQ_FILE" CT_MAIN="$MAIN_WF" CT_NIGHTLY="$NIGHTLY_WF" \ + CT_AWK_JOBS="$awk_jobs" CT_REQ_LINES="$req_lines" python3 - <<'PY' +import itertools, os, re, sys +from fnmatch import fnmatchcase + +def instrument(msg): + print(f"ci-tier: INSTRUMENT ERROR — {msg}; nothing was checked") + sys.exit(2) + +try: + import yaml +except Exception as e: # noqa: BLE001 — any import failure is an instrument failure + instrument(f"PyYAML unavailable ({e.__class__.__name__}: {e})") + +WF_DIR = os.environ["CT_WF_DIR"]; REQ = os.environ["CT_REQ"] +MAIN = os.environ["CT_MAIN"]; NIGHTLY = os.environ["CT_NIGHTLY"] +AWK_JOBS = int(os.environ["CT_AWK_JOBS"]); REQ_LINES = int(os.environ["CT_REQ_LINES"]) + +viol = [] +def V(code, msg): + viol.append(code) + print(f"FAIL [{code}] {msg}") + +# ---- required-checks.txt --------------------------------------------------- +required = [] +with open(REQ, encoding="utf-8") as fh: + for ln, line in enumerate(fh.read().split("\n"), 1): + if line == "" or line.startswith("#"): + continue + if line != line.strip(): + V("whitespace", f"{os.path.basename(REQ)}:{ln}: leading/trailing blank in {line!r} — a check name is matched EXACTLY") + required.append(line.strip()) +if not required: + V("vacuous", "required-checks.txt names no check — tier 1 would be empty") +if len(required) != REQ_LINES: + V("count-mismatch", f"parsed {len(required)} required names, grep counts {REQ_LINES} non-comment lines") +seen = set() +for r in required: + if r in seen: + V("dup-required", f"{r!r} is listed twice in required-checks.txt") + seen.add(r) +REQ_SET = set(required) + +# ---- workflows --------------------------------------------------------------- +def norm_on(on): + if on is None: return {} + if isinstance(on, str): return {on: {}} + if isinstance(on, list): return {str(x): {} for x in on} + if isinstance(on, dict): return {str(k): (v if isinstance(v, dict) else {}) for k, v in on.items()} + instrument(f"unrecognised `on:` shape {type(on).__name__}") + +WFS = {} +for fn in sorted(os.listdir(WF_DIR)): + if not fn.endswith((".yml", ".yaml")): continue + path = os.path.join(WF_DIR, fn) + try: + with open(path, encoding="utf-8") as fh: + doc = yaml.safe_load(fh) + except Exception as e: # noqa: BLE001 + instrument(f"{fn} does not load as YAML ({e.__class__.__name__})") + if not isinstance(doc, dict) or not isinstance(doc.get("jobs"), dict): + instrument(f"{fn} has no top-level `jobs:` mapping") + # PyYAML (YAML 1.1) reads the bare key `on` as the boolean True. + on = doc["on"] if "on" in doc else doc.get(True) + WFS[fn] = {"on": norm_on(on), "jobs": doc["jobs"]} +if MAIN not in WFS: instrument(f"{MAIN} not found among the workflows") +if not WFS: instrument("no workflow files") + +def branch_ok(cfg, br="main"): + if "branches" in cfg: + return any(fnmatchcase(br, str(p)) for p in (cfg["branches"] or [])) + if "branches-ignore" in cfg: + return not any(fnmatchcase(br, str(p)) for p in (cfg["branches-ignore"] or [])) + return True + +def pr_status(on): + """(True, '') when every PR to main triggers this workflow.""" + for key in ("pull_request", "pull_request_target"): + if key not in on: continue + cfg = on[key] + if not branch_ok(cfg): return False, f"`{key}` excludes main" + if "paths" in cfg or "paths-ignore" in cfg: + return False, f"`{key}` is path-filtered — a PR outside the filter never reports" + if "types" in cfg and not {"opened", "synchronize"} <= set(cfg["types"] or []): + return False, f"`{key}` types {cfg['types']} miss opened/synchronize" + return True, "" + return False, "the workflow does not trigger on pull_request" + +def pushes_main(on): + if "push" not in on: return False + cfg = on["push"] + if "branches" in cfg or "branches-ignore" in cfg: return branch_ok(cfg) + if "tags" in cfg or "tags-ignore" in cfg: return False + return True + +def if_str(job): + s = str(job.get("if", "")).strip() if isinstance(job, dict) else "" + m = re.fullmatch(r"\$\{\{\s*(.*?)\s*\}\}", s, re.S) + return m.group(1).strip() if m else s + +def is_always(job): return if_str(job) == "always()" +EVENT_RE = re.compile(r"github\.(event_name|event\.|ref\b|ref_name|ref_type|head_ref|base_ref)") +PR_ONLY = "github.event_name == 'pull_request'" + +def needs(job): + n = job.get("needs", []) if isinstance(job, dict) else [] + return [n] if isinstance(n, str) else list(n or []) + +class Unexpandable(Exception): pass +MATRIX_RE = re.compile(r"\$\{\{\s*matrix\.([A-Za-z0-9_-]+)\s*\}\}") + +def names_of(jid, job): + tmpl = job.get("name") + strat = job.get("strategy") or {} + if not isinstance(strat, dict): raise Unexpandable("strategy is an expression") + m = strat.get("matrix") + combos = [{}] + if m is not None: + if not isinstance(m, dict): raise Unexpandable("matrix is an expression") + keys = [k for k in m if k not in ("include", "exclude")] + for k in keys: + if not isinstance(m[k], list): raise Unexpandable(f"matrix.{k} is an expression") + combos = [dict(zip(keys, v)) for v in itertools.product(*[m[k] for k in keys])] if keys else [] + for ex in (m.get("exclude") or []): + combos = [c for c in combos if not all(c.get(k) == v for k, v in ex.items())] + # GitHub: an include entry extends every ORIGINAL combination whose + # original values it does not overwrite; if it extends none (always + # the case for an include-only matrix), it is a new combination. + base = list(combos) + for inc in (m.get("include") or []): + if not isinstance(inc, dict): raise Unexpandable("matrix.include entry is an expression") + hit = False + for c in (base if keys else []): + if all(c.get(k) == v for k, v in inc.items() if k in keys): + c.update({k: v for k, v in inc.items() if k not in keys}) + hit = True + if not hit: combos.append(dict(inc)) + if not combos: raise Unexpandable("matrix expands to zero legs") + out = [] + for c in combos: + if tmpl is None: + n = jid if not c else f"{jid} ({', '.join(str(v) for v in c.values())})" + else: + def sub(mo): + if mo.group(1) not in c: raise Unexpandable(f"matrix.{mo.group(1)} is not a matrix key") + return str(c[mo.group(1)]) + n = MATRIX_RE.sub(sub, str(tmpl)) + if "${{" in n: raise Unexpandable(f"name {tmpl!r} carries a non-matrix expression") + if n not in out: out.append(n) + return out + +NAMES = {} # (wf, jid) -> [check names] +PRODUCERS = {} # name -> [(wf, jid)] +for wf, w in WFS.items(): + for jid, job in w["jobs"].items(): + if not isinstance(job, dict): + instrument(f"{wf}: job {jid} is not a mapping") + try: + ns = names_of(jid, job) + except Unexpandable as e: + V("unexpandable", f"{wf}:{jid}: cannot derive its check name(s) — {e}") + ns = [] + NAMES[(wf, jid)] = ns + for n in ns: + PRODUCERS.setdefault(n, []).append((wf, jid)) + +def ancestors(wf, jid, seen=None): + seen = set() if seen is None else seen + for n in needs(WFS[wf]["jobs"].get(jid, {})): + if n not in seen: + seen.add(n) + if n in WFS[wf]["jobs"]: ancestors(wf, n, seen) + return seen + +def result_checked(agg, wid): + """Does an UNCONDITIONAL step of `agg` compare needs..result to success?""" + pat = re.compile(r"\$\{\{\s*needs\." + re.escape(wid) + r"\.result\s*\}\}") + for st in agg.get("steps") or []: + if not isinstance(st, dict) or "if" in st: continue + run = str(st.get("run") or "") + if not re.search(r"=\s*\"?success\b", run): continue + if pat.search(run): return True + for var, val in (st.get("env") or {}).items(): + if pat.fullmatch(str(val).strip()) and f'"${var}"' in run: return True + return False + +# ---- (1) + (2): every required name ------------------------------------------ +req_by_wf = {} +for r in required: + prods = PRODUCERS.get(r, []) + if not prods: + V("unproduced", f"required {r!r} is produced by no job in .github/workflows — it never reports, so it blocks every merge") + continue + if len(prods) > 1: + V("ambiguous", f"required {r!r} is produced by {len(prods)} jobs {prods} — either one would satisfy the rule") + continue + wf, jid = prods[0] + req_by_wf[wf] = req_by_wf.get(wf, 0) + 1 + ok, why = pr_status(WFS[wf]["on"]) + if not ok: + V("not-on-pr", f"required {r!r} ({wf}:{jid}) — {why}; a required check that never reports blocks merges forever") + for a in [jid] + sorted(ancestors(wf, jid)): + s = if_str(WFS[wf]["jobs"].get(a, {})) + if EVENT_RE.search(s): + V("not-on-pr", f"required {r!r}: {wf}:{a}{' (a job it needs)' if a != jid else ''} is gated on the event (`if: {s}`) — it cannot be shown to report on pull_request") + if not is_always(WFS[wf]["jobs"][jid]): + for n in needs(WFS[wf]["jobs"][jid]): + nn = NAMES.get((wf, n), []) + if not nn or not all(x in REQ_SET for x in nn): + V("need-not-required", f"required {r!r} ({wf}:{jid}) needs {n} {nn}, which is not required: when {n} fails, {jid} is SKIPPED and a skipped required check is satisfied") + +# ---- (3): the ci.yml main lane ----------------------------------------------- +main_jobs = WFS[MAIN]["jobs"] +if len(main_jobs) != AWK_JOBS or AWK_JOBS == 0: + V("count-mismatch", f"{MAIN}: the YAML loader sees {len(main_jobs)} jobs, the awk key count sees {AWK_JOBS} — a job the classifier examines differs from the job set in the file") +if not pushes_main(WFS[MAIN]["on"]): + V("vacuous", f"{MAIN} does not run on push to main — no main lane to classify") +counts = {"required": 0, "worker": 0, "pr-only": 0} +examined = 0 +print(f"{MAIN} main lane:") +for jid, job in main_jobs.items(): + examined += 1 + ns = NAMES[(MAIN, jid)] + if if_str(job) == PR_ONLY: + counts["pr-only"] += 1 + print(f" pr-only {jid}") + continue + if not ns: + continue # already [unexpandable] + inreq = [n in REQ_SET for n in ns] + if all(inreq): + counts["required"] += 1 + print(f" required {jid}: {' | '.join(ns)}") + continue + if any(inreq): + V("partial-matrix", f"{MAIN}:{jid}: legs {[n for n in ns if n in REQ_SET]} are required but {[n for n in ns if n not in REQ_SET]} are not — every leg colours main") + continue + consumers = [k for k, kj in main_jobs.items() if jid in needs(kj)] + tag = f"{MAIN}:{jid} {ns}" + if not consumers: + V("uncovered", f"{tag} runs on the main lane, is not in required-checks.txt, and no aggregator consumes it: it can turn main red without ever having blocked a merge — require it, make it a worker of a required aggregator, or move it to {NIGHTLY}") + continue + if len(consumers) > 1: + V("multi-consumer", f"{tag} is consumed by {consumers}; a worker is covered only when its ONE consumer is a required aggregator") + continue + agg = consumers[0] + an = NAMES.get((MAIN, agg), []) + bad = False + if not an or not all(x in REQ_SET for x in an): + V("uncovered", f"{tag}: its only consumer {agg} {an} is not required"); bad = True + elif not is_always(main_jobs[agg]): + V("agg-not-always", f"{tag}: aggregator {agg} lacks `if: always()` — when the worker fails the aggregator is SKIPPED, and a skipped required check passes"); bad = True + elif not result_checked(main_jobs[agg], jid): + V("agg-unchecked", f"{tag}: aggregator {agg} has no unconditional step comparing needs.{jid}.result to success — a failed/cancelled/skipped worker would not fail it"); bad = True + if job.get("continue-on-error") not in (None, False): + V("worker-continue-on-error", f"{tag} sets continue-on-error: its result reads success when it failed, so {agg} cannot see the failure"); bad = True + if not bad: + counts["worker"] += 1 + print(f" worker {jid}: {' | '.join(ns)} -> {agg}") +if examined != len(main_jobs) or examined == 0: + V("vacuous", f"examined {examined} of {len(main_jobs)} {MAIN} jobs") + +# ---- (4): nightly ------------------------------------------------------------- +n_jobs = WFS.get(NIGHTLY, {}).get("jobs", {}) +if not n_jobs: + V("nightly-unreported", f"{NIGHTLY} is missing or has no jobs — tier 2 has nowhere to live") +reporters = [j for j, jb in n_jobs.items() if is_always(jb)] +if n_jobs and not reporters: + V("nightly-unreported", f"{NIGHTLY} has no `if: always()` reporter job") +for rj in reporters: + # Code lines only: the reporter's own comments mention `gh issue` (§24). + runs = [ln for s in (n_jobs[rj].get("steps") or []) if isinstance(s, dict) + for ln in str(s.get("run") or "").split("\n") if not ln.lstrip().startswith("#")] + if not any("gh issue" in ln for ln in runs): + V("nightly-unreported", f"{NIGHTLY}:{rj} is the reporter but files no issue (`gh issue` never appears in its steps)") +tier2 = [] +for j in n_jobs: + if j in reporters: continue + tier2.append(j) + owners = [rj for rj in reporters if j in needs(n_jobs[rj])] + if not owners: + V("nightly-unreported", f"{NIGHTLY}:{j} is consumed by no reporter — a failure there is read by no one") + elif not any(result_checked(n_jobs[rj], j) for rj in owners): + V("nightly-unreported", f"{NIGHTLY}:{j}: its reporter never compares needs.{j}.result to success") + +# ---- other workflows: informational --------------------------------------------- +print("other workflows (tier by required-checks.txt; they do not colour the ci.yml badge):") +for (wf, jid), ns in sorted(NAMES.items()): + if wf in (MAIN, NIGHTLY): continue + for n in ns: + print(f" {'required' if n in REQ_SET else 'advisory'} {wf}: {n}") + +per = " ".join(f"{k}={v}" for k, v in sorted(req_by_wf.items())) +if viol: + print(f"ci-tier: FAIL — {len(viol)} violation(s): {' '.join(sorted(set(viol)))}") + sys.exit(1) +print(f"ci-tier: OK — {MAIN}: jobs={examined} (awk={AWK_JOBS}) required={counts['required']} worker={counts['worker']} pr-only={counts['pr-only']}; " + f"required-checks.txt: {len(required)} names (grep={REQ_LINES}), each produced once and reporting on pull_request [{per}]; " + f"{NIGHTLY}: {len(tier2)} tier-2 jobs, each reported ({', '.join(reporters)})") +PY +} + +live() { + command -v gh >/dev/null 2>&1 || { echo "ci-tier --live: INSTRUMENT ERROR — no gh CLI"; return 2; } + local got + got=$(gh api "repos/$REPO/rulesets/$RULESET_ID" \ + --jq '.rules[]|select(.type=="required_status_checks")|.parameters.required_status_checks[].context' 2>&1) \ + || { echo "ci-tier --live: INSTRUMENT ERROR — could not read ruleset $RULESET_ID: $got"; return 2; } + [ -n "$got" ] || { echo "ci-tier --live: INSTRUMENT ERROR — ruleset $RULESET_ID lists no required checks"; return 2; } + local want + want=$(grep -vE '^(#|$)' "$REQ_FILE") + local only_file only_live + only_file=$(comm -23 <(printf '%s\n' "$want" | LC_ALL=C sort) <(printf '%s\n' "$got" | LC_ALL=C sort)) + only_live=$(comm -13 <(printf '%s\n' "$want" | LC_ALL=C sort) <(printf '%s\n' "$got" | LC_ALL=C sort)) + local nf nl + nf=$(printf '%s\n' "$want" | grep -c .); nl=$(printf '%s\n' "$got" | grep -c .) + if [ -z "$only_file" ] && [ -z "$only_live" ]; then + echo "ci-tier --live: OK — ruleset $RULESET_ID requires exactly the $nf checks in required-checks.txt" + return 0 + fi + [ -n "$only_file" ] && printf ' in required-checks.txt, NOT in the ruleset: %s\n' "$only_file" | sed '2,$s/^/ in required-checks.txt, NOT in the ruleset: /' + [ -n "$only_live" ] && printf ' in the ruleset, NOT in required-checks.txt: %s\n' "$only_live" | sed '2,$s/^/ in the ruleset, NOT in required-checks.txt: /' + echo "ci-tier --live: DRIFT — file=$nf ruleset=$nl; sync the ruleset from the file" + return 1 +} + +selftest() { + local tmp pass=0 fail=0 broken=0 + tmp=$(mktemp -d "${TMPDIR:-/tmp}/ci_tier_selftest.XXXXXX") || { echo "selftest: cannot create a temp dir"; return 2; } + trap 'rm -rf "$tmp"' RETURN + echo "ci_tier_check selftest (every fault is planted in a copy of .github/)" + + fresh() { + rm -rf "$tmp/gh"; mkdir -p "$tmp/gh" + cp -R "$ROOT/.github/workflows" "$tmp/gh/workflows" + cp "$ROOT/.github/required-checks.txt" "$tmp/gh/required-checks.txt" + } + # sub FILE OLD NEW — replace exactly one occurrence, or the plant is BROKEN + # (a plant that edits nothing proves nothing: mechanical-gates §20/§137). + sub() { + python3 - "$1" "$2" "$3" <<'PY' +import sys +p, old, new = sys.argv[1], sys.argv[2], sys.argv[3] +s = open(p, encoding="utf-8").read() +n = s.count(old) +if n != 1 or old == new: + print(f"plant anchor found {n} times in {p} (or edits nothing): {old[:60]!r}") + sys.exit(3) +open(p, "w", encoding="utf-8").write(s.replace(old, new)) +PY + } + run_gate() { + CI_TIER_WF_DIR="$tmp/gh/workflows" CI_TIER_REQUIRED="$tmp/gh/required-checks.txt" \ + bash "$SELF" > "$tmp/out" 2>&1 + } + # expect NAME CODE — run the REAL gate on the planted copy; require rc 1 + # and the named check. + expect() { + local name="$1" code="$2" rc=0 + run_gate || rc=$? + if [ "$rc" -eq 1 ] && grep -qF "FAIL [$code]" "$tmp/out"; then + pass=$((pass + 1)); echo " PASS $name -> red via [$code]" + else + fail=$((fail + 1)); echo " FAIL $name: expected rc=1 with [$code], got rc=$rc"; sed 's/^/ | /' "$tmp/out" | tail -8 + fi + } + brk() { broken=$((broken + 1)); echo " BROKEN $1: the plant's anchor no longer exists — update the selftest"; } + + local CI="$tmp/gh/workflows/ci.yml" NY="$tmp/gh/workflows/nightly.yml" REQ="$tmp/gh/required-checks.txt" + + # 0. sanity start: the unmodified copy is green (else every red below is noise). + fresh + local rc=0; run_gate || rc=$? + if [ "$rc" -eq 0 ]; then pass=$((pass + 1)); echo " PASS unmodified copy is green" + else fail=$((fail + 1)); echo " FAIL unmodified copy: rc=$rc"; tail -8 "$tmp/out"; fi + + # 1. a new ci.yml job nobody requires or aggregates (the macos-15-intel shape) + fresh; printf '\n planted-lane:\n name: planted lane\n runs-on: ubuntu-latest\n steps:\n - run: "true"\n' >> "$CI" + expect "new main-lane job, not required, no aggregator" uncovered + + # 2. rename a required job + fresh; sub "$CI" "name: tsan (concurrency race gate)" "name: tsan (renamed)" && expect "required job renamed" unproduced || brk rename + + # 3. a required job made push-only + fresh; sub "$CI" " name: tsan (concurrency race gate) +" " name: tsan (concurrency race gate) + if: github.event_name == 'push' +" && expect "required job made push-only" not-on-pr || brk push-only + + # 4. a required job's PREREQUISITE made push-only (the check would skip on PRs) + fresh; sub "$CI" " scope: + name: scope +" " scope: + name: scope + if: github.ref == 'refs/heads/main' +" && expect "required job's prerequisite made push-only" not-on-pr || brk prereq-push-only + + # 5. another workflow's required check stops triggering on pull_request + fresh; sub "$tmp/gh/workflows/codeql.yml" " pull_request: + branches: [main] +" "" && expect "codeql.yml loses pull_request (Analyze C)" not-on-pr || brk codeql-pr + + # 6. a required workflow gains a path filter (docs-only PRs never report) + fresh; sub "$tmp/gh/workflows/pages.yml" " pull_request: + branches: [main] +" " pull_request: + branches: [main] + paths-ignore: ['**.md'] +" && expect "pages.yml path-filtered (playground)" not-on-pr || brk pages-paths + + # 7. an aggregator loses if: always() + fresh; sub "$CI" " needs: [scope, extensions-http, extensions-gfx, extensions-zlib, extensions-net] + if: always() +" " needs: [scope, extensions-http, extensions-gfx, extensions-zlib, extensions-net] +" && expect "extensions aggregator loses if: always()" agg-not-always || brk agg-always + + # 8. an aggregator stops checking one worker's result + fresh; sub "$CI" 'for result in "$HTTP_RESULT" "$GFX_RESULT" "$ZLIB_RESULT" "$NET_RESULT"; do' \ + 'for result in "$HTTP_RESULT" "$ZLIB_RESULT" "$NET_RESULT"; do' \ + && expect "extensions aggregator stops checking gfx" agg-unchecked || brk agg-unchecked + + # 9. a worker gains continue-on-error + fresh; sub "$CI" " name: extensions / zlib suite +" " name: extensions / zlib suite + continue-on-error: true +" && expect "worker sets continue-on-error" worker-continue-on-error || brk coe + + # 10. a worker gains a second consumer + fresh; sub "$CI" " name: bench (instruction-count regression gate) + needs: scope +" " name: bench (instruction-count regression gate) + needs: [scope, extensions-net] +" && expect "worker consumed by a second job" multi-consumer || brk multi + + # 11. half a matrix required + fresh; sub "$REQ" "linux / clang +" "" && expect "linux / clang dropped from the required set" partial-matrix || brk partial + + # 12. a required job's prerequisite dropped from the required set + fresh; sub "$REQ" "build dev/ci image +" "" && expect "build dev/ci image dropped" need-not-required || brk need + + # 13. a nightly job no reporter reads + fresh; printf '\n planted-nightly:\n name: nightly / planted\n runs-on: ubuntu-latest\n steps:\n - run: "true"\n' >> "$NY" + expect "nightly job outside the reporter's needs" nightly-unreported + + # 14. the nightly reporter stops filing issues + fresh; sub "$NY" 'gh issue create --repo "$REPO" --title "$TITLE" --body "$BODY" "${LABEL_ARGS[@]}"' 'echo would-create' \ + && sub "$NY" 'gh issue comment "$num" --repo "$REPO" --body "$BODY"' 'echo would-comment' \ + && sub "$NY" 'gh issue comment "$num" --repo "$REPO" \' 'echo \' \ + && sub "$NY" 'gh issue reopen "$num"' 'echo reopen' \ + && sub "$NY" 'num=$(gh issue list' 'num=$(echo' \ + && expect "nightly reporter files no issue" nightly-unreported || brk reporter + + # 15. a duplicated required name + fresh; printf 'scope\n' >> "$REQ"; expect "required name listed twice" dup-required + + # 16. a second producer of a required name + fresh; sub "$tmp/gh/workflows/scorecard.yml" " name: Scorecard analysis" " name: scope" \ + && expect "two jobs produce a required name" ambiguous || brk ambiguous + + # 17. a job the awk count sees but the loader does not agree on (flow style) + fresh; printf '\n planted-flow: {name: planted flow, runs-on: ubuntu-latest, steps: [{run: "true"}]}\n' >> "$CI" + expect "flow-style job the awk count cannot see" count-mismatch + + # 18. a job name the gate cannot expand + fresh; sub "$CI" " name: tsan (concurrency race gate)" ' name: tsan ${{ github.ref }}' \ + && expect "non-matrix expression in a job name" unexpandable || brk unexpandable + + # 19. an empty tier 1 + fresh; grep -E '^#' "$ROOT/.github/required-checks.txt" > "$REQ" + expect "required-checks.txt with no names" vacuous + + # 20. trailing blank on a name + # (the blank is built with printf: editors and hooks strip a literal one) + fresh; sub "$REQ" "tsan (concurrency race gate)" "tsan (concurrency race gate)$(printf ' ')" && expect "trailing blank on a required name" whitespace || brk whitespace + + # 21. no PyYAML -> an INSTRUMENT error (rc 2), never a verdict + fresh; mkdir -p "$tmp/noyaml"; printf 'raise ImportError("planted: no PyYAML")\n' > "$tmp/noyaml/yaml.py" + rc=0; PYTHONPATH="$tmp/noyaml" CI_TIER_WF_DIR="$tmp/gh/workflows" CI_TIER_REQUIRED="$REQ" \ + bash "$SELF" > "$tmp/out" 2>&1 || rc=$? + if [ "$rc" -eq 2 ] && grep -qF 'INSTRUMENT ERROR' "$tmp/out" && ! grep -qF 'ci-tier: OK' "$tmp/out"; then + pass=$((pass + 1)); echo " PASS no PyYAML -> rc 2 INSTRUMENT ERROR" + else fail=$((fail + 1)); echo " FAIL no PyYAML: rc=$rc"; tail -4 "$tmp/out"; fi + + echo "ci_tier_check selftest: checks=$((pass + fail + broken)) failures=$fail broken=$broken" + [ "$fail" -eq 0 ] && [ "$broken" -eq 0 ] +} + +case "${1:-}" in + "") check ;; + --selftest) selftest ;; + --live) live ;; + *) echo "usage: $0 [--selftest|--live]" >&2; exit 2 ;; +esac diff --git a/tools/docs_claims_populations.txt b/tools/docs_claims_populations.txt index 708b60ee..b0fd1807 100644 --- a/tools/docs_claims_populations.txt +++ b/tools/docs_claims_populations.txt @@ -57,7 +57,11 @@ NAMES|ROADMAP.md|6 # 63 -> 60 in #1255 round 3: the scope job and its path list were deleted from # pages.yml, and the CI.md paragraph that listed that path list (and the # path-filter wording before it) went with them. +# 60 -> 63 with #1264: the "Platform tiers" section (which replaced "Required +# status checks — what is actually required today") cites +# .github/required-checks.txt, tools/ci_tier_check.sh and nightly.yml where the +# old section cited fewer paths (all resolved). NUMBERS|docs/CI.md|4 -PATHS|docs/CI.md|60 +PATHS|docs/CI.md|63 TARGETS|docs/CI.md|4 NAMES|docs/CI.md|0 From 389d9525c222ffad6f609439d113dc6a75f15b0a Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Wed, 23 Sep 2026 01:39:40 -0500 Subject: [PATCH 2/6] ci: merge queue runs the full main lane; tier gate shrunk to what the queue cannot guarantee (#1264) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner decision (#1264): Rust's model — GitHub's merge queue runs the full main lane on the candidate (main + PRs ahead + this PR) before anything lands, so main is green by construction and nobody rebases to update. - ci.yml, codeql.yml, pages.yml trigger on merge_group (every workflow that produces a required check). - ci.yml: the eight main-lane-only step pairs were gated `github.event_name == 'push'` / `!= 'push'`; now `!= 'pull_request'` / `== 'pull_request'`, so everything push ran also runs in the queue (clang full suite, macOS full suite + [99i], the extension/db/asan-http full suites). The post-merge push run is kept: the badge reads it and it publishes the rolling ci-main image fork PRs run in. - pages.yml: deploy only on push/workflow_dispatch (never from a queue candidate); queue runs get their own concurrency group. - tools/ci_tier_check.sh, 592 -> 229 lines (scope cut by the orchestrator): required names produced once, on pull_request AND merge_group; no condition on a required path can run work on push but not in the queue (job-level: no event reference at all; step-level: only `event_name ==/!= 'pull_request'` or the PR payload; dot and bracket syntax, any case); no continue-on-error on a required path; every ci.yml job required or a worker of one required always() job. Aggregator script correctness is left to code review. Selftest: 9 plants + sanity start + no-PyYAML, incl. round-1 P1/P2/P4 and bracket syntax. - docs/CI.md: tier model with the merge queue and the push run's purpose; corrected round-1 claims (skipped-by-if is satisfied, not blocked; scope/dev-image dependencies; trigger lists of deploy/docker/scorecard); main-lane and risk sections now say the queue catches it before landing. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/required-checks.txt | 14 +- .github/workflows/ci.yml | 75 ++-- .github/workflows/codeql.yml | 2 + .github/workflows/pages.yml | 20 +- docs/CI.md | 151 ++++--- tools/ci_tier_check.sh | 746 +++++++++-------------------------- 6 files changed, 332 insertions(+), 676 deletions(-) diff --git a/.github/required-checks.txt b/.github/required-checks.txt index 95577eed..66c614c7 100644 --- a/.github/required-checks.txt +++ b/.github/required-checks.txt @@ -1,18 +1,20 @@ # TIER 1 (#1264) — the checks that block a merge to main. THE source of truth: # the "Protection" ruleset (id 17713865) is synced FROM this file after merge, -# never edited by hand. tools/ci_tier_check.sh gates it against the workflows -# (every name is produced by exactly one job that runs on pull_request; every -# ci.yml job on the main lane is listed here or is a worker of an aggregator -# listed here); `tools/ci_tier_check.sh --live` diffs it against the ruleset. +# never edited by hand. Required checks are evaluated in the MERGE QUEUE +# (merge_group), which runs the full main lane before anything lands. +# tools/ci_tier_check.sh gates this file against the workflows (each name is +# produced by exactly one job that reports on pull_request AND merge_group; +# no required path runs work only on push); `--live` diffs it against the +# ruleset. # # Format: one exact check-run name per line. A line starting with `#` is a # comment (give the reason for a non-obvious entry). No trailing blanks. # # ci.yml scope -# Every Linux leg runs inside this image; a failed prerequisite SKIPS the +# Every `container:` job runs inside this image; a failed prerequisite SKIPS the # required jobs that need it, and GitHub counts a skipped required check as -# passing — so a required job's `needs` must be required too (#1264). +# passing — so it is required like the jobs that need it (#1264). build dev/ci image werror audit ([99i], cached) gate self-tests (section plan + audit cache key) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e4cdb740..3a659402 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,6 +5,13 @@ on: branches: [main] pull_request: branches: [main] + # The merge queue (#1264, Rust's model): the FULL main lane runs on the + # queue's candidate (current main + the PRs queued ahead + this one) before + # anything lands, so main is green by construction. Every step below that + # is main-lane-only is gated `github.event_name != 'pull_request'`, which is + # true on push AND on merge_group — never `== 'push'`. + merge_group: + types: [checks_requested] permissions: contents: read @@ -43,20 +50,28 @@ concurrency: # probe gates (never a hand-written list). [99i] runs ONCE, in the # `werror audit` job, cached on the audit's inputs. macos-15-intel is # not on this lane. -# main lane (push to main) — the full matrix except macos-15-intel, with -# [99i] owned by the one `werror audit` job instead of being repeated ten -# times. The merge-er waits for this; contributors do not. +# main lane (merge_group AND push to main) — the full matrix except +# macos-15-intel. It runs in the MERGE QUEUE (#1264, Rust's model) on the +# candidate commit (main + the PRs queued ahead + this one) and nothing +# lands unless it is green, so main is green by construction and nobody +# rebases a PR to "update" it. Main-lane-only steps are gated +# `github.event_name != 'pull_request'` (true on push and merge_group), +# never `== 'push'`. The post-merge push run re-tests the commit the queue +# already tested: it is kept because the README badge reads it and it +# publishes the rolling `ci-main` dev image that fork PRs run in. # TIERS (#1264): every job in this file is TIER 1 — listed in # .github/required-checks.txt, or a worker of an aggregator listed there. # A job that is neither must go to nightly.yml; tools/ci_tier_check.sh -# (the `gate self-tests` job) fails otherwise. Adding a job here means -# adding its check name to that file (the ruleset is synced from it). +# (the `gate self-tests` job) fails otherwise, and also fails on a step +# whose condition would run it on push but not in the queue. Adding a job +# here means adding its check name to that file (the ruleset is synced +# from it). # nightly (.github/workflows/nightly.yml) — macos-15-intel (ONLY here, #1264) # and the full valgrind corpus, with a tracking issue on failure. # # The risk this accepts, stated: a variant-specific regression in a -# NON-variant section reaches main before it is caught. Main still runs the -# full matrix before anything is released. +# NON-variant section passes the PR lane and is caught in the queue, before +# it lands; the contributor waits only for the fast lane. jobs: # Is this PR docs-only? A change touching nothing but *.md cannot alter C or @@ -354,11 +369,11 @@ jobs: fi python3 -c 'import yaml; print("yaml", yaml.__version__)' # Platform tiers (#1264): every name in .github/required-checks.txt is - # produced by one job that reports on pull_request, and every job in - # THIS file that runs on a push to main is required or a worker of a - # required aggregator — so main's colour is decided by the merge gate. - # ~1 s + ~15 s of planted faults. Needs the PyYAML installed above; a - # missing loader is exit 2 (instrument error), never a pass. + # produced by one job that reports on pull_request AND merge_group, no + # required path runs work only on push, and every job in THIS file is + # required or a worker of a required aggregator. ~1 s + ~10 s of planted + # faults. Needs the PyYAML installed above; a missing loader is exit 2 + # (instrument error), never a pass. - if: needs.scope.outputs.code == 'true' name: Platform tiers — required-checks.txt vs the workflows (#1264) run: | @@ -460,7 +475,7 @@ jobs: # only thing it no longer runs is [99i], which the `werror audit` job # owns for this run and which prints a SKIP naming that job. - name: Run test suite (full) - if: matrix.cc == 'gcc' || github.event_name == 'push' + if: matrix.cc == 'gcc' || github.event_name != 'pull_request' env: EIGS_SKIP_WERROR_AUDIT: 1 # [99zd]'s live arms. Without this the roadmap gate's milestone and @@ -473,10 +488,10 @@ jobs: # clang on a PR: the value of this leg is the BUILD (-Werror fires at # compile time, and clang's codegen differs), not a tenth execution of # the same ~263 sections the gcc leg just ran on the same commit. It - # runs the derived core-smoke plan instead. On a push to main it runs + # runs the derived core-smoke plan instead. On the main lane (queue, push) it runs # the full suite like every other leg. - name: Run test suite (derived core-smoke plan) - if: matrix.cc == 'clang' && github.event_name != 'push' + if: matrix.cc == 'clang' && github.event_name == 'pull_request' env: EIGS_SUITE_SECTIONS: core EIGS_SKIP_WERROR_AUDIT: 1 @@ -703,15 +718,15 @@ jobs: # self-test (11 min) — the single largest block of this leg's 15 min. # Its verdict cannot differ by platform anyway: the only conditional in # the Makefile is LDFLAGS, a LINK flag, and the audit reads COMPILE - # invocations. On a push to main it still runs here in full, so the + # invocations. On the main lane (queue, push) it still runs here in full, so the # gate's BSD-userland portability keeps an exercise at merge time. - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run test suite (PR lane; [99i] owned by the werror audit job) env: EIGS_SKIP_WERROR_AUDIT: 1 run: cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run test suite (main lane, [99i] included) run: cd tests && bash run_all_tests.sh @@ -833,14 +848,14 @@ jobs: # decide whether to skip — plus a fixed core smoke, and the job fails if # the http binary unlocks fewer probe-gated chunks than its floor (a # broken registration otherwise collapses the plan silently). - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run derived http+model section plan env: EIGS_SUITE_SECTIONS: http EIGS_SKIP_WERROR_AUDIT: 1 run: cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run full suite against http+model build env: EIGS_SKIP_WERROR_AUDIT: 1 @@ -881,14 +896,14 @@ jobs: name: Build gfx variant run: make gfx - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run derived gfx section plan (audio [62], [120b], [132], [133], [134]) env: EIGS_SUITE_SECTIONS: gfx EIGS_SKIP_WERROR_AUDIT: 1 run: cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run full suite against gfx build (audio [62], containment [132], gfx examples [97]) env: EIGS_SKIP_WERROR_AUDIT: 1 @@ -915,14 +930,14 @@ jobs: name: Build zlib variant run: make zlib - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run derived zlib section plan (executes DEFLATE section [124b]) env: EIGS_SUITE_SECTIONS: zlib EIGS_SKIP_WERROR_AUDIT: 1 run: cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run full suite against zlib build (executes DEFLATE section [124]) env: EIGS_SKIP_WERROR_AUDIT: 1 @@ -949,14 +964,14 @@ jobs: name: Build net variant run: make net - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run derived net section plan (executes network section [125]) env: EIGS_SUITE_SECTIONS: net EIGS_SKIP_WERROR_AUDIT: 1 run: cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run full suite against net build (executes network section [125]) env: EIGS_SKIP_WERROR_AUDIT: 1 @@ -1002,7 +1017,7 @@ jobs: name: Build full variant (http+model+db) run: make full - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run derived full-variant section plan with live DATABASE_URL env: DATABASE_URL: postgres://eigs:eigs_test@db:5432/eigs_test @@ -1010,7 +1025,7 @@ jobs: EIGS_SKIP_WERROR_AUDIT: 1 run: cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run full suite with live DATABASE_URL env: DATABASE_URL: postgres://eigs:eigs_test@db:5432/eigs_test @@ -1302,7 +1317,7 @@ jobs: # sanitizers. This job was 26 min on #1158 — the second-longest on the # board — and the core sections it shared with `asan + ubsan / core and # LSP` are already sanitized there on the same commit. - - if: needs.scope.outputs.code == 'true' && github.event_name != 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name == 'pull_request' name: Run derived asan-http section plan under sanitizers env: ASAN_OPTIONS: detect_leaks=1 @@ -1311,7 +1326,7 @@ jobs: EIGS_SKIP_WERROR_AUDIT: 1 run: make asan-http && cd tests && bash run_all_tests.sh - - if: needs.scope.outputs.code == 'true' && github.event_name == 'push' + - if: needs.scope.outputs.code == 'true' && github.event_name != 'pull_request' name: Run suite under sanitizers with the HTTP+model extensions env: ASAN_OPTIONS: detect_leaks=1 diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 9aafd9e3..e725f8fe 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -5,6 +5,8 @@ on: branches: [main] pull_request: branches: [main] + merge_group: # the merge queue (#1264): a required check must report there + types: [checks_requested] schedule: - cron: '23 7 * * 1' diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index b286653b..0106c451 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -38,9 +38,10 @@ # check NAME independently of the worker's. THIS is the check # to require on main. # -# Deploy is unchanged: Configure Pages, the artifact upload and the `deploy` -# job run only on push/dispatch, never on a pull request, and deploy needs -# the aggregator. +# Deploy: Configure Pages and the artifact upload run on every non-PR event +# (push, dispatch, and the merge queue, so the queue exercises exactly what a +# push will); the `deploy` job runs only on push/dispatch — never from a PR or +# a queue candidate — and needs the aggregator. name: Docs site on: @@ -48,16 +49,19 @@ on: branches: [main] pull_request: branches: [main] + merge_group: # the merge queue (#1264): a required check must report there + types: [checks_requested] workflow_dispatch: permissions: contents: read # Deploys from main still serialize on ONE group, so a newer main push -# supersedes an older deploy. A pull request gets a group of its OWN ref: -# sharing `pages` would let every PR push cancel an in-flight main deploy. +# supersedes an older deploy. A pull request or a merge-queue candidate gets a +# group of its OWN ref: sharing `pages` would let it cancel an in-flight main +# deploy. concurrency: - group: ${{ github.event_name == 'pull_request' && format('pages-pr-{0}', github.ref) || 'pages' }} + group: ${{ (github.event_name == 'pull_request' || github.event_name == 'merge_group') && format('pages-pr-{0}', github.ref) || 'pages' }} cancel-in-progress: true jobs: @@ -132,7 +136,9 @@ jobs: echo "OK: the real emcc build of web/build.sh passed" deploy: - if: github.event_name != 'pull_request' + # Publish only what LANDED: never from a pull request, and never from a + # merge-queue candidate that may still be rejected (#1264). + if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' needs: playground runs-on: ubuntu-latest permissions: diff --git a/docs/CI.md b/docs/CI.md index a7563281..3602f4fd 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -525,14 +525,16 @@ All three tools carry a planted-fault `--selftest` with a pinned case count and a `--contract`, and the suite runs the live pass, the contract and the selftest of each as `[99zd]`. -## Main lane (push to `main`) — the full matrix +## Main lane (the merge queue, then push to `main`) — the full matrix Everything above runs in full: macOS (`macos-latest`), every variant job on the complete suite, `linux / clang` on the complete suite. The only thing that does not run ten times is [99i], which the `werror audit` job owns. -This is the real exit gate. #1138 and #1158 both carried lanes only CI could -run. Contributors never wait on it; whoever merges does. +This is the real exit gate, and it runs **in the merge queue** (`merge_group`) +on the commit that will land, before it lands — see **Platform tiers** below. +#1138 and #1158 both carried lanes only CI could run. Contributors never wait +on it and never rebase to satisfy it; the queue does both. ## Nightly (`.github/workflows/nightly.yml`) @@ -1038,77 +1040,79 @@ Main CI did not finish green from 2026-09-16 to 2026-09-22 although every required check passed on every merge: one lane that no pull request had to pass (`macos / macos-15-intel`, main lane only) hit its timeout on nearly every push. The README badge is the status of the whole `ci.yml` workflow on `main`, -so **any** `ci.yml` job that can fail there colours it, required or not. When -red is normal, a real regression is invisible. +so **any** `ci.yml` job that can fail there colours it, required or not. -The fix copies Rust, CPython and Go: +The fix copies Rust (tiers plus a merge queue), CPython and Go: - **Tier 1** — the checks listed in `.github/required-checks.txt`. They block - a merge (the ruleset requires them), and they are the only things on the - main lane, so they are what decides `main`'s colour. + a merge, and they are evaluated **in the merge queue**. +- **The merge queue.** A PR that passed the fast PR lane joins GitHub's merge + queue. The queue builds a candidate commit (current `main` + the PRs queued + ahead of it + this PR) and runs the **full main lane** on it (the + `merge_group` event). Nothing lands unless every required check is green + there, so `main` is green by construction, and **contributors never rebase + just to update a PR** — the queue tests the combination for them. Every + workflow that produces a required check (`ci.yml`, `codeql.yml`, + `pages.yml`) triggers on `merge_group`, and every main-lane-only step is + gated `github.event_name != 'pull_request'` (true on push *and* in the + queue), never `== 'push'`. +- **The post-merge push run** re-tests the commit the queue already tested. It + stays: the README badge reads it, it publishes the rolling `ci-main` dev + image that fork PRs run in, and `pages.yml` deploys the site only on push + (never from a queue candidate that may still be rejected). - **Tier 2** — slow and port lanes, in `.github/workflows/nightly.yml` (today `macos-15-intel` and the full valgrind corpus). They never colour `main`; a failure opens or appends to one tracking issue. ### The source of truth: `.github/required-checks.txt` -One exact check-run name per line; a line starting with `#` is a comment and -gives the reason for a non-obvious entry. The ruleset **"Protection"** -(`~DEFAULT_BRANCH`) is synced *from* this file — it is never edited by hand. -`bash tools/ci_tier_check.sh --live` diffs the file against the live ruleset -(read-only) and names every check on one side only. - -To change tier 1: edit the file in a PR (the gate below must stay green), merge, -and the orchestrator syncs the ruleset from the file; `--live` then prints OK. -Between the merge and the sync, `--live` reports the drift by name — that is -expected, and it is why CI does not run `--live`. +One exact check-run name per line; a line starting with `#` is a comment giving +the reason for a non-obvious entry. The ruleset **"Protection"** +(`~DEFAULT_BRANCH`) is synced *from* this file, never edited by hand. To change +tier 1: edit the file in a PR, merge, and the orchestrator syncs the ruleset; +`bash tools/ci_tier_check.sh --live` (read-only) then prints OK. Between merge +and sync it names the drift — expected, which is why CI does not run `--live`. ### What `tools/ci_tier_check.sh` enforces -It runs in the `gate self-tests` job on every code PR, followed by its -`--selftest` (planted faults, each required to go red through a *named* check). -It parses every workflow with PyYAML (a missing loader is exit 2, an instrument -error — never a pass) and fails when: - -- a required name is produced by **no** job (`[unproduced]`) or by more than one - (`[ambiguous]`); -- a required job may not report on a pull request (`[not-on-pr]`): its - workflow does not trigger on `pull_request` for `main`, the trigger is - path-filtered, or the job — or any job it transitively `needs` — has a - job-level `if:` that mentions the event. A required check that never reports - blocks every merge forever; -- a required job without `if: always()` needs a job that is not required - (`[need-not-required]`): a failed prerequisite **skips** its dependants, and - GitHub treats a skipped required check as satisfied; -- a `ci.yml` job on the main lane is neither required nor a covered worker - (`[uncovered]`), or only some of its matrix legs are required - (`[partial-matrix]`). This is the `macos-15-intel` shape, made red; -- a worker is not *covered* — its only consumer must be a required aggregator - (`[multi-consumer]` otherwise) that has `if: always()` (`[agg-not-always]`) - and compares `needs..result` to `success` in an unconditional step - (`[agg-unchecked]`), and the worker may not set `continue-on-error` - (`[worker-continue-on-error]`); -- a nightly job is not read by an `if: always()` reporter that checks its - result and files an issue (`[nightly-unreported]`); -- a population does not match its independent count (`[count-mismatch]`: the - YAML loader's `ci.yml` job count against an awk count of the keys under - `jobs:`, the parsed required names against a `grep` count), or is empty - (`[vacuous]`). - -It prints the classification of every job, so the tables below are its output -in prose, not a second list to keep in sync. +Only what the queue cannot guarantee by itself. It runs in the `gate +self-tests` job, followed by its `--selftest` (each planted fault must go red +through its named check). A missing PyYAML is exit 2, never a pass. + +- `[unproduced]` / `[ambiguous]` — a required name is produced by no job, or by + more than one. +- `[not-on-pr]` / `[not-in-queue]` — the producing workflow does not trigger on + `pull_request` to `main` (or is path-filtered), or does not trigger on + `merge_group`. Either way the check never *reports* there, and a required + check that never reports blocks every merge until someone overrides it. +- `[event-condition]` — on a required path (a required job, the jobs it + transitively needs, and the `ci.yml` workers), a condition could run work on + push that the queue skips. A job-level `if:` may not mention the event at + all: a job **skipped** by its `if:` reports a *satisfied* required check, so a + job-level event filter lets a merge through untested. A step `if:` may + mention the event only as `github.event_name ==/!= 'pull_request'`, or via + the PR payload `github.event.pull_request.*` (empty on push and in the + queue alike). Dot and bracket syntax are both read. +- `[continue-on-error]` — a job or step on a required path sets it, so its + failure would not fail the check. +- `[uncovered]` — a `ci.yml` job is neither required nor the worker of exactly + one required `if: always()` aggregator. A failing non-required job does not + stop the queue, yet it colours the badge: the `macos-15-intel` shape. + +Whether an aggregator's script really fails on every non-success worker +result is a code-review question, not this gate's. ### Every `ci.yml` job, classified | Job (check name) | Tier | Why | |---|---|---| -| `scope` | 1 | decides docs-only; every job needs it | -| `build dev/ci image` | 1 | every Linux leg runs inside it; required because the jobs that need it are required (a failed image would *skip* them) — added by #1264 | +| `scope` | 1 | decides docs-only; the runtime legs read its output | +| `build dev/ci image` | 1 | the image every `container:` job (the Linux legs, the extension/ASan workers, db, the audits, the differentials, freestanding) runs inside; required because required jobs `needs` it, and a failed prerequisite *skips* them — added by #1264 | | `werror audit ([99i], cached)` | 1 | gate | | `gate self-tests (section plan + audit cache key)` | 1 | gate; runs this checker | | `linux / gcc` | 1 | the one full suite on a PR | -| `linux / clang` | 1 | clang `-Werror` build + core smoke on a PR, full suite on main — added by #1264 ("Linux gcc/clang"); a half-required matrix is red | -| `macos / macos-latest` | 1 | the one macOS leg on the PR lane | +| `linux / clang` | 1 | clang `-Werror` build + core smoke on a PR, full suite on the main lane — added by #1264 (tier 1 is "Linux gcc/clang") | +| `macos / macos-latest` | 1 | the one macOS leg; full suite with [99i] on the main lane | | `extensions (http+model+gfx suite; embed/lsp/jit-smoke)` | 1 | **aggregator** | | `extensions / http+model and ancillary checks`, `/ gfx suite`, `/ zlib suite`, `/ net suite` | 1, via the aggregator | workers | | `asan + ubsan (full suite)` | 1 | **aggregator**; also re-derives shard coverage and sums the leak tally | @@ -1119,13 +1123,12 @@ in prose, not a second list to keep in sync. | `valgrind (memcheck smoke, JIT off)` | 1 | the smoke spread (the full corpus is tier 2) | | `tsan (concurrency race gate)` | 1 | gate | | `install.sh (interpreter + eigenlsp on PATH)` | 1 | gate | -| `bench (instruction-count regression gate)` | 1 | gate | +| `bench (instruction-count regression gate)` | 1 | gate (compares against `origin/main`, in the queue too) | | `nightly / macos-15-intel full suite` | **2** (`nightly.yml`) | port lane, slow: hit its timeout on nearly every main push (#1265) | | `nightly / valgrind (full corpus, JIT off)` | **2** (`nightly.yml`) | slow; the PR and main lanes run the smoke spread | No `ci.yml` job moved to nightly in #1264 beyond `macos-15-intel` (#1265): -every other job already runs on pull requests, so each was made tier 1 rather -than demoted. +every other job already runs on pull requests, so each was made tier 1. ### Checks from other workflows @@ -1134,31 +1137,21 @@ file: | Check (workflow) | Tier | Why | |---|---|---| -| `Analyze C` (`codeql.yml`) | 1 | runs on every PR to `main`, no path filter | -| `playground (real emcc wasm32 build)` (`pages.yml`) | 1 | **aggregator** over `playground / build (…)`; reports on every PR | +| `Analyze C` (`codeql.yml`) | 1 | runs on every PR to `main` and in the queue, no path filter | +| `playground (real emcc wasm32 build)` (`pages.yml`) | 1 | **aggregator** over `playground / build (…)`; reports on every PR and in the queue | | `playground / build (real emcc, web/build.sh) + docs site` (`pages.yml`) | advisory | the worker; required through the aggregator | -| `deploy` (`pages.yml`) | advisory | push-only (`if: github.event_name != 'pull_request'`) — could never be required | -| `codspeed (simulation)` (`codspeed.yml`), and the CodSpeed app's `CodSpeed Performance Analysis` | advisory | path-filtered (`paths-ignore: '**.md'`), so it does not report on docs-only PRs; the instruction-count gate that blocks is `bench` | -| `build` (`docker.yml`) | advisory | push and tags only | -| `Scorecard analysis` (`scorecard.yml`) | advisory | push/schedule only; a posture score, not a correctness gate | +| `deploy` (`pages.yml`) | advisory | runs only on push and `workflow_dispatch` — it publishes, it does not test | +| `codspeed (simulation)` (`codspeed.yml`), and the CodSpeed app's `CodSpeed Performance Analysis` | advisory | path-filtered (`paths-ignore: '**.md'`), so it never reports on a docs-only PR; the instruction-count gate that blocks is `bench` | +| `build` (`docker.yml`) | advisory | runs on push to `main`, `v*` tags and `workflow_dispatch`, never on a PR | +| `Scorecard analysis` (`scorecard.yml`) | advisory | runs on push, schedule and `branch_protection_rule`, never on a PR; a posture score | | `Analyze (python)`, `Analyze (javascript-typescript)` | advisory | CodeQL *default setup* (a GitHub app, not a workflow file) over the repo's non-C code; the C analysis that blocks is `Analyze C` | | `issue-triage / …` (`issue-triage.yml`), `release.yml` jobs | advisory | not triggered by PRs or pushes to `main` | -### The two traps the tier set has to respect - -- **A required check that never reports blocks every merge forever.** That is - why `macos-15-intel` could never have been made required, why the `scope` - job's runtime legs report success in seconds on a docs-only PR instead of - being skipped, and why `[not-on-pr]` exists. -- **A skipped required check is satisfied.** That is why the aggregators run - `if: always()` and treat `skipped`, `cancelled` and a missing result as - failure, and why a required job's prerequisites must be required too. - ## The risk this accepts -A variant-specific regression in a *non-variant* section reaches `main` before -anything catches it — for example a clang-only miscompile in a section the -core-smoke plan does not cover. Main runs the full matrix before anything is -released, so the window is between merge and the next main run, and nothing -ships through it. That trade is deliberate: it buys back roughly half the -machine-minutes and more than half the contributor wait. +A variant-specific regression in a *non-variant* section — for example a +clang-only miscompile in a section the core-smoke plan does not cover — passes +the fast PR lane. It is caught in the merge queue, which runs the full matrix +before the PR lands, so it never reaches `main`; the cost is a rejected queue +entry instead of a red PR check. That trade is deliberate: it buys back roughly +half the machine-minutes and more than half the contributor wait. diff --git a/tools/ci_tier_check.sh b/tools/ci_tier_check.sh index e8299be7..dca76bd2 100755 --- a/tools/ci_tier_check.sh +++ b/tools/ci_tier_check.sh @@ -1,592 +1,230 @@ #!/usr/bin/env bash -# tools/ci_tier_check.sh — platform tiers (#1264): the main lane's colour means -# something. +# tools/ci_tier_check.sh — platform tiers (#1264), the part the merge queue +# cannot guarantee by itself. # -# THE MODEL (copied from Rust / CPython / Go, owner decision on #1264): -# TIER 1 — the checks in .github/required-checks.txt. They block a merge -# (the "Protection" ruleset is synced FROM that file) and they are -# what decides whether `main` is green. -# TIER 2 — slow or port lanes (.github/workflows/nightly.yml). They never -# colour main; a failure opens or appends to a tracking issue. +# Tier 1 = the names in .github/required-checks.txt (the ruleset is synced from +# that file). The merge queue runs the full main lane on `merge_group` and +# lands nothing that is not green there, so main is green by construction — +# PROVIDED the required checks report in the queue and test what push tests. +# That proviso is what this gate pins: +# [unproduced]/[ambiguous] every required name is produced by exactly one +# job in .github/workflows/; +# [not-on-pr]/[not-in-queue] its workflow triggers on pull_request (for main, +# no path filter) AND on merge_group; +# [event-condition] on a REQUIRED PATH (a required job, the jobs it +# transitively needs, and ci.yml workers) no +# condition can run work on push but not in the +# queue. Job-level `if:` may not mention the event +# at all; a step `if:` may mention it only as +# `github.event_name ==/!= 'pull_request'` (same +# truth on push and merge_group) or through the +# PR payload `github.event.pull_request.*` (empty on +# both). Dot and bracket syntax, any case; +# [continue-on-error] no job or step on a required path sets it — a +# failure there would not fail the check; +# [uncovered] every ci.yml job is required, or a worker whose +# ONE consumer is a required `if: always()` job (a +# non-required job that fails does not block the +# queue, yet colours main's badge). +# Whether an aggregator's script really fails on a bad worker result is a +# code-review question, not this gate's. Populations: the loader's ci.yml job +# count == an awk count > 0; required names == a grep count > 0. # -# WHY A GATE. Main CI was red from 2026-09-16 to 2026-09-22 while every -# required check passed: ONE lane that no PR had to pass (macos-15-intel, on -# the main lane only) timed out on nearly every push. The README badge is the -# status of the whole `ci.yml` workflow on main, so ANY ci.yml job that can -# fail there colours it — required or not. The invariant that makes red mean -# "a merge broke something" is therefore structural, and this gate pins it: -# -# (1) every name in required-checks.txt is produced by EXACTLY ONE job -# across .github/workflows/, and that job reports on pull_request: -# its workflow triggers on pull_request for main with no path filter, -# and neither it nor any job it transitively `needs` is gated on the -# event (a required check that never reports blocks merges forever); -# (2) a required job without `if: always()` needs only REQUIRED jobs — a -# failed prerequisite SKIPS its dependants, and GitHub treats a -# `skipped` required check as satisfied; -# (3) every ci.yml job on the main lane is exactly one of -# (a) required — ALL of its matrix legs (a half-required matrix is red); -# (b) a worker whose ONLY consumer is a required aggregator that has -# `if: always()` and, in an unconditional step, compares this -# worker's `needs..result` against `success` (so a failed, -# cancelled or skipped worker fails the aggregator); the worker -# may not set `continue-on-error`; -# anything else must move to nightly.yml (tier 2); -# (4) every nightly.yml job is consumed by an `if: always()` reporter that -# checks its result and files an issue (`gh issue`) — a silent nightly -# is not a tier, it is a deletion. -# -# POPULATIONS (mechanical-gates §5/§121/§122). Both are DECLARED sets, so both -# are pinned exactly, each against a count taken by a different mechanism: -# * ci.yml job ids: the YAML loader's count == an awk count of the two-space -# keys under `jobs:` > 0; -# * required names: the parsed count == `grep -cv` of non-comment lines > 0. -# -# WHAT IT DOES NOT PROVE (residuals): -# * that the LIVE ruleset matches the file — that is `--live` (read-only -# `gh api`; needs a token that can read rulesets). CI does not run it: the -# ruleset is synced after merge, so a PR that edits the file is expected to -# differ from the live ruleset until then; -# * `if:` expressions are not evaluated. A job-level `if:` that mentions the -# event (github.event_name / github.event.* / github.ref*) on a required -# job or its ancestors is RED because it cannot be proven to report on a -# PR; a ci.yml job is taken off the main lane only by the exact -# `if: github.event_name == 'pull_request'`; -# * step-level `if:`s inside a required job are not judged (a required job -# whose steps all skip on a docs-only PR reports success by design — -# that is the `scope` job's contract); -# * checks from GitHub apps outside .github/workflows/ (CodeQL default -# setup's `Analyze (python)`, CodSpeed's app check) are invisible here; a -# required name produced only by an app would read as [unproduced]. -# -# Usage: -# tools/ci_tier_check.sh check the tree (exit 0 OK, 1 violation, -# 2 instrument error: no PyYAML, a file -# that does not load — never a verdict) -# tools/ci_tier_check.sh --selftest plant each violation class in a copy of -# the workflows and require the NAMED -# check to go red -# tools/ci_tier_check.sh --live diff required-checks.txt against the -# live ruleset (read-only) -# -# Env (the selftest points these at its copy): CI_TIER_WF_DIR, -# CI_TIER_REQUIRED, CI_TIER_MAIN (ci.yml), CI_TIER_NIGHTLY (nightly.yml). - +# Usage: tools/ci_tier_check.sh [--selftest | --live] +# (no flag) exit 0 OK, 1 violation, 2 instrument error (no PyYAML, a file +# that does not load) — never a verdict +# --selftest plants each class in a copy of .github/ and requires the named +# check to go red +# --live read-only diff of required-checks.txt against the live ruleset +# (not run in CI: the ruleset is synced after merge) set -u ROOT="$(cd "$(dirname "$0")/.." && pwd)" -SELF="$ROOT/tools/ci_tier_check.sh" WF_DIR="${CI_TIER_WF_DIR:-$ROOT/.github/workflows}" REQ_FILE="${CI_TIER_REQUIRED:-$ROOT/.github/required-checks.txt}" -MAIN_WF="${CI_TIER_MAIN:-ci.yml}" -NIGHTLY_WF="${CI_TIER_NIGHTLY:-nightly.yml}" -RULESET_ID="${CI_TIER_RULESET_ID:-17713865}" -REPO="${CI_TIER_REPO:-InauguralSystems/EigenScript}" +SELF="$ROOT/tools/ci_tier_check.sh" check() { - local main="$WF_DIR/$MAIN_WF" - [ -f "$main" ] || { echo "ci-tier: INSTRUMENT ERROR — $main does not exist; nothing was checked"; return 2; } - [ -f "$REQ_FILE" ] || { echo "ci-tier: INSTRUMENT ERROR — $REQ_FILE does not exist; nothing was checked"; return 2; } - local awk_jobs req_lines - # Independent count of ci.yml job ids: two-space keys under `jobs:` until - # the next top-level key. - awk_jobs=$(awk ' - /^jobs:[[:space:]]*(#.*)?$/ { j = 1; next } - j && /^[^[:space:]#]/ { j = 0 } - j && /^ [A-Za-z0-9_-]+:[[:space:]]*(#.*)?$/ { n++ } - END { print n + 0 }' "$main") - req_lines=$(grep -cvE '^(#|$)' "$REQ_FILE") - CT_WF_DIR="$WF_DIR" CT_REQ="$REQ_FILE" CT_MAIN="$MAIN_WF" CT_NIGHTLY="$NIGHTLY_WF" \ - CT_AWK_JOBS="$awk_jobs" CT_REQ_LINES="$req_lines" python3 - <<'PY' + [ -f "$WF_DIR/ci.yml" ] && [ -f "$REQ_FILE" ] \ + || { echo "ci-tier: INSTRUMENT ERROR — ci.yml or required-checks.txt missing"; return 2; } + CT_AWK=$(awk '/^jobs:/ {j=1; next} j && /^[^ #]/ {j=0} + j && /^ [A-Za-z0-9_-]+:[[:space:]]*(#.*)?$/ {n++} END {print n+0}' "$WF_DIR/ci.yml") \ + CT_GREP=$(grep -cvE '^(#|$)' "$REQ_FILE") CT_WF="$WF_DIR" CT_REQ="$REQ_FILE" python3 - <<'PY' import itertools, os, re, sys from fnmatch import fnmatchcase - -def instrument(msg): - print(f"ci-tier: INSTRUMENT ERROR — {msg}; nothing was checked") - sys.exit(2) - +def instrument(m): print(f"ci-tier: INSTRUMENT ERROR — {m}; nothing was checked"); sys.exit(2) try: import yaml -except Exception as e: # noqa: BLE001 — any import failure is an instrument failure - instrument(f"PyYAML unavailable ({e.__class__.__name__}: {e})") - -WF_DIR = os.environ["CT_WF_DIR"]; REQ = os.environ["CT_REQ"] -MAIN = os.environ["CT_MAIN"]; NIGHTLY = os.environ["CT_NIGHTLY"] -AWK_JOBS = int(os.environ["CT_AWK_JOBS"]); REQ_LINES = int(os.environ["CT_REQ_LINES"]) - -viol = [] -def V(code, msg): - viol.append(code) - print(f"FAIL [{code}] {msg}") - -# ---- required-checks.txt --------------------------------------------------- -required = [] -with open(REQ, encoding="utf-8") as fh: - for ln, line in enumerate(fh.read().split("\n"), 1): - if line == "" or line.startswith("#"): - continue - if line != line.strip(): - V("whitespace", f"{os.path.basename(REQ)}:{ln}: leading/trailing blank in {line!r} — a check name is matched EXACTLY") - required.append(line.strip()) -if not required: - V("vacuous", "required-checks.txt names no check — tier 1 would be empty") -if len(required) != REQ_LINES: - V("count-mismatch", f"parsed {len(required)} required names, grep counts {REQ_LINES} non-comment lines") -seen = set() -for r in required: - if r in seen: - V("dup-required", f"{r!r} is listed twice in required-checks.txt") - seen.add(r) -REQ_SET = set(required) - -# ---- workflows --------------------------------------------------------------- -def norm_on(on): - if on is None: return {} - if isinstance(on, str): return {on: {}} - if isinstance(on, list): return {str(x): {} for x in on} - if isinstance(on, dict): return {str(k): (v if isinstance(v, dict) else {}) for k, v in on.items()} - instrument(f"unrecognised `on:` shape {type(on).__name__}") - -WFS = {} -for fn in sorted(os.listdir(WF_DIR)): +except Exception as e: # noqa: BLE001 + instrument(f"PyYAML unavailable ({e})") +bad = [] +def V(code, msg): bad.append(code); print(f"FAIL [{code}] {msg}") + +req = [l for l in open(os.environ["CT_REQ"], encoding="utf-8").read().split("\n") if l and not l.startswith("#")] +if not req or len(req) != int(os.environ["CT_GREP"]) or len(set(req)) != len(req): + V("vacuous", f"required-checks.txt: parsed {len(req)} names, {len(set(req))} distinct, grep counts {os.environ['CT_GREP']}") +REQ = set(req) + +W = {} +for fn in sorted(os.listdir(os.environ["CT_WF"])): if not fn.endswith((".yml", ".yaml")): continue - path = os.path.join(WF_DIR, fn) - try: - with open(path, encoding="utf-8") as fh: - doc = yaml.safe_load(fh) - except Exception as e: # noqa: BLE001 - instrument(f"{fn} does not load as YAML ({e.__class__.__name__})") - if not isinstance(doc, dict) or not isinstance(doc.get("jobs"), dict): - instrument(f"{fn} has no top-level `jobs:` mapping") - # PyYAML (YAML 1.1) reads the bare key `on` as the boolean True. - on = doc["on"] if "on" in doc else doc.get(True) - WFS[fn] = {"on": norm_on(on), "jobs": doc["jobs"]} -if MAIN not in WFS: instrument(f"{MAIN} not found among the workflows") -if not WFS: instrument("no workflow files") - -def branch_ok(cfg, br="main"): - if "branches" in cfg: - return any(fnmatchcase(br, str(p)) for p in (cfg["branches"] or [])) - if "branches-ignore" in cfg: - return not any(fnmatchcase(br, str(p)) for p in (cfg["branches-ignore"] or [])) - return True - -def pr_status(on): - """(True, '') when every PR to main triggers this workflow.""" - for key in ("pull_request", "pull_request_target"): - if key not in on: continue - cfg = on[key] - if not branch_ok(cfg): return False, f"`{key}` excludes main" - if "paths" in cfg or "paths-ignore" in cfg: - return False, f"`{key}` is path-filtered — a PR outside the filter never reports" - if "types" in cfg and not {"opened", "synchronize"} <= set(cfg["types"] or []): - return False, f"`{key}` types {cfg['types']} miss opened/synchronize" - return True, "" - return False, "the workflow does not trigger on pull_request" - -def pushes_main(on): - if "push" not in on: return False - cfg = on["push"] - if "branches" in cfg or "branches-ignore" in cfg: return branch_ok(cfg) - if "tags" in cfg or "tags-ignore" in cfg: return False - return True - -def if_str(job): - s = str(job.get("if", "")).strip() if isinstance(job, dict) else "" - m = re.fullmatch(r"\$\{\{\s*(.*?)\s*\}\}", s, re.S) - return m.group(1).strip() if m else s - -def is_always(job): return if_str(job) == "always()" -EVENT_RE = re.compile(r"github\.(event_name|event\.|ref\b|ref_name|ref_type|head_ref|base_ref)") -PR_ONLY = "github.event_name == 'pull_request'" - -def needs(job): - n = job.get("needs", []) if isinstance(job, dict) else [] - return [n] if isinstance(n, str) else list(n or []) - -class Unexpandable(Exception): pass -MATRIX_RE = re.compile(r"\$\{\{\s*matrix\.([A-Za-z0-9_-]+)\s*\}\}") - -def names_of(jid, job): - tmpl = job.get("name") - strat = job.get("strategy") or {} - if not isinstance(strat, dict): raise Unexpandable("strategy is an expression") - m = strat.get("matrix") + try: doc = yaml.safe_load(open(os.path.join(os.environ["CT_WF"], fn), encoding="utf-8")) + except Exception as e: instrument(f"{fn} does not load ({e.__class__.__name__})") # noqa: BLE001 + if not isinstance(doc, dict) or not isinstance(doc.get("jobs"), dict): instrument(f"{fn} has no jobs: mapping") + on = doc.get("on", doc.get(True)) # YAML 1.1 reads bare `on` as True + on = {on: {}} if isinstance(on, str) else {str(k): {} for k in on} if isinstance(on, list) else \ + {str(k): (v if isinstance(v, dict) else {}) for k, v in (on or {}).items()} + W[fn] = (on, doc["jobs"]) + +def triggers(on, ev): + if ev not in on: return False + c = on[ev] + if "paths" in c or "paths-ignore" in c: return False + if "branches" in c: return any(fnmatchcase("main", str(p)) for p in c["branches"] or []) + return not any(fnmatchcase("main", str(p)) for p in c.get("branches-ignore") or []) + +def cond(x): + s = str(x).strip(); m = re.fullmatch(r"\$\{\{(.*)\}\}", s, re.S) + return (m.group(1) if m else s).strip() +def needs(j): n = j.get("needs") or []; return [n] if isinstance(n, str) else list(n) +def always(j): return cond(j.get("if", "")) == "always()" +G = r"github\s*(?:\.\s*{0}\b|\[\s*['\"]{0}['\"]\s*\])" +PR_ATOM = re.compile(G.format("event_name") + r"\s*[=!]=\s*['\"]pull_request['\"]|['\"]pull_request['\"]\s*[=!]=\s*" + G.format("event_name"), re.I) +PR_BODY = re.compile(G.format("event") + r"\s*(?:\.\s*pull_request\b|\[\s*['\"]pull_request['\"]\s*\])", re.I) +def github_refs(s, step): + if step: s = PR_BODY.sub("_", PR_ATOM.sub("_", s)) + return re.search(r"\bgithub\b", s, re.I) + +def names(jid, j): + m = (j.get("strategy") or {}).get("matrix") if isinstance(j.get("strategy"), dict) else None combos = [{}] - if m is not None: - if not isinstance(m, dict): raise Unexpandable("matrix is an expression") + if isinstance(m, dict): keys = [k for k in m if k not in ("include", "exclude")] - for k in keys: - if not isinstance(m[k], list): raise Unexpandable(f"matrix.{k} is an expression") combos = [dict(zip(keys, v)) for v in itertools.product(*[m[k] for k in keys])] if keys else [] - for ex in (m.get("exclude") or []): - combos = [c for c in combos if not all(c.get(k) == v for k, v in ex.items())] - # GitHub: an include entry extends every ORIGINAL combination whose - # original values it does not overwrite; if it extends none (always - # the case for an include-only matrix), it is a new combination. - base = list(combos) - for inc in (m.get("include") or []): - if not isinstance(inc, dict): raise Unexpandable("matrix.include entry is an expression") - hit = False - for c in (base if keys else []): - if all(c.get(k) == v for k, v in inc.items() if k in keys): - c.update({k: v for k, v in inc.items() if k not in keys}) - hit = True - if not hit: combos.append(dict(inc)) - if not combos: raise Unexpandable("matrix expands to zero legs") + combos += [i for i in m.get("include") or [] if not keys] out = [] for c in combos: - if tmpl is None: - n = jid if not c else f"{jid} ({', '.join(str(v) for v in c.values())})" - else: - def sub(mo): - if mo.group(1) not in c: raise Unexpandable(f"matrix.{mo.group(1)} is not a matrix key") - return str(c[mo.group(1)]) - n = MATRIX_RE.sub(sub, str(tmpl)) - if "${{" in n: raise Unexpandable(f"name {tmpl!r} carries a non-matrix expression") + n = re.sub(r"\$\{\{\s*matrix\.([\w-]+)\s*\}\}", lambda mo: str(c.get(mo.group(1), "?")), str(j.get("name", jid))) if n not in out: out.append(n) return out - -NAMES = {} # (wf, jid) -> [check names] -PRODUCERS = {} # name -> [(wf, jid)] -for wf, w in WFS.items(): - for jid, job in w["jobs"].items(): - if not isinstance(job, dict): - instrument(f"{wf}: job {jid} is not a mapping") - try: - ns = names_of(jid, job) - except Unexpandable as e: - V("unexpandable", f"{wf}:{jid}: cannot derive its check name(s) — {e}") - ns = [] - NAMES[(wf, jid)] = ns - for n in ns: - PRODUCERS.setdefault(n, []).append((wf, jid)) - -def ancestors(wf, jid, seen=None): - seen = set() if seen is None else seen - for n in needs(WFS[wf]["jobs"].get(jid, {})): - if n not in seen: - seen.add(n) - if n in WFS[wf]["jobs"]: ancestors(wf, n, seen) - return seen - -def result_checked(agg, wid): - """Does an UNCONDITIONAL step of `agg` compare needs..result to success?""" - pat = re.compile(r"\$\{\{\s*needs\." + re.escape(wid) + r"\.result\s*\}\}") - for st in agg.get("steps") or []: - if not isinstance(st, dict) or "if" in st: continue - run = str(st.get("run") or "") - if not re.search(r"=\s*\"?success\b", run): continue - if pat.search(run): return True - for var, val in (st.get("env") or {}).items(): - if pat.fullmatch(str(val).strip()) and f'"${var}"' in run: return True - return False - -# ---- (1) + (2): every required name ------------------------------------------ -req_by_wf = {} -for r in required: - prods = PRODUCERS.get(r, []) - if not prods: - V("unproduced", f"required {r!r} is produced by no job in .github/workflows — it never reports, so it blocks every merge") - continue - if len(prods) > 1: - V("ambiguous", f"required {r!r} is produced by {len(prods)} jobs {prods} — either one would satisfy the rule") - continue - wf, jid = prods[0] - req_by_wf[wf] = req_by_wf.get(wf, 0) + 1 - ok, why = pr_status(WFS[wf]["on"]) - if not ok: - V("not-on-pr", f"required {r!r} ({wf}:{jid}) — {why}; a required check that never reports blocks merges forever") - for a in [jid] + sorted(ancestors(wf, jid)): - s = if_str(WFS[wf]["jobs"].get(a, {})) - if EVENT_RE.search(s): - V("not-on-pr", f"required {r!r}: {wf}:{a}{' (a job it needs)' if a != jid else ''} is gated on the event (`if: {s}`) — it cannot be shown to report on pull_request") - if not is_always(WFS[wf]["jobs"][jid]): - for n in needs(WFS[wf]["jobs"][jid]): - nn = NAMES.get((wf, n), []) - if not nn or not all(x in REQ_SET for x in nn): - V("need-not-required", f"required {r!r} ({wf}:{jid}) needs {n} {nn}, which is not required: when {n} fails, {jid} is SKIPPED and a skipped required check is satisfied") - -# ---- (3): the ci.yml main lane ----------------------------------------------- -main_jobs = WFS[MAIN]["jobs"] -if len(main_jobs) != AWK_JOBS or AWK_JOBS == 0: - V("count-mismatch", f"{MAIN}: the YAML loader sees {len(main_jobs)} jobs, the awk key count sees {AWK_JOBS} — a job the classifier examines differs from the job set in the file") -if not pushes_main(WFS[MAIN]["on"]): - V("vacuous", f"{MAIN} does not run on push to main — no main lane to classify") -counts = {"required": 0, "worker": 0, "pr-only": 0} -examined = 0 -print(f"{MAIN} main lane:") -for jid, job in main_jobs.items(): - examined += 1 - ns = NAMES[(MAIN, jid)] - if if_str(job) == PR_ONLY: - counts["pr-only"] += 1 - print(f" pr-only {jid}") - continue - if not ns: - continue # already [unexpandable] - inreq = [n in REQ_SET for n in ns] - if all(inreq): - counts["required"] += 1 - print(f" required {jid}: {' | '.join(ns)}") - continue - if any(inreq): - V("partial-matrix", f"{MAIN}:{jid}: legs {[n for n in ns if n in REQ_SET]} are required but {[n for n in ns if n not in REQ_SET]} are not — every leg colours main") - continue - consumers = [k for k, kj in main_jobs.items() if jid in needs(kj)] - tag = f"{MAIN}:{jid} {ns}" - if not consumers: - V("uncovered", f"{tag} runs on the main lane, is not in required-checks.txt, and no aggregator consumes it: it can turn main red without ever having blocked a merge — require it, make it a worker of a required aggregator, or move it to {NIGHTLY}") - continue - if len(consumers) > 1: - V("multi-consumer", f"{tag} is consumed by {consumers}; a worker is covered only when its ONE consumer is a required aggregator") - continue - agg = consumers[0] - an = NAMES.get((MAIN, agg), []) - bad = False - if not an or not all(x in REQ_SET for x in an): - V("uncovered", f"{tag}: its only consumer {agg} {an} is not required"); bad = True - elif not is_always(main_jobs[agg]): - V("agg-not-always", f"{tag}: aggregator {agg} lacks `if: always()` — when the worker fails the aggregator is SKIPPED, and a skipped required check passes"); bad = True - elif not result_checked(main_jobs[agg], jid): - V("agg-unchecked", f"{tag}: aggregator {agg} has no unconditional step comparing needs.{jid}.result to success — a failed/cancelled/skipped worker would not fail it"); bad = True - if job.get("continue-on-error") not in (None, False): - V("worker-continue-on-error", f"{tag} sets continue-on-error: its result reads success when it failed, so {agg} cannot see the failure"); bad = True - if not bad: - counts["worker"] += 1 - print(f" worker {jid}: {' | '.join(ns)} -> {agg}") -if examined != len(main_jobs) or examined == 0: - V("vacuous", f"examined {examined} of {len(main_jobs)} {MAIN} jobs") - -# ---- (4): nightly ------------------------------------------------------------- -n_jobs = WFS.get(NIGHTLY, {}).get("jobs", {}) -if not n_jobs: - V("nightly-unreported", f"{NIGHTLY} is missing or has no jobs — tier 2 has nowhere to live") -reporters = [j for j, jb in n_jobs.items() if is_always(jb)] -if n_jobs and not reporters: - V("nightly-unreported", f"{NIGHTLY} has no `if: always()` reporter job") -for rj in reporters: - # Code lines only: the reporter's own comments mention `gh issue` (§24). - runs = [ln for s in (n_jobs[rj].get("steps") or []) if isinstance(s, dict) - for ln in str(s.get("run") or "").split("\n") if not ln.lstrip().startswith("#")] - if not any("gh issue" in ln for ln in runs): - V("nightly-unreported", f"{NIGHTLY}:{rj} is the reporter but files no issue (`gh issue` never appears in its steps)") -tier2 = [] -for j in n_jobs: - if j in reporters: continue - tier2.append(j) - owners = [rj for rj in reporters if j in needs(n_jobs[rj])] - if not owners: - V("nightly-unreported", f"{NIGHTLY}:{j} is consumed by no reporter — a failure there is read by no one") - elif not any(result_checked(n_jobs[rj], j) for rj in owners): - V("nightly-unreported", f"{NIGHTLY}:{j}: its reporter never compares needs.{j}.result to success") - -# ---- other workflows: informational --------------------------------------------- -print("other workflows (tier by required-checks.txt; they do not colour the ci.yml badge):") -for (wf, jid), ns in sorted(NAMES.items()): - if wf in (MAIN, NIGHTLY): continue - for n in ns: - print(f" {'required' if n in REQ_SET else 'advisory'} {wf}: {n}") - -per = " ".join(f"{k}={v}" for k, v in sorted(req_by_wf.items())) -if viol: - print(f"ci-tier: FAIL — {len(viol)} violation(s): {' '.join(sorted(set(viol)))}") - sys.exit(1) -print(f"ci-tier: OK — {MAIN}: jobs={examined} (awk={AWK_JOBS}) required={counts['required']} worker={counts['worker']} pr-only={counts['pr-only']}; " - f"required-checks.txt: {len(required)} names (grep={REQ_LINES}), each produced once and reporting on pull_request [{per}]; " - f"{NIGHTLY}: {len(tier2)} tier-2 jobs, each reported ({', '.join(reporters)})") +NAMES = {(wf, jid): names(jid, j) for wf, (_, js) in W.items() for jid, j in js.items()} +PROD = {} +for k, ns in NAMES.items(): + for n in ns: PROD.setdefault(n, []).append(k) + +def closure(wf, jid, seen): + if (wf, jid) in seen or jid not in W[wf][1]: return + seen.add((wf, jid)) + for n in needs(W[wf][1][jid]): closure(wf, n, seen) + +# (a) every required name: one producer, on pull_request AND merge_group +path = set() +for r in req: + p = PROD.get(r, []) + if len(p) != 1: + V("unproduced" if not p else "ambiguous", f"required {r!r} has {len(p)} producing jobs {p} (0 never reports and blocks every merge)"); continue + wf, jid = p[0] + if not triggers(W[wf][0], "pull_request"): V("not-on-pr", f"required {r!r}: {wf} does not run on every pull_request to main") + if not triggers(W[wf][0], "merge_group"): V("not-in-queue", f"required {r!r}: {wf} does not trigger on merge_group — it never reports in the queue") + closure(wf, jid, path) + +# (d) every ci.yml job is required or a worker of one required always() job +ci = W["ci.yml"][1] +if len(ci) != int(os.environ["CT_AWK"]) or not ci: + V("vacuous", f"ci.yml: the loader sees {len(ci)} jobs, awk sees {os.environ['CT_AWK']}") +counts = {"required": 0, "worker": 0} +for jid, j in ci.items(): + if all(n in REQ for n in NAMES[("ci.yml", jid)]): counts["required"] += 1; continue + cons = [k for k, kj in ci.items() if jid in needs(kj)] + if len(cons) == 1 and always(ci[cons[0]]) and all(n in REQ for n in NAMES[("ci.yml", cons[0])]): + counts["worker"] += 1; closure("ci.yml", jid, path); continue + V("uncovered", f"ci.yml:{jid} {NAMES[('ci.yml', jid)]} is not required and not the worker of ONE required `if: always()` job — it can colour main without blocking the queue") + +# (b) + (c) on every job of a required path +for wf, jid in sorted(path): + j = W[wf][1][jid] + if github_refs(cond(j.get("if", "")), False): + V("event-condition", f"{wf}:{jid}: job-level `if: {cond(j['if'])}` mentions the event — it may skip in the queue or on a PR") + if j.get("continue-on-error") not in (None, False): + V("continue-on-error", f"{wf}:{jid} sets continue-on-error: its failure would not fail the check") + for i, st in enumerate(j.get("steps") or []): + if not isinstance(st, dict): continue + if github_refs(cond(st.get("if", "")), True): + V("event-condition", f"{wf}:{jid} step {i} ({st.get('name', st.get('uses', '?'))}): `if: {cond(st['if'])}` — only `github.event_name ==/!= 'pull_request'` may select the lane (never `== 'push'`)") + if st.get("continue-on-error") not in (None, False): + V("continue-on-error", f"{wf}:{jid} step {i} ({st.get('name', '?')}) sets continue-on-error") + +if bad: print(f"ci-tier: FAIL — {len(bad)} violation(s): {' '.join(sorted(set(bad)))}"); sys.exit(1) +print(f"ci-tier: OK — ci.yml jobs={len(ci)} (awk={os.environ['CT_AWK']}) required={counts['required']} worker={counts['worker']}; " + f"{len(req)} required names, each produced once on pull_request+merge_group; {len(path)} jobs on required paths, no push-only condition, no continue-on-error") PY } live() { - command -v gh >/dev/null 2>&1 || { echo "ci-tier --live: INSTRUMENT ERROR — no gh CLI"; return 2; } - local got - got=$(gh api "repos/$REPO/rulesets/$RULESET_ID" \ - --jq '.rules[]|select(.type=="required_status_checks")|.parameters.required_status_checks[].context' 2>&1) \ - || { echo "ci-tier --live: INSTRUMENT ERROR — could not read ruleset $RULESET_ID: $got"; return 2; } - [ -n "$got" ] || { echo "ci-tier --live: INSTRUMENT ERROR — ruleset $RULESET_ID lists no required checks"; return 2; } - local want + local got want + got=$(gh api repos/InauguralSystems/EigenScript/rulesets/17713865 --jq \ + '.rules[]|select(.type=="required_status_checks")|.parameters.required_status_checks[].context' 2>&1) && [ -n "$got" ] \ + || { echo "ci-tier --live: INSTRUMENT ERROR — cannot read the ruleset: $got"; return 2; } want=$(grep -vE '^(#|$)' "$REQ_FILE") - local only_file only_live - only_file=$(comm -23 <(printf '%s\n' "$want" | LC_ALL=C sort) <(printf '%s\n' "$got" | LC_ALL=C sort)) - only_live=$(comm -13 <(printf '%s\n' "$want" | LC_ALL=C sort) <(printf '%s\n' "$got" | LC_ALL=C sort)) - local nf nl - nf=$(printf '%s\n' "$want" | grep -c .); nl=$(printf '%s\n' "$got" | grep -c .) - if [ -z "$only_file" ] && [ -z "$only_live" ]; then - echo "ci-tier --live: OK — ruleset $RULESET_ID requires exactly the $nf checks in required-checks.txt" - return 0 - fi - [ -n "$only_file" ] && printf ' in required-checks.txt, NOT in the ruleset: %s\n' "$only_file" | sed '2,$s/^/ in required-checks.txt, NOT in the ruleset: /' - [ -n "$only_live" ] && printf ' in the ruleset, NOT in required-checks.txt: %s\n' "$only_live" | sed '2,$s/^/ in the ruleset, NOT in required-checks.txt: /' - echo "ci-tier --live: DRIFT — file=$nf ruleset=$nl; sync the ruleset from the file" - return 1 + local d; d=$(diff <(LC_ALL=C sort <<<"$want") <(LC_ALL=C sort <<<"$got") | sed -n 's/^< / file only: /p; s/^> / ruleset only: /p') + [ -z "$d" ] && { echo "ci-tier --live: OK — the ruleset requires exactly the $(grep -c . <<<"$want") checks in the file"; return 0; } + echo "$d"; echo "ci-tier --live: DRIFT — sync the ruleset from required-checks.txt"; return 1 } selftest() { - local tmp pass=0 fail=0 broken=0 - tmp=$(mktemp -d "${TMPDIR:-/tmp}/ci_tier_selftest.XXXXXX") || { echo "selftest: cannot create a temp dir"; return 2; } - trap 'rm -rf "$tmp"' RETURN - echo "ci_tier_check selftest (every fault is planted in a copy of .github/)" - - fresh() { - rm -rf "$tmp/gh"; mkdir -p "$tmp/gh" - cp -R "$ROOT/.github/workflows" "$tmp/gh/workflows" - cp "$ROOT/.github/required-checks.txt" "$tmp/gh/required-checks.txt" + local t pass=0 fail=0; t=$(mktemp -d "${TMPDIR:-/tmp}/ci_tier.XXXXXX") || return 2 + trap 'rm -rf "$t"' RETURN + fresh() { rm -rf "$t/w"; cp -R "$ROOT/.github/workflows" "$t/w"; cp "$ROOT/.github/required-checks.txt" "$t/req"; } + sub() { python3 -c 'import sys; p,o,n=sys.argv[1:]; s=open(p).read(); assert s.count(o)==1 and o!=n, o; open(p,"w").write(s.replace(o,n))' "$@"; } + expect() { # expect DESC CODE — the plant must be live (sub succeeded) and red via CODE + local rc=0; CI_TIER_WF_DIR="$t/w" CI_TIER_REQUIRED="$t/req" bash "$SELF" > "$t/out" 2>&1 || rc=$? + if { [ "$2" = OK ] && [ $rc -eq 0 ]; } || { [ $rc -eq 1 ] && grep -qF "FAIL [$2]" "$t/out"; }; then + pass=$((pass + 1)); echo " PASS $1 -> ${2}" + else fail=$((fail + 1)); echo " FAIL $1: expected [$2], rc=$rc"; tail -4 "$t/out"; fi } - # sub FILE OLD NEW — replace exactly one occurrence, or the plant is BROKEN - # (a plant that edits nothing proves nothing: mechanical-gates §20/§137). - sub() { - python3 - "$1" "$2" "$3" <<'PY' -import sys -p, old, new = sys.argv[1], sys.argv[2], sys.argv[3] -s = open(p, encoding="utf-8").read() -n = s.count(old) -if n != 1 or old == new: - print(f"plant anchor found {n} times in {p} (or edits nothing): {old[:60]!r}") - sys.exit(3) -open(p, "w", encoding="utf-8").write(s.replace(old, new)) -PY - } - run_gate() { - CI_TIER_WF_DIR="$tmp/gh/workflows" CI_TIER_REQUIRED="$tmp/gh/required-checks.txt" \ - bash "$SELF" > "$tmp/out" 2>&1 - } - # expect NAME CODE — run the REAL gate on the planted copy; require rc 1 - # and the named check. - expect() { - local name="$1" code="$2" rc=0 - run_gate || rc=$? - if [ "$rc" -eq 1 ] && grep -qF "FAIL [$code]" "$tmp/out"; then - pass=$((pass + 1)); echo " PASS $name -> red via [$code]" - else - fail=$((fail + 1)); echo " FAIL $name: expected rc=1 with [$code], got rc=$rc"; sed 's/^/ | /' "$tmp/out" | tail -8 - fi - } - brk() { broken=$((broken + 1)); echo " BROKEN $1: the plant's anchor no longer exists — update the selftest"; } - - local CI="$tmp/gh/workflows/ci.yml" NY="$tmp/gh/workflows/nightly.yml" REQ="$tmp/gh/required-checks.txt" - - # 0. sanity start: the unmodified copy is green (else every red below is noise). - fresh - local rc=0; run_gate || rc=$? - if [ "$rc" -eq 0 ]; then pass=$((pass + 1)); echo " PASS unmodified copy is green" - else fail=$((fail + 1)); echo " FAIL unmodified copy: rc=$rc"; tail -8 "$tmp/out"; fi - - # 1. a new ci.yml job nobody requires or aggregates (the macos-15-intel shape) - fresh; printf '\n planted-lane:\n name: planted lane\n runs-on: ubuntu-latest\n steps:\n - run: "true"\n' >> "$CI" - expect "new main-lane job, not required, no aggregator" uncovered - - # 2. rename a required job - fresh; sub "$CI" "name: tsan (concurrency race gate)" "name: tsan (renamed)" && expect "required job renamed" unproduced || brk rename - - # 3. a required job made push-only - fresh; sub "$CI" " name: tsan (concurrency race gate) -" " name: tsan (concurrency race gate) - if: github.event_name == 'push' -" && expect "required job made push-only" not-on-pr || brk push-only - - # 4. a required job's PREREQUISITE made push-only (the check would skip on PRs) - fresh; sub "$CI" " scope: - name: scope -" " scope: - name: scope - if: github.ref == 'refs/heads/main' -" && expect "required job's prerequisite made push-only" not-on-pr || brk prereq-push-only - - # 5. another workflow's required check stops triggering on pull_request - fresh; sub "$tmp/gh/workflows/codeql.yml" " pull_request: - branches: [main] -" "" && expect "codeql.yml loses pull_request (Analyze C)" not-on-pr || brk codeql-pr - - # 6. a required workflow gains a path filter (docs-only PRs never report) - fresh; sub "$tmp/gh/workflows/pages.yml" " pull_request: - branches: [main] -" " pull_request: + broken() { fail=$((fail + 1)); echo " FAIL $1: plant anchor missing (BROKEN, not a pass)"; } + local CI="$t/w/ci.yml" TS=" name: tsan (concurrency race gate) +" + fresh; expect "unmodified copy" OK + fresh; sub "$CI" "$TS" " name: tsan renamed +" && expect "(a) required job renamed" unproduced || broken rename + fresh; sub "$t/w/codeql.yml" " merge_group:" " workflow_call:" && expect "(a) codeql.yml loses merge_group" not-in-queue || broken queue + fresh; sub "$t/w/pages.yml" " pull_request: branches: [main] - paths-ignore: ['**.md'] -" && expect "pages.yml path-filtered (playground)" not-on-pr || brk pages-paths - - # 7. an aggregator loses if: always() - fresh; sub "$CI" " needs: [scope, extensions-http, extensions-gfx, extensions-zlib, extensions-net] +" "" && expect "(a) pages.yml loses pull_request" not-on-pr || broken pr + # job-level: even the lane atom is red (a PR would SKIP the job = satisfied) + fresh; sub "$CI" "$TS" "$TS if: github.event_name != 'pull_request' +" && expect "(b) required job skipped on PRs (job-level lane atom)" event-condition || broken job-push + fresh; sub "$CI" " name: Compile-check LSP (macOS) + run: make lsp CC=clang +" " name: Compile-check LSP (macOS) + run: make lsp CC=clang + - if: github.event_name == 'push' + name: planted main-only step + run: exit 1 +" && expect "(b) P1: push-only failing step in a required job" event-condition || broken p1 + fresh; sub "$CI" "if: matrix.cc == 'gcc' || github.event_name != 'pull_request'" \ + "if: matrix.cc == 'gcc' || github['EVENT_NAME'] == 'push'" \ + && expect "(b) bracket syntax, push-only step" event-condition || broken bracket + fresh; sub "$CI" "Require complete sanitizer coverage +" "Require complete sanitizer coverage + continue-on-error: true +" && expect "(c) P2: aggregator step continue-on-error" continue-on-error || broken p2 + fresh; sub "$CI" " needs: [scope, sanitizers-core, sanitizers-http] + if: always() +" " needs: [scope, sanitizers-core, sanitizers-http] if: always() -" " needs: [scope, extensions-http, extensions-gfx, extensions-zlib, extensions-net] -" && expect "extensions aggregator loses if: always()" agg-not-always || brk agg-always - - # 8. an aggregator stops checking one worker's result - fresh; sub "$CI" 'for result in "$HTTP_RESULT" "$GFX_RESULT" "$ZLIB_RESULT" "$NET_RESULT"; do' \ - 'for result in "$HTTP_RESULT" "$ZLIB_RESULT" "$NET_RESULT"; do' \ - && expect "extensions aggregator stops checking gfx" agg-unchecked || brk agg-unchecked - - # 9. a worker gains continue-on-error - fresh; sub "$CI" " name: extensions / zlib suite -" " name: extensions / zlib suite continue-on-error: true -" && expect "worker sets continue-on-error" worker-continue-on-error || brk coe - - # 10. a worker gains a second consumer - fresh; sub "$CI" " name: bench (instruction-count regression gate) - needs: scope -" " name: bench (instruction-count regression gate) - needs: [scope, extensions-net] -" && expect "worker consumed by a second job" multi-consumer || brk multi - - # 11. half a matrix required - fresh; sub "$REQ" "linux / clang -" "" && expect "linux / clang dropped from the required set" partial-matrix || brk partial - - # 12. a required job's prerequisite dropped from the required set - fresh; sub "$REQ" "build dev/ci image -" "" && expect "build dev/ci image dropped" need-not-required || brk need - - # 13. a nightly job no reporter reads - fresh; printf '\n planted-nightly:\n name: nightly / planted\n runs-on: ubuntu-latest\n steps:\n - run: "true"\n' >> "$NY" - expect "nightly job outside the reporter's needs" nightly-unreported - - # 14. the nightly reporter stops filing issues - fresh; sub "$NY" 'gh issue create --repo "$REPO" --title "$TITLE" --body "$BODY" "${LABEL_ARGS[@]}"' 'echo would-create' \ - && sub "$NY" 'gh issue comment "$num" --repo "$REPO" --body "$BODY"' 'echo would-comment' \ - && sub "$NY" 'gh issue comment "$num" --repo "$REPO" \' 'echo \' \ - && sub "$NY" 'gh issue reopen "$num"' 'echo reopen' \ - && sub "$NY" 'num=$(gh issue list' 'num=$(echo' \ - && expect "nightly reporter files no issue" nightly-unreported || brk reporter - - # 15. a duplicated required name - fresh; printf 'scope\n' >> "$REQ"; expect "required name listed twice" dup-required - - # 16. a second producer of a required name - fresh; sub "$tmp/gh/workflows/scorecard.yml" " name: Scorecard analysis" " name: scope" \ - && expect "two jobs produce a required name" ambiguous || brk ambiguous - - # 17. a job the awk count sees but the loader does not agree on (flow style) - fresh; printf '\n planted-flow: {name: planted flow, runs-on: ubuntu-latest, steps: [{run: "true"}]}\n' >> "$CI" - expect "flow-style job the awk count cannot see" count-mismatch - - # 18. a job name the gate cannot expand - fresh; sub "$CI" " name: tsan (concurrency race gate)" ' name: tsan ${{ github.ref }}' \ - && expect "non-matrix expression in a job name" unexpandable || brk unexpandable - - # 19. an empty tier 1 - fresh; grep -E '^#' "$ROOT/.github/required-checks.txt" > "$REQ" - expect "required-checks.txt with no names" vacuous - - # 20. trailing blank on a name - # (the blank is built with printf: editors and hooks strip a literal one) - fresh; sub "$REQ" "tsan (concurrency race gate)" "tsan (concurrency race gate)$(printf ' ')" && expect "trailing blank on a required name" whitespace || brk whitespace - - # 21. no PyYAML -> an INSTRUMENT error (rc 2), never a verdict - fresh; mkdir -p "$tmp/noyaml"; printf 'raise ImportError("planted: no PyYAML")\n' > "$tmp/noyaml/yaml.py" - rc=0; PYTHONPATH="$tmp/noyaml" CI_TIER_WF_DIR="$tmp/gh/workflows" CI_TIER_REQUIRED="$REQ" \ - bash "$SELF" > "$tmp/out" 2>&1 || rc=$? - if [ "$rc" -eq 2 ] && grep -qF 'INSTRUMENT ERROR' "$tmp/out" && ! grep -qF 'ci-tier: OK' "$tmp/out"; then - pass=$((pass + 1)); echo " PASS no PyYAML -> rc 2 INSTRUMENT ERROR" - else fail=$((fail + 1)); echo " FAIL no PyYAML: rc=$rc"; tail -4 "$tmp/out"; fi - - echo "ci_tier_check selftest: checks=$((pass + fail + broken)) failures=$fail broken=$broken" - [ "$fail" -eq 0 ] && [ "$broken" -eq 0 ] +" && expect "(c) P4: required aggregator continue-on-error" continue-on-error || broken p4 + fresh; printf '\n planted:\n runs-on: ubuntu-latest\n steps:\n - run: "true"\n' >> "$CI" + expect "new ci.yml job, neither required nor a worker" uncovered + fresh; mkdir -p "$t/ny"; echo 'raise ImportError("planted")' > "$t/ny/yaml.py" + local rc=0; PYTHONPATH="$t/ny" CI_TIER_WF_DIR="$t/w" CI_TIER_REQUIRED="$t/req" bash "$SELF" > "$t/out" 2>&1 || rc=$? + if [ $rc -eq 2 ]; then pass=$((pass + 1)); echo " PASS no PyYAML -> rc 2 INSTRUMENT ERROR"; else fail=$((fail + 1)); echo " FAIL no PyYAML: rc=$rc"; fi + echo "ci_tier_check selftest: checks=$((pass + fail)) failures=$fail" + [ $fail -eq 0 ] } case "${1:-}" in - "") check ;; - --selftest) selftest ;; - --live) live ;; + "") check ;; --selftest) selftest ;; --live) live ;; *) echo "usage: $0 [--selftest|--live]" >&2; exit 2 ;; esac From 046b2d6b02da466b21bc0031e89208869f0b5063 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Wed, 23 Sep 2026 02:05:52 -0500 Subject: [PATCH 3/6] ci: tier gate follows event-derived values through env/outputs/step outputs (#1264) Round-2 critic: `env: X: ${{ github.event_name == 'push' }}` + a step `if: env.X == 'true'` on required tsan skipped in merge_group and ran on push, and the gate was green. - On a required path, env / job outputs / workflow env / matrix values may not derive from the event (github.event_name/event/ref*/head_ref/base_ref, dot or bracket, bare `github`), except the PR-shaped forms. - An `if:` reading env.*, vars.*, needs.*.outputs or steps.*.outputs is resolved recursively; unresolvable is red. Step outputs pass only through WAIVE, pinned to a sha256 of the reviewed step (scope's `detect`, the [99i] cache `restore`); a stale waiver is red. - ci.yml dev-image reads $GITHUB_EVENT_NAME instead of an event-derived env. - Selftest +1 plant: the critic's tsan env alias, required to red through both the `if:` resolution and the value check. - tools/ci_tier_check.sh 229 -> 260 lines (budget 260). Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/ci.yml | 6 +- tools/ci_tier_check.sh | 140 ++++++++++++++++++++++++--------------- 2 files changed, 89 insertions(+), 57 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3a659402..9368b961 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -163,13 +163,15 @@ jobs: env: OWNER: ${{ github.repository_owner }} IS_FORK: ${{ github.event.pull_request.head.repo.fork == true }} - EVENT: ${{ github.event_name }} + # The event is read from the runner's GITHUB_EVENT_NAME, not an + # expression: tools/ci_tier_check.sh reds event-derived expression + # values on a required path (#1264). Only push advances ci-main. run: | BASE="ghcr.io/${OWNER,,}/eigenscript-dev" if [ "$IS_FORK" = "true" ]; then echo "image=$BASE:ci-main" >> "$GITHUB_OUTPUT" echo "tags=" >> "$GITHUB_OUTPUT" - elif [ "$EVENT" = "push" ]; then + elif [ "$GITHUB_EVENT_NAME" = "push" ]; then echo "image=$BASE:ci-${{ github.sha }}" >> "$GITHUB_OUTPUT" echo "tags=$BASE:ci-${{ github.sha }},$BASE:ci-main" >> "$GITHUB_OUTPUT" else diff --git a/tools/ci_tier_check.sh b/tools/ci_tier_check.sh index dca76bd2..cb6210d2 100755 --- a/tools/ci_tier_check.sh +++ b/tools/ci_tier_check.sh @@ -3,40 +3,25 @@ # cannot guarantee by itself. # # Tier 1 = the names in .github/required-checks.txt (the ruleset is synced from -# that file). The merge queue runs the full main lane on `merge_group` and -# lands nothing that is not green there, so main is green by construction — -# PROVIDED the required checks report in the queue and test what push tests. -# That proviso is what this gate pins: -# [unproduced]/[ambiguous] every required name is produced by exactly one -# job in .github/workflows/; -# [not-on-pr]/[not-in-queue] its workflow triggers on pull_request (for main, -# no path filter) AND on merge_group; -# [event-condition] on a REQUIRED PATH (a required job, the jobs it -# transitively needs, and ci.yml workers) no -# condition can run work on push but not in the -# queue. Job-level `if:` may not mention the event -# at all; a step `if:` may mention it only as -# `github.event_name ==/!= 'pull_request'` (same -# truth on push and merge_group) or through the -# PR payload `github.event.pull_request.*` (empty on -# both). Dot and bracket syntax, any case; -# [continue-on-error] no job or step on a required path sets it — a -# failure there would not fail the check; -# [uncovered] every ci.yml job is required, or a worker whose -# ONE consumer is a required `if: always()` job (a -# non-required job that fails does not block the -# queue, yet colours main's badge). -# Whether an aggregator's script really fails on a bad worker result is a -# code-review question, not this gate's. Populations: the loader's ci.yml job -# count == an awk count > 0; required names == a grep count > 0. -# -# Usage: tools/ci_tier_check.sh [--selftest | --live] -# (no flag) exit 0 OK, 1 violation, 2 instrument error (no PyYAML, a file -# that does not load) — never a verdict -# --selftest plants each class in a copy of .github/ and requires the named -# check to go red -# --live read-only diff of required-checks.txt against the live ruleset -# (not run in CI: the ruleset is synced after merge) +# it). The merge queue runs the full main lane on merge_group and lands nothing +# that is not green there — PROVIDED the required checks report in the queue +# and test what push tests. This gate pins that proviso: +# [unproduced]/[ambiguous] each required name has exactly one producing job; +# [not-on-pr]/[not-in-queue] on pull_request (main, unfiltered) AND merge_group; +# [event-condition] on a REQUIRED PATH (required jobs, ci.yml workers, their +# needs-closure) nothing may differ between push and merge_group: a job +# `if:` may not read the event at all; a step `if:` only as +# `github.event_name ==/!= 'pull_request'` or the PR payload +# `github.event.pull_request.*` (dot or bracket syntax, any case); env / +# outputs / matrix values may not derive from the event; an `if:` that +# reads env.*, vars.*, needs.*.outputs or steps.*.outputs must resolve to +# such values (unresolvable = red; step outputs only via a pinned WAIVE); +# [continue-on-error] set on no job or step of a required path; +# [uncovered] every ci.yml job is required or the worker of ONE required +# `if: always()` job (else it colours the badge, blocking nothing). +# Aggregator scripts are code review's. Populations: loader ci.yml jobs == awk +# count > 0; names == grep count > 0. [--selftest|--live]; exit 1 violation, 2 +# instrument error. --live: read-only diff vs the ruleset (not in CI: synced post-merge). set -u ROOT="$(cd "$(dirname "$0")/.." && pwd)" WF_DIR="${CI_TIER_WF_DIR:-$ROOT/.github/workflows}" @@ -49,7 +34,7 @@ check() { CT_AWK=$(awk '/^jobs:/ {j=1; next} j && /^[^ #]/ {j=0} j && /^ [A-Za-z0-9_-]+:[[:space:]]*(#.*)?$/ {n++} END {print n+0}' "$WF_DIR/ci.yml") \ CT_GREP=$(grep -cvE '^(#|$)' "$REQ_FILE") CT_WF="$WF_DIR" CT_REQ="$REQ_FILE" python3 - <<'PY' -import itertools, os, re, sys +import hashlib, itertools, os, re, sys from fnmatch import fnmatchcase def instrument(m): print(f"ci-tier: INSTRUMENT ERROR — {m}; nothing was checked"); sys.exit(2) try: @@ -73,7 +58,7 @@ for fn in sorted(os.listdir(os.environ["CT_WF"])): on = doc.get("on", doc.get(True)) # YAML 1.1 reads bare `on` as True on = {on: {}} if isinstance(on, str) else {str(k): {} for k in on} if isinstance(on, list) else \ {str(k): (v if isinstance(v, dict) else {}) for k, v in (on or {}).items()} - W[fn] = (on, doc["jobs"]) + W[fn] = (on, doc["jobs"], doc.get("env") or {}) def triggers(on, ev): if ev not in on: return False @@ -90,9 +75,42 @@ def always(j): return cond(j.get("if", "")) == "always()" G = r"github\s*(?:\.\s*{0}\b|\[\s*['\"]{0}['\"]\s*\])" PR_ATOM = re.compile(G.format("event_name") + r"\s*[=!]=\s*['\"]pull_request['\"]|['\"]pull_request['\"]\s*[=!]=\s*" + G.format("event_name"), re.I) PR_BODY = re.compile(G.format("event") + r"\s*(?:\.\s*pull_request\b|\[\s*['\"]pull_request['\"]\s*\])", re.I) -def github_refs(s, step): - if step: s = PR_BODY.sub("_", PR_ATOM.sub("_", s)) - return re.search(r"\bgithub\b", s, re.I) +EVP = r"(?:event_name|event|ref|ref_name|ref_type|head_ref|base_ref)\b" +EV = re.compile(r"\bgithub\b(?!\s*(?:\.\s*(?!" + EVP + r")\w|\[\s*['\"](?!" + EVP + r")\w+['\"]\s*\]))", re.I) +IND = re.compile(r"\b(env|vars)\s*(?:\.\s*|\[\s*['\"])([\w-]+)['\"]?\s*\]?|\b(needs|steps)\s*(?:\.\s*|\[\s*['\"])([\w-]+)['\"]?\s*\]?" + r"\s*(?:\.\s*|\[\s*['\"])(outputs|result|outcome|conclusion)['\"]?\s*\]?(?:\s*(?:\.\s*|\[\s*['\"])([\w-]+)['\"]?\s*\]?)?", re.I) +# Step outputs an `if:` reads, reviewed as the same on push and merge_group: +# scope's `code` (true on every non-PR event) and the [99i] cache restore (keyed +# on audit inputs only). Pinned to the reviewed STEP — any edit is red until +# re-reviewed and re-pinned (sha256 of yaml.safe_dump(step, sort_keys=True)). +WAIVE = {("ci.yml", "scope", "detect"): "6611d97dc6f0b4e7", ("ci.yml", "werror-audit", "restore"): "fd12b71bc65e9a90"} +used = set() +def exprs(v): return re.findall(r"\$\{\{(.*?)\}\}", str(v), re.S) +def bad_expr(e, wf, jid, pr_ok, depth=0): + """Why expression `e` (in wf:jid) may differ between push and merge_group, or None.""" + t = PR_BODY.sub("_", PR_ATOM.sub("_", e)) if pr_ok else e + if EV.search(t): return f"`{e.strip()}` reads the event" + if re.search(r"\b(env|vars|steps|needs)\b", IND.sub("_", t), re.I): return f"`{e.strip()}` reads a context this gate cannot resolve" + for m in IND.finditer(t): + (ctx, name, kind, job, field, key), j = m.groups(), W[wf][1].get(jid, {}) + if ctx == "vars" or depth > 4: return f"`{m.group(0)}` cannot be resolved" + if ctx == "env": + vals = [str(d[name]) for d in [s.get("env") for s in j.get("steps") or [] if isinstance(s, dict)] + [j.get("env"), W[wf][2]] + if isinstance(d, dict) and name in d] + if not vals: return f"`env.{name}` is not declared in {wf}:{jid}" + why = next((w for v in vals for x in exprs(v) for w in [bad_expr(x, wf, jid, pr_ok, depth + 1)] if w), None) + elif field in ("result", "outcome", "conclusion"): continue + elif kind == "needs": + v = (W[wf][1].get(job, {}).get("outputs") or {}).get(key) + if v is None: return f"`needs.{job}.outputs.{key}` is not declared" + why = next((w for x in exprs(v) for w in [bad_expr(x, wf, job, pr_ok, depth + 1)] if w), None) + else: + src = next((yaml.safe_dump(s, sort_keys=True) for s in j.get("steps") or [] if isinstance(s, dict) and s.get("id") == job), None) + pin = WAIVE.get((wf, jid, job)) + if pin and src is not None and hashlib.sha256(src.encode()).hexdigest()[:16] == pin: used.add((wf, jid, job)); continue + return f"`steps.{job}.outputs.{key}` comes from a script (not waived, or the waived script changed)" + if why: return f"{m.group(0).strip()} -> {why}" + return None def names(jid, j): m = (j.get("strategy") or {}).get("matrix") if isinstance(j.get("strategy"), dict) else None @@ -106,7 +124,7 @@ def names(jid, j): n = re.sub(r"\$\{\{\s*matrix\.([\w-]+)\s*\}\}", lambda mo: str(c.get(mo.group(1), "?")), str(j.get("name", jid))) if n not in out: out.append(n) return out -NAMES = {(wf, jid): names(jid, j) for wf, (_, js) in W.items() for jid, j in js.items()} +NAMES = {(wf, jid): names(jid, j) for wf, (_, js, _e) in W.items() for jid, j in js.items()} PROD = {} for k, ns in NAMES.items(): for n in ns: PROD.setdefault(n, []).append(k) @@ -142,17 +160,24 @@ for jid, j in ci.items(): # (b) + (c) on every job of a required path for wf, jid in sorted(path): j = W[wf][1][jid] - if github_refs(cond(j.get("if", "")), False): - V("event-condition", f"{wf}:{jid}: job-level `if: {cond(j['if'])}` mentions the event — it may skip in the queue or on a PR") - if j.get("continue-on-error") not in (None, False): - V("continue-on-error", f"{wf}:{jid} sets continue-on-error: its failure would not fail the check") + why = bad_expr(cond(j.get("if", "")), wf, jid, False) + if why: V("event-condition", f"{wf}:{jid}: job-level `if:` — {why}; a job skipped in the queue or on a PR is a satisfied check") + # values an `if:` may read through indirection: event-free or PR-shaped only + vals = [(k, v) for d in (j.get("env"), j.get("outputs"), W[wf][2]) if isinstance(d, dict) for k, v in d.items()] + vals += [("strategy", yaml.safe_dump(j.get("strategy") or {}))] + [(k, v) for st in j.get("steps") or [] + if isinstance(st, dict) for k, v in (st.get("env") or {}).items()] + for k, v in vals: + for x in exprs(v): + if EV.search(PR_BODY.sub("_", PR_ATOM.sub("_", x))): + V("event-condition", f"{wf}:{jid}: `{k}: ${{{{{x}}}}}` derives a value from the event") + if j.get("continue-on-error") not in (None, False): V("continue-on-error", f"{wf}:{jid} sets continue-on-error: its failure would not fail the check") for i, st in enumerate(j.get("steps") or []): if not isinstance(st, dict): continue - if github_refs(cond(st.get("if", "")), True): - V("event-condition", f"{wf}:{jid} step {i} ({st.get('name', st.get('uses', '?'))}): `if: {cond(st['if'])}` — only `github.event_name ==/!= 'pull_request'` may select the lane (never `== 'push'`)") - if st.get("continue-on-error") not in (None, False): - V("continue-on-error", f"{wf}:{jid} step {i} ({st.get('name', '?')}) sets continue-on-error") + why = bad_expr(cond(st.get("if", "")), wf, jid, True) + if why: V("event-condition", f"{wf}:{jid} step {i} ({st.get('name', st.get('uses', '?'))}): `if:` — {why}; only `github.event_name ==/!= 'pull_request'` may select the lane") + if st.get("continue-on-error") not in (None, False): V("continue-on-error", f"{wf}:{jid} step {i} ({st.get('name', '?')}) sets continue-on-error") +if set(WAIVE) - used: V("event-condition", f"waiver(s) {sorted(set(WAIVE) - used)} matched nothing — stale; remove or re-pin") if bad: print(f"ci-tier: FAIL — {len(bad)} violation(s): {' '.join(sorted(set(bad)))}"); sys.exit(1) print(f"ci-tier: OK — ci.yml jobs={len(ci)} (awk={os.environ['CT_AWK']}) required={counts['required']} worker={counts['worker']}; " f"{len(req)} required names, each produced once on pull_request+merge_group; {len(path)} jobs on required paths, no push-only condition, no continue-on-error") @@ -175,9 +200,9 @@ selftest() { trap 'rm -rf "$t"' RETURN fresh() { rm -rf "$t/w"; cp -R "$ROOT/.github/workflows" "$t/w"; cp "$ROOT/.github/required-checks.txt" "$t/req"; } sub() { python3 -c 'import sys; p,o,n=sys.argv[1:]; s=open(p).read(); assert s.count(o)==1 and o!=n, o; open(p,"w").write(s.replace(o,n))' "$@"; } - expect() { # expect DESC CODE — the plant must be live (sub succeeded) and red via CODE + expect() { # expect DESC CODE [LIT [LIT]] — the plant is live (sub succeeded), red via CODE (and LITERAL) local rc=0; CI_TIER_WF_DIR="$t/w" CI_TIER_REQUIRED="$t/req" bash "$SELF" > "$t/out" 2>&1 || rc=$? - if { [ "$2" = OK ] && [ $rc -eq 0 ]; } || { [ $rc -eq 1 ] && grep -qF "FAIL [$2]" "$t/out"; }; then + if { [ "$2" = OK ] && [ $rc -eq 0 ]; } || { [ $rc -eq 1 ] && grep -qF "FAIL [$2]" "$t/out" && grep -qF -- "${3:-FAIL}" "$t/out" && grep -qF -- "${4:-FAIL}" "$t/out"; }; then pass=$((pass + 1)); echo " PASS $1 -> ${2}" else fail=$((fail + 1)); echo " FAIL $1: expected [$2], rc=$rc"; tail -4 "$t/out"; fi } @@ -205,6 +230,14 @@ selftest() { fresh; sub "$CI" "if: matrix.cc == 'gcc' || github.event_name != 'pull_request'" \ "if: matrix.cc == 'gcc' || github['EVENT_NAME'] == 'push'" \ && expect "(b) bracket syntax, push-only step" event-condition || broken bracket + # round-2 critic: a push-only value reached through env (the `if:` itself must red) + fresh; sub "$CI" "$TS" "$TS env: + RUN_MAIN_CHECK: \${{ github.event_name == 'push' }} +" && sub "$CI" " run: make tsan +" " run: make tsan + - if: env.RUN_MAIN_CHECK == 'true' + run: exit 1 +" && expect "(b) env alias of a push-only value" event-condition "env.RUN_MAIN_CHECK -> " "derives a value from the event" || broken alias fresh; sub "$CI" "Require complete sanitizer coverage " "Require complete sanitizer coverage continue-on-error: true @@ -219,12 +252,9 @@ selftest() { expect "new ci.yml job, neither required nor a worker" uncovered fresh; mkdir -p "$t/ny"; echo 'raise ImportError("planted")' > "$t/ny/yaml.py" local rc=0; PYTHONPATH="$t/ny" CI_TIER_WF_DIR="$t/w" CI_TIER_REQUIRED="$t/req" bash "$SELF" > "$t/out" 2>&1 || rc=$? - if [ $rc -eq 2 ]; then pass=$((pass + 1)); echo " PASS no PyYAML -> rc 2 INSTRUMENT ERROR"; else fail=$((fail + 1)); echo " FAIL no PyYAML: rc=$rc"; fi + [ $rc -eq 2 ] && { pass=$((pass + 1)); echo " PASS no PyYAML -> rc 2"; } || { fail=$((fail + 1)); echo " FAIL no PyYAML: rc=$rc"; } echo "ci_tier_check selftest: checks=$((pass + fail)) failures=$fail" [ $fail -eq 0 ] } -case "${1:-}" in - "") check ;; --selftest) selftest ;; --live) live ;; - *) echo "usage: $0 [--selftest|--live]" >&2; exit 2 ;; -esac +case "${1:-}" in "") check ;; --selftest) selftest ;; --live) live ;; *) echo "usage: $0 [--selftest|--live]" >&2; exit 2 ;; esac From 0435dbd6c6e9bb53e68e4d8c43fe8a6a9b32d76e Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Wed, 23 Sep 2026 02:07:19 -0500 Subject: [PATCH 4/6] docs(CI): describe the indirection rule and waivers; point accepted limits at #1278 Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CI.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/CI.md b/docs/CI.md index 3602f4fd..582812a0 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -1092,7 +1092,14 @@ through its named check). A missing PyYAML is exit 2, never a pass. job-level event filter lets a merge through untested. A step `if:` may mention the event only as `github.event_name ==/!= 'pull_request'`, or via the PR payload `github.event.pull_request.*` (empty on push and in the - queue alike). Dot and bracket syntax are both read. + queue alike). Dot and bracket syntax are both read. The same rule covers + indirection: an `env`, job `outputs`, workflow `env` or matrix value on a + required path may not read the event (outside those two forms), and an + `if:` that reads `env.*`, `needs.*.outputs` or `steps.*.outputs` is traced + to where the value is set — unresolvable is red, `vars.*` is always red. + Two reviewed step outputs are waived by a hash of their step (`scope`'s + docs-only check, the `werror audit` cache restore); a waiver that matches + nothing is red. - `[continue-on-error]` — a job or step on a required path sets it, so its failure would not fail the check. - `[uncovered]` — a `ci.yml` job is neither required nor the worker of exactly @@ -1100,7 +1107,9 @@ through its named check). A missing PyYAML is exit 2, never a pass. stop the queue, yet it colours the badge: the `macos-15-intel` shape. Whether an aggregator's script really fails on every non-success worker -result is a code-review question, not this gate's. +result is a code-review question, not this gate's. The gate's other accepted +limits (matrix `include`/`exclude`, expressions inside `run:` scripts, +`schedule:` triggers) are listed in #1278. ### Every `ci.yml` job, classified From 8b7e0bf1426ba2930c1617248b566d0cab2bd305 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Wed, 23 Sep 2026 02:11:34 -0500 Subject: [PATCH 5/6] ci(bench): in the merge queue, compare against the candidate's base, not origin/main A queue candidate is main + the PRs queued ahead + this PR. Comparing its Ir against origin/main charged each PR for the cost of the PRs ahead of it: A +3% then B +3% read as B +6% > 5% and falsely ejected B (/code-review on #1273). merge_group now uses merge_group.base_sha, read from the event payload inside the script (the tier gate forbids event values via ${{ }}). Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/ci.yml | 21 +++++++++++++++++---- docs/CI.md | 2 +- 2 files changed, 18 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9368b961..42839a76 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1577,10 +1577,23 @@ jobs: run: ./build.sh - if: needs.scope.outputs.code == 'true' - name: Build origin/main in a worktree (same environment) + name: Build the baseline in a worktree (same environment) + # In the merge queue the candidate is main + the PRs queued AHEAD + this + # one, so the baseline is the candidate's base (merge_group.base_sha), + # not origin/main — otherwise a PR is charged for the Ir cost of the PRs + # ahead of it and falsely rejected (#1273 code review). Read from the + # event payload inside the script, never via ${{ }} (tier gate rule). run: | - git fetch --no-tags --depth=1 origin main - git worktree add /tmp/main-ref origin/main + base=origin/main + if [ "$GITHUB_EVENT_NAME" = merge_group ]; then + base=$(jq -r '.merge_group.base_sha' "$GITHUB_EVENT_PATH") + [ -n "$base" ] && [ "$base" != null ] || { echo "merge_group event without base_sha"; exit 1; } + git fetch --no-tags --depth=1 origin "$base" + else + git fetch --no-tags --depth=1 origin main + fi + echo "baseline: $base" + git worktree add /tmp/main-ref "$base" ( cd /tmp/main-ref && ./build.sh ) - if: needs.scope.outputs.code == 'true' @@ -1588,5 +1601,5 @@ jobs: run: EIGENSCRIPT="$PWD/src/eigenscript" bash bench/check_regression.sh --selftest - if: needs.scope.outputs.code == 'true' - name: Regression gate — Ir of this commit vs origin/main + name: Regression gate — Ir of this commit vs its baseline run: EIGENSCRIPT="$PWD/src/eigenscript" bash bench/check_regression.sh --vs /tmp/main-ref/src/eigenscript diff --git a/docs/CI.md b/docs/CI.md index 582812a0..fdd15413 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -1132,7 +1132,7 @@ limits (matrix `include`/`exclude`, expressions inside `run:` scripts, | `valgrind (memcheck smoke, JIT off)` | 1 | the smoke spread (the full corpus is tier 2) | | `tsan (concurrency race gate)` | 1 | gate | | `install.sh (interpreter + eigenlsp on PATH)` | 1 | gate | -| `bench (instruction-count regression gate)` | 1 | gate (compares against `origin/main`, in the queue too) | +| `bench (instruction-count regression gate)` | 1 | gate (baseline: `origin/main` on a PR; the candidate's `merge_group.base_sha` in the queue, so a PR is never charged for the PRs queued ahead of it) | | `nightly / macos-15-intel full suite` | **2** (`nightly.yml`) | port lane, slow: hit its timeout on nearly every main push (#1265) | | `nightly / valgrind (full corpus, JIT off)` | **2** (`nightly.yml`) | slow; the PR and main lanes run the smoke spread | From 0ddd5709ea34d2d17be3c828f7b2705bba9c6930 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Wed, 23 Sep 2026 02:15:33 -0500 Subject: [PATCH 6/6] ci_tier_check: re-anchor two selftest plants off compiler-named lines The werror audit ([99i]) reads any tools/ line naming a compiler as a possible compile invocation; the plants anchored on 'make lsp CC=clang' and 'matrix.cc == gcc' tripped it (GATE ERROR x4 on 0435dbd). Both now anchor on the macOS main-lane suite step. Selftest 12/12; gate size 262 lines. Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/ci_tier_check.sh | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/tools/ci_tier_check.sh b/tools/ci_tier_check.sh index cb6210d2..7cd9fa2e 100755 --- a/tools/ci_tier_check.sh +++ b/tools/ci_tier_check.sh @@ -219,16 +219,18 @@ selftest() { # job-level: even the lane atom is red (a PR would SKIP the job = satisfied) fresh; sub "$CI" "$TS" "$TS if: github.event_name != 'pull_request' " && expect "(b) required job skipped on PRs (job-level lane atom)" event-condition || broken job-push - fresh; sub "$CI" " name: Compile-check LSP (macOS) - run: make lsp CC=clang -" " name: Compile-check LSP (macOS) - run: make lsp CC=clang + fresh; sub "$CI" " name: Run test suite (main lane, [99i] included) + run: cd tests && bash run_all_tests.sh +" " name: Run test suite (main lane, [99i] included) + run: cd tests && bash run_all_tests.sh - if: github.event_name == 'push' name: planted main-only step run: exit 1 " && expect "(b) P1: push-only failing step in a required job" event-condition || broken p1 - fresh; sub "$CI" "if: matrix.cc == 'gcc' || github.event_name != 'pull_request'" \ - "if: matrix.cc == 'gcc' || github['EVENT_NAME'] == 'push'" \ + fresh; sub "$CI" "&& github.event_name != 'pull_request' + name: Run test suite (main lane, [99i] included)" \ + "&& github['EVENT_NAME'] == 'push' + name: Run test suite (main lane, [99i] included)" \ && expect "(b) bracket syntax, push-only step" event-condition || broken bracket # round-2 critic: a push-only value reached through env (the `if:` itself must red) fresh; sub "$CI" "$TS" "$TS env: