Skip to content

fix(nightly): separate "blocked at a gate" from "driver broke", fix the page detail - #196

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/nightly-outcome-contract
Aug 4, 2026
Merged

fix(nightly): separate "blocked at a gate" from "driver broke", fix the page detail#196
Jammy2211 merged 1 commit into
mainfrom
feature/nightly-outcome-contract

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Refs #194 (the triage that surfaced both). Two contained fixes to the nightly release driver's reporting — no change to any gate, threshold or release decision.

1. The exit-code / conclusion contract

nightly.sh already distinguished the cases in its exit codes (0 reported, 2 blocked at a gate, 3 not GREEN, 1 driver error), but nightly-release.yml ran it as a bare run: step, so every non-zero became a red run. The result: eight consecutive red nights (2026-07-28 → 08-04) that were all the gate working correctly, in a channel that consequently nobody watched.

The workflow now classifies the code:

exit meaning job
0 shipped / skipped / dry-run success
2, 3 blocked at a gate — no release made success + ::warning:: + a named Blocked at a gate step + job summary
1, other driver error — the night was NOT judged failure

Red is now reserved for the one state that needs a human to look at this workflow. A blocked night is an expected outcome — Slack already carries which gate stopped it.

So "blocked" cannot quietly become "green", bin/overnight_status.sh reads that step name and gives a blocked run its own line and tally, separate from both green and failed. Without this the change would have traded a false alarm for a silent one.

2. The stop-summary count and the None

The 2026-08-04 page read:

1 failed: autolens database/start_here.py, None verify_install

Both halves are wrong, and the real stage_report.json from that run shows why:

"summary":  {"failed": 1, "passed": 654, "skipped": 102, "timeout": 0},
"failures": [{"project": "autolens", "script": ".../database/start_here.py"},
             {"project": null, "script": "verify_install", "reason": "verify_install FAILED"}]

summary.failed counts scripts; failures[] also carries non-script legs with project: null. The old formatter listed both under the script count (so "1 failed" introduced two items) and interpolated the null as a literal None.

The formatter moves out of its heredoc into agents/conductors/release/stage_failure_summary.py — a file can carry a test, a heredoc cannot — and reports the two kinds as separate segments:

1 failed: autolens database/start_here.py; verify_install FAILED

API Changes

None. No gate, threshold, or release decision changes; no public Python API is touched. nightly.sh's exit codes keep their existing meanings — they are now documented as a contract and consumed accordingly.

Verification

  • Real-data control: ran the new formatter against the actual stage_report.json downloaded from the failing 2026-08-04 Stage-3 run → 1 failed: autolens database/start_here.py; verify_install FAILED (was …, None verify_install).
  • 11 new tests in tests/test_stage_failure_summary.py, shaped from that real report: no stringified null, count/list agreement, check-only failures, missing reason, truncation past 3, malformed counts and entries, absent/garbled file.
  • overnight_status.sh control-tested both ways with a gh shim — blocked step present → ⏸ blocked — no release made + blocked tally; absent → all green, no . Exit 0 in both (the summary lines are separate ifs precisely so a false test can't become a non-zero exit that reads as a tool failure to /wake_up). Also run against live GitHub data: today's red nightly still shows , and the five currently-green runs produce no false .
  • Full suite: 204 passed (193 + 11), and the tenant firewall still reports OK — nightly.sh and overnight_status.sh are both allowlisted files and gained no new instance facts.

The workflow change itself only takes effect on the next scheduled run; the classification logic is plain case on the exit code, and the two outcome steps are if:-gated on it.

…he page detail

Two fixes to the nightly release driver's reporting, both found triaging
eight consecutive red nights that were all the gate working correctly.

Outcome contract. nightly.sh already distinguished blocked (exit 2/3)
from a driver error (exit 1), but the workflow flattened every non-zero
into a red run — so a night the gate correctly stopped looked identical
to a broken driver, and the channel became one nobody watched. The
workflow now classifies the exit code: 0/2/3 leave the job green, with a
named "Blocked at a gate" step plus a ::warning:: and a job summary when
it stopped; only a driver error (exit 1, the night was never judged)
turns the run red. Red now means "this workflow needs a human".

So a blocked night cannot become silently green, overnight_status.sh
reads that step and gives it its own ⏸ line and tally, distinct from
both green and failed.

Page detail. The Stage-3 summary read

    1 failed: <project> database/start_here.py, None verify_install

because summary.failed counts SCRIPTS while failures[] also carries
non-script legs (project: null, reason: "verify_install FAILED") — so
the count disagreed with its own list and the null stringified as
"None". Moved the formatter out of its heredoc into
stage_failure_summary.py, which reports the two kinds as separate
segments, and covered it with tests built from the real stage report:

    1 failed: <project> database/start_here.py; verify_install FAILED

Verified against that real 2026-08-04 report, and the blocked/not-blocked
branches of overnight_status.sh control-tested both ways with a gh shim.

Refs #194

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit a2264fe into main Aug 4, 2026
2 checks passed
@Jammy2211
Jammy2211 deleted the feature/nightly-outcome-contract branch August 4, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant