fix(nightly): separate "blocked at a gate" from "driver broke", fix the page detail - #196
Merged
Merged
Conversation
…he page detail
Two fixes to the nightly release driver's reporting, both found triaging
eight consecutive red nights that were all the gate working correctly.
Outcome contract. nightly.sh already distinguished blocked (exit 2/3)
from a driver error (exit 1), but the workflow flattened every non-zero
into a red run — so a night the gate correctly stopped looked identical
to a broken driver, and the channel became one nobody watched. The
workflow now classifies the exit code: 0/2/3 leave the job green, with a
named "Blocked at a gate" step plus a ::warning:: and a job summary when
it stopped; only a driver error (exit 1, the night was never judged)
turns the run red. Red now means "this workflow needs a human".
So a blocked night cannot become silently green, overnight_status.sh
reads that step and gives it its own ⏸ line and tally, distinct from
both green and failed.
Page detail. The Stage-3 summary read
1 failed: <project> database/start_here.py, None verify_install
because summary.failed counts SCRIPTS while failures[] also carries
non-script legs (project: null, reason: "verify_install FAILED") — so
the count disagreed with its own list and the null stringified as
"None". Moved the formatter out of its heredoc into
stage_failure_summary.py, which reports the two kinds as separate
segments, and covered it with tests built from the real stage report:
1 failed: <project> database/start_here.py; verify_install FAILED
Verified against that real 2026-08-04 report, and the blocked/not-blocked
branches of overnight_status.sh control-tested both ways with a gh shim.
Refs #194
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #194 (the triage that surfaced both). Two contained fixes to the nightly release driver's reporting — no change to any gate, threshold or release decision.
1. The exit-code / conclusion contract
nightly.shalready distinguished the cases in its exit codes (0reported,2blocked at a gate,3not GREEN,1driver error), butnightly-release.ymlran it as a barerun:step, so every non-zero became a red run. The result: eight consecutive red nights (2026-07-28 → 08-04) that were all the gate working correctly, in a channel that consequently nobody watched.The workflow now classifies the code:
02,3::warning::+ a namedBlocked at a gatestep + job summary1, otherRed is now reserved for the one state that needs a human to look at this workflow. A blocked night is an expected outcome — Slack already carries which gate stopped it.
So "blocked" cannot quietly become "green",
bin/overnight_status.shreads that step name and gives a blocked run its own⏸line and tally, separate from both green and failed. Without this the change would have traded a false alarm for a silent one.2. The stop-summary count and the
NoneThe 2026-08-04 page read:
Both halves are wrong, and the real
stage_report.jsonfrom that run shows why:summary.failedcounts scripts;failures[]also carries non-script legs withproject: null. The old formatter listed both under the script count (so "1 failed" introduced two items) and interpolated the null as a literalNone.The formatter moves out of its heredoc into
agents/conductors/release/stage_failure_summary.py— a file can carry a test, a heredoc cannot — and reports the two kinds as separate segments:API Changes
None. No gate, threshold, or release decision changes; no public Python API is touched.
nightly.sh's exit codes keep their existing meanings — they are now documented as a contract and consumed accordingly.Verification
stage_report.jsondownloaded from the failing 2026-08-04 Stage-3 run →1 failed: autolens database/start_here.py; verify_install FAILED(was…, None verify_install).tests/test_stage_failure_summary.py, shaped from that real report: no stringified null, count/list agreement, check-only failures, missingreason, truncation past 3, malformed counts and entries, absent/garbled file.overnight_status.shcontrol-tested both ways with aghshim — blocked step present →⏸ blocked — no release made+ blocked tally; absent → all green, no⏸. Exit 0 in both (the summary lines are separateifs precisely so a false test can't become a non-zero exit that reads as a tool failure to/wake_up). Also run against live GitHub data: today's red nightly still shows✗, and the five currently-green runs produce no false⏸.nightly.shandovernight_status.share both allowlisted files and gained no new instance facts.The workflow change itself only takes effect on the next scheduled run; the classification logic is plain
caseon the exit code, and the two outcome steps areif:-gated on it.