ci(release): gate releases on a two-sided auto-update canary (#5222)

[sanity]

Problem

Auto-update was broken fleet-wide for two consecutive releases (v0.2.120 and v0.2.121) and nothing noticed. #5104 made the node's release-tag fetch return the tag verbatim (v0.2.121) and normalised it at only one of its two consumers. The detection path kept the raw tag, semver::Version::parse("v0.2.121") failed, every update was dropped with a warn!, and ~1,100 nodes had to be told to run freenet update by hand — because a broken updater cannot deliver its own fix.

Nothing caught it because every signal we had was one-sided: the release built, published, installed and ran. The one machine positioned to notice — framework, the designated real-NAT pre-release smoke peer — had been running with --disable-auto-update since a #5040 measurement window nine days earlier. The existing pre-release smoke test validates contract subscribe/UPDATE over real NAT; it never tested the updater.

Approach

scripts/auto-update-canary.sh, wired in as two gates.

Gate A — pre-flight, BLOCKING. Runs the binary that is about to ship and requires its updater to read GitHub's current release tag. It sits inside attach-to-release between asset upload and un-draft, so the assets are real but nothing has reached a user: a failure costs a stuck draft rather than a stranded fleet. Adds ~1 minute.

Gate B — self-update, post-publish. Takes the previous release and requires it to detect this one, exit 42, and self-replace via freenet update. That is the transition the fleet actually makes — not "the log looks right" but "the old version ends up on the new one". It cannot run earlier: the detection path is hardwired to GitHub's /releases/latest, and a draft release does not appear there. So Gate A is the blocking half and Gate B is the alarm (red job + River dev-room message).

scripts/release.sh is also changed: its publish_draft_release() un-drafted the release as soon as all assets were attached, without checking the workflow's conclusion — which is exactly the window Gate A opens between upload and publish. The local release driver would have raced in and published a release whose updater the gate was mid-way through rejecting, silently turning a blocking gate into no gate. It now refuses (non-zero, so the driver aborts rather than announcing) unless the tag's cross-compile attach job concluded successfully. (Deliberately the job, not the run: reading the run would treat a failing non-blocking Gate B as a failed release.)

Both gates are needs-chained steps rather than release: published listeners, so they cannot silently stop running if RELEASE_PAT lapses — that token already suppresses downstream events when unset (#4118), which is the same silent-lapse shape as the incident.

Why the assertion is two-sided

assert_detection_healthy requires the Startup update check against GitHub line to be present and Startup update check: failed to parse to be absent. Absence of the error alone proves nothing: it is equally consistent with the check never running, which is exactly what --disable-auto-update, a dirty build, or a node that never reached the update task all produce.

The marker is deliberately truncated at parse so it covers both arms of compare_versions_for_startup — the latest-version arm (the #5221 break) and the current-version arm. Both return None and both then reach the completion line, so under a longer marker a node that failed the current-version parse looked identical to a healthy one.

The canary also fails outright if the node under test has auto-update disabled. That turns "the canary was quietly switched off" from a human-memory dependency — which lapsed for nine days and is the actual root cause of the two-release blind spot — into a red build.

GitHub being unreachable is a distinct third outcome (retry), so a network blip neither fails a good release nor gets papered over with a retry that would also swallow a real parse failure.

Gate A asserts a positive fact, not the absence of an error

The node now logs the release it actually compared against (latest=), and the gate resolves the expected value independently from the same releases/latest redirect the node uses, then requires them to match.

This is load-bearing. "No error appeared" is satisfied by a component that is silently wrong as well as by one that works: a version_from_tag regressed to a constant, or a normaliser truncating 0.2.123 to 0.2.12, still parses, still compares, still declines to update, and logs a completion line byte-identical to a healthy node's. Verified by mutation — a truncating comparator drives the gate red with a specific diagnosis, while the unmutated build on the same command passes.

Testing

Correction to an earlier revision of this section. It previously claimed preflight v0.2.122 → passes. That is no longer true and should not be relied on: Gate A now requires two markers that no already-shipped binary emits — Startup update check complete (was a compiled-out debug!) and the new latest= observed-release line. Running preflight against a published binary such as v0.2.123 therefore returns a blocking UNVERIFIED, correctly. Gate A's subject is the binary about to ship, which carries both markers.

Validated at this head:

Gate	Case	Result
A	preflight against a release build of this branch	passes; reports the release compared against
A	comparator mutated to truncate the version	fails, naming the wrong release it compared against
A	preflight v0.2.120	fails — would have blocked that release
A	preflight against any published binary	UNVERIFIED by construction — markers absent (see above)
B	selfupdate 0.2.119 -> 0.2.122	passes end-to-end (detect, exit 42, freenet update, new binary)
B	selfupdate 0.2.121 -> 0.2.122	fails — the fleet's actual situation

The counterfactual holds: had Gate A existed at v0.2.120, it would have blocked publication and the outage would not have happened.

Suites, run unfiltered: auto-update-canary_test.sh 59, release_canary_wiring_test.sh 14, release_wait_for_binaries_test.sh 16, auto-update-canary_lifecycle_test.sh 9. shellcheck -x clean; release.sh is now in the CI shellcheck list.

Every pin here has been mutation-tested — applied the regression it names, observed it go red, reverted. That discipline is the point of the PR and it repeatedly caught pins that verified nothing (below).

Review

Full-tier across several rounds and multiple independent blind lenses (code-first, testing, adversarial, plus repeated verification passes). Together they found fifteen findings, every one fixed. The ones worth naming, because they are the PR's own subject recurring inside it:

Four pins that verified nothing.

• A whole-file grep for the parse-failure marker was satisfied by a // comment in auto_update.rs's own #[cfg(test)] block. Rewording the production warn! left all assertions green while the gate reported OK for a node that had failed to parse. Fixed by shortening the marker and binding the pin to the warn! macro.
• The trigger-site count pin derived the expected count from the very regex it audited, so a site the regex missed was invisible to the count too. Now anchored on update_tx.send(new_version), a structural fact the regex cannot influence.
• The source pin: trigger phrase was satisfied by the Move release pipeline to GitHub Actions, with HTTP update-agent on gateways #4073 refusal line — whitespace-stripped, its needle is a substring of not triggering auto-update, so the pin tracked a line saying the opposite of what it pinned.
• An assertion passed no want_msg, so the harness compared only exit codes and the message could be replaced with a no-op undetected.

Two routes to silently disabling the gate, both leaving every wiring assertion green: if: always() on the publish step, and || true appended to the canary invocation. The second matters more — it is the reflex fix when a gate false-positives at 2am and it does not look like disabling a gate. Root cause was structural: the wiring test policed the canary step while nothing asserted anything about the publish step, which is the step whose conditional execution is the gate. Both now pinned.

A live bug in the release driver, pre-existing on main. verify_required_binaries piped a variable into grep -xqF under set -euo pipefail; grep -q short-circuits, the producer takes SIGPIPE, pipefail promotes 141, and a binary that is present reads as missing. Measured at 46 in 20,000 under CPU contention, 0 in a quiet window. Consequence: wait_for_binaries returns 1 from a bare call under set -e and the driver dies after publish, before gateway updates and announcements — verbatim the failure release.sh warns about. It also made release_wait_for_binaries_test.sh genuinely flaky (~3 failures in 25 runs, a different binary each time); after the fix, 30 of 30 runs pass. All eight status-consuming sites in release.sh were fixed, the file added to the SIGPIPE pin, and the hazard documented in .claude/rules/bug-prevention-patterns.md with an audit grep.

Gate A destroyed its own evidence. The two branches most likely to fire on a healthy release printed no node log at all, and cleanup then deleted the workdir — while RELEASING.md told the operator to read the job log. Fixed; the log is now retained.

Overlapping runs were misdiagnosed. Two preflight runs on one host collided on fixed ports; the node exited 43 (ALREADY_RUNNING) and the gate reported "the startup update check never ran" — an auto-update fault for a port collision — then returned 1, skipping the retry that would have fixed it. Ports are now redrawn per attempt, exit 43 is classified environmental (rc=2, retried), and the header's false "safe to run on a machine already running a node" claim is corrected.

Also fixed: the gate's own curl had no --retry, so a single transient blip blocked a release; a gateway-index fetch failure was reported as "the update check never ran", pointing at the wrong subsystem; the MARKER_LATEST_SEEN_SINCE constant and Gate B's version gate were unpinned and survived inversion; and a code comment stated the wrong failure direction for grep -a.

Limitations, stated plainly

• Gate A verifies that the shipping binary can parse the current latest tag. It cannot verify it against a tag format that does not exist yet. It would have caught fix(update): restore the node-bucket pin that #5103 silently voided #5104; it would not catch a future change to the tag format itself. Gate B covers that one release later, the earliest the real detection path can be exercised at all.
• A wrong comparison of correct values is invisible to Gate A. Inverting latest_ver > current_ver logs correctly, declines to update, and the gate stays green — structurally, because the shipping binary is by construction newer than latest, so "declined" is the right outcome. Covered by the Rust unit tests; Gate B only exercises the previous release's binary.
• Gate B's own code is never executed by a test. The download, extraction check, exit-42 assertion, freenet update invocation and version comparison are exercised only in production. Deliberately deferred rather than dropped; recorded in docs/RELEASING.md.
• A node process was once observed outliving the script by ~4 minutes, reaped by its own timeout. Never reproduced, and the existing lifecycle pin cannot see it because it drives a fake node with none of a real node's shutdown behaviour. Documented rather than chased.

Also done (not in this diff)

The stale 99-disable-auto-update-5040.conf drop-in has been deleted from framework and systemctl --user daemon-reload run; the running process is now freenet network with no --disable-auto-update, and the #5040 capture is confirmed complete (results posted 2026-07-30). The node was not stopped and stays on the network.

Refs #5222, #5221, #5104.

[AI-assisted - Claude]

[github-actions]

Rule Review: Minor commit-message hygiene only

Rules checked: git-workflow.md, code-style.md, testing.md
Files reviewed: 12 (2 matched code-style.md/testing.md scope: crates/core/src/bin/commands/auto_update.rs, crates/core/src/bin/freenet.rs; the rest — workflow YAML, shell scripts, docs — fall outside the paths these three rule files cover)

The two Rust changes are limited to adding tracing::info! calls (raising debug!→info! so release builds emit the marker, plus a new positive-fact latest= log line) with load-bearing WHY comments explaining the level choice. No .unwrap(), no spawns, no retry/backoff loops, no biased; select, no new config fields, no public API — nothing in code-style.md or testing.md is implicated by these hunks.

The rest of the diff (CI workflow wiring, scripts/*.sh, docs/RELEASING.md) isn't in scope for the three loaded rule files (code-style.md is Rust-only via its path frontmatter, testing.md is scoped to crates/core/**), so it wasn't evaluated against them here.

Warnings

None.

Info

• commits.txt:19 (303ac7c2 wip: in-progress review fixups (recovered from stopped session)) — subject line uses wip:, which is not one of the valid conventional-commit prefixes (rule: git-workflow.md "WHEN creating a commit message"). Likely immaterial if the PR is squash-merged under a compliant title, but flagging per the letter of the rule.

---

Rule review against .claude/rules/. WARNING findings block merge.

[sanity]

Multi-perspective review (Full tier) — all findings addressed

This touches release/CI configuration, which is a Full-tier surface, so it got two independent blind reviewers with distinct lenses: code-first (read the code before the description, flag intent/implementation mismatches) and release-pipeline-risk (single question: could this break or stall releases?). Both read the checked-out code, not just the diff.

They agreed on more than I expected, and between them found four blocking issues. All are fixed in f4eb3f9 and the gates were re-validated against real binaries afterwards.

Blocking

1. Gate A did not actually block anything. scripts/release.sh's publish_draft_release() un-drafts the release the moment all assets are attached, without checking the workflow's conclusion — which is exactly the multi-minute window this PR opens between upload and publish. The local release driver would have raced in and published a release whose updater the gate was in the middle of rejecting, silently turning a blocking gate into no gate. It now refuses while the tag's cross-compile run is unfinished or failed.

This was the most important finding: without it, the PR's central claim was false.

2. attach-to-release kept timeout-minutes: 10, which the canary's retry path could exceed (measured baseline for that job is ~40s, so nobody had ever needed the headroom). GitHub would have cancelled the job mid-canary, skipping the Publish step and leaving a permanently stuck draft — the precise failure the retry existed to ride out. Raised to 25, canary step bounded at 12, and the worst case cut (180s node timeout, 2 attempts, no trailing sleep).

3. check-token-coalesce.yml would have failed this PR deterministically — Gate B's read-only gh api step used a bare GITHUB_TOKEN. Marked # coalesce-exempt: with a reason.

4. The failure notification covered only the non-blocking gate. A Gate A failure fails attach-to-release, which skips Gate B, so needs.auto-update-selfupdate-canary.result is 'skipped' and no message went out. The blocking gate was the silent one — a release stuck as an unpublished draft with nobody told. That is the same silent-fail-closed shape this PR exists to remove, reintroduced inside the fix. The notification now covers both gates and reports which failed.

Important

• MARKER_TRIGGERED was 'triggering auto-update' — a substring of the Move release pipeline to GitHub Actions, with HTTP update-agent on gateways #4073 refusal line "...not triggering auto-update". A node that deliberately declined an update read as one that requested it. Anchored on the full positive phrase, and pinned in both directions in the test.
• Gate A's retry could not produce a different answer. It wiped only the logs, leaving the node's persisted GitHub rate-limit cooldown in place, so every attempt re-read the same cooldown and reported the same INDETERMINATE without asking GitHub again. Each attempt now gets a fresh state tree.
• Gate B cried wolf on infrastructure failures, alarming "the fleet will not converge" when GitHub was merely unreachable. It still fails (green on an unverified run is the vacuous pass this whole change is against) but is worded as UNVERIFIED, and the notification no longer asserts which failure occurred.
• A node that fails to boot was diagnosed as a broken updater, pointing whoever is on call at the wrong subsystem.
• The Gate A recovery advice was actively harmful. release.yml publishes to crates.io before pushing the tag, so a Gate A block leaves the crate live with no GitHub release — and the docs told you to delete the tag, stranding an unyankable crates.io version pointing at nothing. Rewritten to name the split state, try a re-run first for infrastructure failures, and warn that a dirty local build disables auto-update and so cannot reproduce the gate.

Plus unchecked tar/curl in Gate B, curl --max-time, and an exact rather than substring version match (0.2.12 matched 0.2.121).

Dismissed, with reasons

• "Gate A should not block on a node that fails to boot" — kept blocking. A shipping binary that cannot start at all is worth stopping a release for; the fix was to make the diagnosis distinct so it does not send someone hunting in the update path.
• ".[0] of the releases API is created_at-ordered, not semver-ordered" — real, but harmless for freenet's linear release flow, and a semver sort would pick a less representative previous release for the fleet. Noted, not changed.
• "first-release / empty-jq case fails" — correct fail-closed shape on a repo with one release; not reachable here.

Re-validation after the fixes

Because the changes touched the marker, the retry loop and the version match, both gates were re-run against real release binaries and the real published v0.2.122:

Gate	Case	Result
A	preflight v0.2.120	fails (correct)
A	preflight v0.2.122	passes
B	selfupdate 0.2.119 -> 0.2.122	passes end-to-end
B	selfupdate 0.2.121 -> 0.2.122	fails (correct)

shellcheck clean; the 9-case self-test passes; mutation testing still catches deletion of each assertion branch.

[AI-assisted - Claude]

[sanity]

Second review round — re-reviewed the fix commit, found a blocking bug in it

Per the per-code-content rule, the first review went stale when I pushed fixes, so I re-reviewed the fix commit blind. That was worth doing: the fix commit had introduced a blocking bug and a false-red flake, either of which would have been worse than the original gap.

Blocking: release.sh announced a release it had just refused to publish

publish_draft_release() returned 0 when the new guard declined, so the caller reported success and the driver went on to update the gateways and announce to Matrix and River — for a release still sitting as an unpublished draft. It now returns non-zero (release.sh is set -euo pipefail, so that aborts the driver), and the function checks is it still a draft first, so an already-published release stays a clean no-op.

The same guard was also inverted. An empty gh run list does not yield "" as I assumed — it yields the literal null:null, because jq interpolates .[0] == null. Verified:

$ gh run list --workflow=cross-compile.yml --branch v9.9.9-nonexistent \
    --json status,conclusion --jq '.[0] | "\(.status):\(.conclusion)"'
null:null

So "no run found" refused, while the only route to publish-anyway was gh itself failing — the fail-open the guard exists to prevent, sitting exactly where an auth expiry or rate limit lands. Now anything other than completed:success refuses.

(The reviewer also checked the thing most likely to have silently voided the guard — whether gh run list --branch matches tag-triggered runs. It does: --branch v0.2.122 returns completed:success.)

False red: the canary failed a healthy binary

Caught by re-running the gates against real binaries rather than trusting the diff. Two causes, both verified empirically:

• The process-group kill didn't work. set -m gave the subshell its own group, but job control is inherited, so the timeout inside started a group of its own and the group kill missed it (pgids differ). Every run left a node alive holding its ports. exec-ing the timeout collapses the two. My first test of this fix used exec -a and so wasn't representative of the real multi-statement subshell — retested in the real shape.
• The poll loop's budget was dishonest, charging 3s per pass while each pass also paid for a grep and a process check. On a loaded machine a slow-booting node was reported as "the startup update check never ran". A false blocking failure is worse than no canary: the first person to hit it learns to override the gate. Now measured against the clock.

Marker: my previous fix broke it a different way

Narrowing MARKER_TRIGGERED to one call site's full phrase (fixing the substring bug) missed two of the four real trigger sites — freenet.rs logs triggering auto-update at :524, :650, :731, :853, and the #4073 refusal at :519. Matching the phrase and subtracting the refusal is the only form with both properties. The test now drives all five lines through the real helper.

The markers are now also pinned against crates/core/src/bin/*.rs, so a reword there fails CI rather than leaving the canary matching strings nothing emits — the self-matching-pin failure mode AGENTS.md warns about, which my previous hardcoded assertions had.

Re-validation

Gate	Case	Result
A	preflight v0.2.120 × 2 rounds	fails (correct), 0 surviving nodes
A	preflight v0.2.122 × 2 rounds	passes, 0 surviving nodes
B	selfupdate 0.2.119 → 0.2.122	passes end-to-end
B	selfupdate 0.2.121 → 0.2.122	fails (correct)

Mutation testing confirms the assertions catch reverting each fix, including the one-site anchoring bug above. 18 self-test assertions, shellcheck clean.

Accepted as follow-ups, not fixed here

• .[0] of the releases API is created_at-ordered, not semver-ordered. Harmless for freenet's linear release flow.
• Gate B's freenet update spends one unauthenticated api.github.com request from a shared runner IP. Bounded, and it fails loudly rather than silently, but it is the most likely source of a future flaky Gate B.

[AI-assisted - Claude]

[sanity]

Rule-review warning addressed, plus a correction to my own commit message

The rule-review/warnings check flagged that the previous commit fixed two bugs in code no test exercised. It was right — auto-update-canary_test.sh only drives the pure functions, so the part that actually runs a process, which is where both bugs were, had zero coverage.

New: scripts/auto-update-canary_lifecycle_test.sh

Drives the real run_node_until_check and cmd_preflight against a fake node binary that emits the same log lines. No network, no real node, deterministic, ~10s:

1. a healthy binary must make the gate exit 0 — can it ever go green?
2. a broken updater must make it exit 1 — can it ever go red?
3. NODE_EXIT must carry the node's own exit 42, not our SIGTERM
4. no node may outlive run_node_until_check

Case 4 is the load-bearing one and is mutation-verified: dropping the exec and reverting to a plain kill $node_pid makes it fail. Case 1 is verified to fail when the gate is forced to reject everything.

Correction: my previous commit message stated a wrong cause

Checking the rule-review's premise made me re-derive the reasoning, and one claim does not hold. The previous commit said the counting-based poll loop "charged 3s per pass while each pass also paid for a grep and a process check, so its window expired well before the nominal timeout."

That is wrong, and wrong in the opposite direction: charging 3s per pass while each pass costs slightly more than 3s makes the loop run marginally longer than nominal, not shorter. It was never the cause of the false "check never ran" verdict. The real cause was entirely the leaked nodes — earlier runs' nodes were still alive stealing CPU, so the node under test booted too slowly to log its check inside the window.

Moving to wall-clock is still the right way to express a time budget, but it fixed no bug, and the code comment no longer claims it did.

Related overclaim, same root: the new test file initially described case 1 as pinning the trap ... RETURN scoping bug. That bug was fixed during local development and never reached a commit, so there is nothing to pin against — which is exactly how the overclaim surfaced, since a hand-written mutation of it did not reproduce the failure. Case 1 now says what it actually guarantees.

Both corrections are in fba043e68. Nothing about the gates' behaviour changes; the earlier validation results stand.

I have not added the review-override label — the warning identified a real gap and it is now closed rather than waived.

[AI-assisted - Claude]

[sanity]

Fixed: Gate A could pass with no evidence

Adversarial review found a blocking bug in the gate this PR adds. Pushed a fix in 13876fc.

The bug

run_node_until_check slept a flat sleep 5 after the "Startup update check against GitHub" line, then killed the node's process group. That line is logged before the network request, and the request is bounded by PROBE_CHAIN_TIMEOUT = 10s (whole-chain: DNS, connect, redirects). So a GitHub that answered between 5s and 10s — loaded runner, redirect hop, mild rate-limit backoff — got its node SIGTERM'd before it could log success, failed to parse latest version, or failed to fetch latest version.

assert_detection_healthy then saw CHECK_RAN present, no DISABLED, no PARSE_FAIL, no FETCH_FAIL, and fell through to return 0 — OK.

A binary carrying the exact #5104/#5221 bug this PR exists to catch would have passed Gate A and been published, whenever its parse failure happened to land more than 5s after the check-ran line.

Why waiting longer was not the whole fix

The deeper problem: there was no success marker at all. The healthy Gate A outcome — the shipping binary is newer than the latest release, so the check finishes without triggering — was a tracing::debug!, and crates/core/Cargo.toml sets release_max_level_info, which compiles debug! out of release builds entirely.

So on every shipped binary the most common outcome of the whole check was invisible, and "finished, staying put" was byte-for-byte identical to "killed mid-request". Polling for an outcome that is never emitted would just have turned every healthy release INDETERMINATE and blocked every release.

The fix

• freenet.rs — promote that outcome to INFO ("Startup update check complete"). It is reached on every non-triggering path, so it asserts only that the check ended; the WARN above it, if any, still says what it found.
• auto-update-canary.sh — poll for a terminal outcome instead of sleeping, on a budget with real headroom over PROBE_CHAIN_TIMEOUT (20s, CANARY_OUTCOME_WAIT_SECS), returning the instant it arrives. The happy path is now faster than the old fixed sleep.
• assert_detection_healthy — require a terminal outcome. A check that started and never finished is INDETERMINATE (exit 2, retried, then failed as UNVERIFIED), never OK. Unknown and pass are now different answers.

Proof, both directions

Lifecycle case 5 is the regression pin: a fake node that logs its parse failure 8s after the check line — inside what production allows, outside what the gate used to watch.

Running the new test file against the pre-fix script (extracted via git show HEAD:scripts/auto-update-canary.sh):

ok   - cmd_preflight returns 0 for a healthy binary
ok   - cmd_preflight fails a binary whose updater cannot parse the tag
ok   - NODE_EXIT preserves the node's own exit 42
ok   - no node survives run_node_until_check
FAIL - cmd_preflight returned OK for a binary whose updater FAILED TO PARSE, because the
       failure was logged 8s after the check started (the #5236 vacuous pass)
FAIL - cmd_preflight returned OK for a node that logged NO outcome

After the fix, all six pass, and case 5 asserts on the diagnosis (could not parse the version GitHub returned), not just the exit code — the point is that the canary now sees the parse failure, not merely that it stopped saying OK.

Also mutation-tested the new source pins: demoting the completion marker back to tracing::debug! fails CI with a specific message. All 22 pre-existing assertions still pass, unchanged.

Two related findings from the same review

• ci.yml shellchecked four scripts but not the three canary scripts, despite this PR claiming they were shellcheck clean — nothing was holding them to it. Added (with -x, since the tests source the canary). All seven pass.
• release.sh gated on the cross-compile run's aggregate status, which also covers Gate B — deliberately non-blocking, and running only after the release is published. A Gate B failure therefore returned non-zero from wait_for_binaries, which is called bare under set -e, aborting the driver before the gateway updates and the Matrix/River announcements for a release that had published perfectly well. Now watches the attach-to-release job's own conclusion; an unknown state still refuses to publish.

One disclosure

A previous release built before this change cannot emit the completion marker. That affects only Gate B, whose subject is the previous release: a healthy one triggers an update and settles on that, but an old one that neither triggers nor fails now reports UNVERIFIED instead of "did NOT decide to update". Still a refusal, and still the correct one — just a less specific message, and only until the previous release is itself post-fix.

Not merged, no auto-merge set.

[AI-assisted - Claude]

[sanity]

Consolidated review — 5 independent lenses

Reviewed at c91a8a4bd. Lenses: two testing, two skeptical, one big-picture, each blind
to the others. Between them they executed both suites, ran nine-plus mutations against
throwaway copies, reconstructed the pre-fix script to check the claimed reproduction, and
traced the gate into the real node source. Findings I verified myself are marked.

The headline claim holds and is mutation-verified: the original vacuous pass is
genuinely fixed, the process-running path (spawn / poll / kill / classify) is now
exercised rather than only the pure helpers, and no path was found where a missing
marker, timeout, crashed node, missing binary, network failure, or unparseable log yields
rc=0 on today's code.

But two demonstrated false-greens remain, both one ordinary edit away from live. The
gate's green is an inference from the absence of two exact strings, the enumeration is
incomplete, and the pin meant to keep it honest is vacuous.

---

BLOCKING — the parse-failure source pin matches a comment, not the code

auto-update-canary_test.sh:226 pins MARKER_PARSE_FAIL with a whole-file grep -qF.
The string occurs twice in crates/core/src/bin/commands/auto_update.rs:

• :1546 — the production tracing::warn! the pin exists to protect
• :1757 — inside a // comment in the #[cfg(test)] mod tests block

I confirmed both occurrences. Mutation-proved end to end by one lens: reword ONLY the
production warn! and all 22 assertions still pass, including
ok - source pin: parse-failure marker. A node then carrying the #5221 bug logs
check-ran + reworded-warn + check-complete, and the real script returns rc=0 with
OK: startup update check ran to completion and parsed GitHub's response.

So a routine reword of a log message silently turns Gate A back into the thing this PR
removes, CI green throughout. This is the trap AGENTS.md names directly ("a bare anchor
matches a later occurrence").

Being fixed two ways, because the first alone would work only by luck: shorten the marker
(below), and bind the pin to the tracing::warn! call itself, the way the INFO-level
check-complete pin at _test.sh:217 already does for its target. Mutation-tested against
that exact reword.

BLOCKING — an unenumerated parse failure, with the gate printing the opposite

auto_update.rs:1535 emits "Startup update check: failed to parse CURRENT version...".
No marker matches it; it returns None, falls through to the completion line, and reads
healthy. Demonstrated by execution on a three-line fixture:

OK: startup update check ran to completion and parsed GitHub's response.   rc=0

This is #5104's mistake applied to the other operand. Not reachable with today's
build_info::VERSION (Cargo enforces semver), which is the only reason it is not filed
higher. Fix is one string: MARKER_PARSE_FAIL='Startup update check: failed to parse'
covers both operands — and, as it happens, stops the comment match above too.

IMPORTANT — nothing pins the workflow wiring

Nothing asserts that cross-compile.yml still runs the preflight step, or that it runs
before gh release edit --draft=false. Delete the step or move the publish above it
and the blocking gate becomes a no-op with the whole suite green — the "silently
removable gate" shape #5222 exists to eliminate. ATTACH_JOB_NAME (release.sh:1206) must
also match cross-compile.yml:362 verbatim, with no pin on the pair.

The repo already has this pattern: release_mergequeue_test.sh:45 greps release.yml for
gh pr merge, and merge_group_concurrency_test.sh pins concurrency keys. Being added,
with order asserted rather than mere presence.

IMPORTANT — an unprotected gh call can abort the release driver mid-build

release.sh:1381 (added here) is a bare run_status=$(gh run view ... 2>/dev/null) under
set -euo pipefail, inside wait_for_binaries, which is called bare at :1598 — so
errexit is live. A transient gh failure kills the whole driver. Verified:

$ bash -c 'set -euo pipefail; f(){ return 1; }; x=$(f); echo reached'; echo "exit:$?"
exit:1          # "reached" never printed

The consequence is in the script's own comment at :1216-1218 — the abort happens before
the gateway update and the Matrix/River announcement, so "a release that published
perfectly well would silently never be announced." The sibling at :1370
(attach_job_state) has the same shape and is pre-existing; both are being fixed, since
leaving one is leaving the bug.

Related, same function: publish_draft_release's is_draft=$(gh release view ... || echo "false") returns 0 on a gh failure — "unknown, nothing to gate" — while every other
unknown in that function refuses with 1. It cannot publish an ungated release, but it lets
the driver proceed for a release that may still be a draft.

IMPORTANT — the new release.sh publish-gate logic has no tests

release.sh:1206-1275 adds ~100 lines of new decision logic on the release critical path,
in a function whose own comment records a prior fail-open. Nothing covers job_state
values "" / in_progress: / completed:failure / completed:success.

IMPORTANT — Gate A's healthy verdict is byte-identical to a silently-wrong comparator

The expected answer IS "no newer version", so version_from_tag regressing to a constant,
or a truncating normaliser whose output still parses, produces the same log and the same
green. Gate A can only see failures that emit a WARN.

The structural fix dissolves this and the two BLOCKING items as a class rather than one at
a time: have the completion line carry what it compared against (latest = ... beside the
existing current = ...) and have the canary assert it equals the tag CI already knows.
That converts absence-of-error into a positive equality check. Worth considering as a
follow-up even after the string fixes land.

IMPORTANT — what Gate A proves is narrower than its name suggests

Gate A exercises fetch -> tag normalise -> semver parse -> compare, and nothing else. It
does not exercise signature verification, checksum matching, asset download, binary
swap, the exit-42 supervisor plumbing, or crash-loop rollback. Gate B does cover
download/signature/checksum/swap, but it is post-publish, non-blocking, and runs the
previous release's binary — so a break in the shipping binary's installer half is
caught one release later. That is one release of exposure rather than two, better than the
incident, but the same shape.

RELEASING.md describes Gate A accurately but never states its limits, and the PR's
"Limitation, stated plainly" section names only the future-tag-format case. A paragraph is
being added, because the next person hitting a broken installer will reasonably ask "we
have an auto-update canary, why didn't it catch this?"

---

Minor

• A fifth trigger site the detector cannot see. auto-update-canary.sh:73-83
  enumerates "FOUR such sites" and _test.sh:151 claims to cover "ALL FOUR". There are
  five: freenet.rs:609 logs "Urgent update confirmed on GitHub, triggering IMMEDIATE
  auto-update", which MARKER_TRIGGERED does not match. Fail-closed, but comment and test
  both assert an enumeration that is wrong.
• pipefail + SIGPIPE in node_decided_to_update / node_check_settled
  (:155-156, :168): grep ... | grep -q . returns FALSE on a log that does contain
  the marker once output exceeds the 64 KB pipe buffer. Measured: 400 matching lines
  rc=0, 700 lines rc=141, 30/30 reproducible above threshold. Unreachable at canary
  volumes and fail-closed, but a live trap for anyone reusing these helpers.
• CANARY_TIMEOUT_SECS is unvalidated (:114) while its two neighbours at :125 and
  :140 both get numeric guards. A non-numeric override dies at the arithmetic under
  set -u.
• The INFO pin is not function-bounded. It is a cross-file scrape so the self-match
  trap cannot apply, but it would be satisfied by the literal appearing anywhere in
  freenet.rs including its test module. One occurrence today.
• Lifecycle case 4's leak check is timing-dependent — the fake runs under a 15s
  timeout and the test greps ~6s in, so on a loaded runner the leak could already be
  dead and the case would pass vacuously.
• Log rate limiting is on in release builds (1000 events/s global, per-callsite cap).
  A dropped WARN with a surviving completion line is a theoretical false green; the window
  is microseconds and a canary node is nowhere near that rate, but
  FREENET_DISABLE_LOG_RATE_LIMIT=1 in the canary subshell removes the class for free.
• Fixture drift: case 5's fake models a PRE-ci(release): gate releases on a two-sided auto-update canary (#5222) #5236 binary (parse-fail, no completion
  line). A post-ci(release): gate releases on a two-sided auto-update canary (#5222) #5236 binary with a Fleet cannot auto-update: #5104 made the node parse the raw 'v0.2.121' tag as semver and silently drop every update #5221-class bug logs parse-fail THEN completion. The
  verdict is identical either way, so a missing fixture rather than a hole.

Confirmed sound

assert_detection_healthy has no OK-without-evidence path — all seven exits enumerated,
and the only return 0 requires real emitted-log evidence, never inferred from absence.
INDETERMINATE is not an ignore-hatch: both callers convert it to hard failure once retries
exhaust, so it buys a retry, not a pass. No CI-chasing found — the timeout raise is
bounded and justified, CANARY_OUTCOME_WAIT_SECS=20 is derived from the real
PROBE_CHAIN_TIMEOUT=10s rather than tuned until green, and the sleep-to-polling change
tightened the gate. Nothing removed or weakened in the test files (22 -> 28, all
additions). The debug!->info! promotion is justified, fires at most once per process
on a non-hot path, is not an enumerated telemetry event, and carries a 17-line guard
comment naming the script that greps it. Both un-draft sites are gated. shellcheck -x
clean. The new tests run on every PR and block merge.

Filed separately, deliberately out of scope here

#5274 — Gate B (cmd_selfupdate) has zero test coverage, and the "node never logs at all"
path is untested while running ~8.3 min against a 12-min step budget.

Lenses: code-first, testing x2, skeptical x2, big-picture.

[AI-assisted - Claude]