fix(canary): isolate TMPDIR so Gate A stops blocking healthy releases - #5290
Conversation
Gate A blocked v0.2.124 on a binary that was perfectly healthy.
The node's contract web directory is `std::env::temp_dir()/freenet/webs`
(client_api.rs:255), hardwired -- it does not follow `--data-dir`. The canary
isolated HOME, config, data and log dirs but not TMPDIR, so that path escaped
its sandbox. `cross-compile.yml` stages the binary it is about to gate at
`/tmp/freenet`, which is exactly the path the node then tries to create a
directory under: `create_dir_all` hit ENOTDIR against the binary FILE, the node
panicked (exit 101) before the update task spawned, and Gate A correctly
reported "the startup update check never ran" -- of a harness fault, not a
product one.
Verified against the SHIPPED v0.2.124 artifact, staged at $TMPDIR/freenet to
reproduce the collision:
fixed rc=0 "OK: the node compared against '0.2.123', which matches
GitHub's latest release."
control rc=1 node exited with code 101, panicked at client_api.rs:256:
"Failed to create contract web directory at .../freenet/webs:
Not a directory (os error 20)"
Relocating the binary alone is NOT sufficient: with the default TMPDIR the same
artifact still came back UNVERIFIED on a host where /tmp/freenet already exists
as another user's directory. Isolating TMPDIR is what fixes it, and it closes
the second half of the header's isolation claim -- the PORTS half was corrected
earlier; two nodes on one machine also shared /tmp/freenet/webs until now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
Rule Review: TMPDIR isolation fix — no blocking issuesRules checked: git-workflow.md This is a WarningsNone. Info
Rule review against |
…s, pin it Review of #5290 found the TMPDIR fix correct but its surrounding claims wrong in three places, plus the fix itself unpinned. The path is not what the comments said it was. `contract_web_path` (`client_api.rs`) is VESTIGIAL: `"webs"` has exactly one occurrence in `crates/`, the `create_dir_all` itself, and unpacked web contracts live under `default_webapp_cache_dir` instead. So "two nodes on one machine share /tmp/freenet/webs" was wrong -- they would share an always-empty, never-read directory. The real hazard is that the mkdir can FAIL and panic the node at startup: `$TMPDIR/freenet` being a file (the CI case, ENOTDIR) or another user's directory without a `webs` child (EACCES). Both comments now say that. Two escapes from the isolation the header claims: - Gate B leaked. The `freenet update` subshell exported only HOME, and `download_and_install` stages the downloaded tarball in `tempfile::tempdir()`, so a release tarball landed in the ambient system temp dir. It now scopes TMPDIR too. Safe for the swap: `replace_binary` copies to a `.freenet.new.tmp` beside the DESTINATION and renames there, so the atomic same-filesystem rename never involves TMPDIR. - The webapp cache escaped via XDG. `default_webapp_cache_dir` resolves through `directories::ProjectDirs`, which reads XDG_CACHE_HOME ahead of HOME, and `FREENET_WEBAPP_CACHE_DIR` overrides both; `WebappCache::with_root` create_dir_all's it on every boot. On a host with XDG_CACHE_HOME set the canary wrote outside its workdir. Fails soft (a warn), so it never blocked a release -- scoped now so the header's claim is true rather than nearly true. Nothing pinned the fix: TMPDIR appeared in none of the four suites, so deleting `export TMPDIR` left everything green and the fault would resurface only at the next release, as a blocked release blaming the product. Added a cross-file source-scrape pin asserting TMPDIR is exported before BOTH launch sites -- before the node's `exec` (which replaces the shell, so a later export never runs) and before `freenet update`. Mutation-proven in both directions. Also corrected in the collision comment: it reported "UNVERIFIED" as the outcome of the collision case, which is the verdict for an exhausted rc=2 retry loop, not for this. The collision produces rc=1 via `assert_detection_healthy`. It now cites the reverted-fix control instead (fixed rc=0 vs control rc=1, exit 101, "Not a directory (os error 20)"), the stronger evidence the comment had omitted. Minor: "Until v0.2.124" -> "Through v0.2.124" (this ships in 0.2.125), and docs/RELEASING.md now warns that reproducing locally against ./target/release/freenet is precisely the environment where this class of fault cannot occur -- which is why local validation went 4/4 green while CI blocked. Suites: canary 60 (was 59, +1 pin), lifecycle 9, wiring 14, wait_for_binaries 16. shellcheck -x clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
…rect a false claim Three things, from the testing lens on #5290. A REAL test, not a source scrape. Lifecycle case 9 boots the canary against a fake node that IS the regular file staged at `$TMPDIR/freenet` -- the CI layout -- and models `client_api.rs`'s hardwired `create_dir_all(temp_dir()/freenet/webs)`. The ENOTDIR comes from the kernel refusing to mkdir under a file, not from a fixture printing a panic on cue. Verified both directions: green here, and against origin/main's script it goes red with "the canary let its own TMPDIR reach the node ... a HEALTHY release was blocked (exit 1)". Case 10 is its companion: the fake dumps its own environment and the test asserts all four exports the canary makes. That closes a sibling gap -- deleting `export HOME=` or `export FREENET_SUPERVISED=1` also left all four suites green, because nothing observed them. On the origin/main control exactly the two TMPDIR assertions fail and the other twelve pass, so the new cases are not over-broad. The source-scrape pin stays, demoted in its own comment to what it actually is: the weaker check. It cannot tell `export TMPDIR="$work/tmp"` from `export TMPDIR=/tmp`. It earns its place on the GATE B half (which cannot be exercised without downloading a real release) and on ORDER. Its previous comment claimed no behavioural test was possible -- that is now false and is corrected rather than left to rot. A FALSE claim, removed. The comment said "Relocating the binary alone is NOT sufficient; the isolation is the fix." Tested and it is wrong: pre-fix script, a scratch TMPDIR containing no `freenet` entry, binary staged outside it -> rc=0, "the node compared against '0.2.121'". Relocating the staged binary in cross-compile.yml WOULD have unblocked v0.2.124. The contrary evidence came from a host where /tmp/freenet already existed as another user's directory -- the EACCES case generalised into a claim about the ENOTDIR case. The comment now states the true, narrower reason isolation is still the better fix: relocation leaves the canary sharing the caller's temp dir, so on a machine already running a node under another user the gate can still panic on a /tmp/freenet it does not own. A wrong assertion inside an otherwise precise block is worse than one on its own -- it borrows the credibility of its neighbours. `mkdir -p "$work/tmp"` moved out of the node subshell into the workdir setup in `run_node_until_check`, and guarded. Gate A did not care (the node's own create_dir_all builds its parents) but Gate B does: `tempfile::tempdir()` requires TMPDIR to EXIST and will not create it, so the export added for Gate B needed the directory to exist before the installer runs. Inside the backgrounded subshell a failed mkdir was also invisible -- no `set -e`, output to node.out -- so a broken workdir would have surfaced as a mystery update-path failure instead of as itself. Suites: canary 60, lifecycle 14 (was 9), wiring 14, wait_for_binaries 16. shellcheck -x clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
…ting Cases 9 and 10 look redundant and are not. Rewriting the export to `TMPDIR=/tmp` leaves case 9 GREEN on any host where /tmp/freenet is a usable directory (it is one on nova) -- the node's mkdir then succeeds somewhere useless instead of failing. Case 10 reads the value the node actually got, so it fails on that mutation unconditionally. Deleting the export fails both, and the source scrape is green on both. Recorded so the next reader does not delete one as a duplicate of the other. Observed, not reasoned: each claim here is a mutation that was applied and run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
The `fix:`-must-add-a-test rule already knew that shell-tooling fixes are tested by `*_test.sh` self-tests rather than Rust `#[test]`s -- its own comment says the shell branch exists "so such fixes aren't forced to claim test-exempt". But it counted only `check ` assertions, and `auto-update-canary_lifecycle_test.sh` reports through `ok`/`bad`. So it scored 0 against this PR, which adds a behavioural lifecycle case that reproduces the fault it fixes and goes red without it (mutation-verified: drop `export TMPDIR` and the suite fails with two assertions naming #5290). The rule demanded `test-exempt` for a PR carrying exactly the test the rule wants. That is the same shape as the bug this PR fixes and as several found reviewing it today: a gate that cannot see its own subject, reporting on something other than the thing under test. Counting `ok` alongside `check`: 5 added assertions detected here, 0 under the old pattern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
…lly ships it The constant names the first release whose binary emits the observed-latest marker, and Gate B skips its positive equality check for anything below it. It said 0.2.124 -- but 0.2.124 was never PUBLISHED: Gate A blocked it on the TMPDIR harness bug (#5290), and a draft release is invisible to `/releases/latest`. So no release a node can actually reach emits that line until 0.2.125. Left at 0.2.124, Gate B would have demanded the marker from 0.2.123 -- a binary never built to log it -- and gone red on a healthy transition, on the very gate we are relying on to catch a repeat of the 0.2.120/121 auto-update break. Caught by the constant's own pin (added in #5290) when the bump moved the crate to 0.2.125. Working as designed: the pin exists because raising the constant silently disarms Gate B, and it is anchored to the crate version precisely so a stale value cannot survive a release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
…gether Addresses the Full-tier review on #5303. The relation pin this PR added was wrong in the same way `main`'s was, with the sign flipped, and the review found a silent failure neither pin could see. DELETE THE RELATION PIN (review 1b). Two have now been tried and both were correct in one phase and wrong in another. `constant >= crate` (#5290) rested on "no published release emits the marker yet" and detonated the moment 0.2.125 published. `constant <= crate` is right after publication and wrong during a marker REWORD: a marker reworded on main first ships in crate+1, so the correct constant is crate+1, which the pin calls a failure -- while the value that makes it green makes Gate B demand the new text from a published binary emitting the old one, i.e. a post-publish red canary and a Matrix alarm blaming the node. A `<= crate+1` variant fires on a reword too; the reviewer who proposed it withdrew it after testing. The generalisation, recorded in the comment so a third attempt is not made: the constant tracks RELEASE HISTORY and the crate version tracks THIS TREE, and no fixed relation between two quantities on different clocks holds across both publication and reword. The freeze is phase-independent and covers strictly more -- the relation only ever caught "strictly above crate version", while the freeze catches every wrong value including the empty string and the most likely wrong one. It also cannot be tripped by `version.workspace = true`, which broke the relation pin's Cargo.toml read. FREEZE THE PAIR (review 1a, the highest-value change). Reword MARKER_LATEST_SEEN everywhere a developer must touch it and leave the constant alone, and the whole suite passes -- verified, zero red. The source pin interpolates `$MARKER_LATEST_SEEN`, so it follows the rename by construction, and nothing else looks at the marker. A stale constant is the DEFAULT outcome of a reword rather than an escape from a warning. `MARKER_LATEST_SEEN_FROZEN` now sits beside the version freeze, so a reword fails in the same edit with the version constant named. version_at_least ACCEPTS GARBAGE (review, new finding). Measured: `"not-a-version" >= "0.2.125"` was TRUE (non-numeric sorts after digits under `sort -V`) and `"0.2.125" >= ""` was TRUE. Both now refuse, loudly, via `is_dotted_version`. Pre-release tags are refused rather than ordered, because `sort -V` puts `0.2.125-rc.1` above `0.2.125` where semver puts it below. `prev_emits_latest_seen` deliberately goes the OTHER way on malformed input: it ARMS Gate B rather than skipping. The two functions answer different questions. "Is a >= b" has no answer for garbage, so the comparator refuses; "should Gate B assert anything" does, and skipping is the silent direction this file exists to remove. Also from the review: 2. `gate_b_arm_case "0.2.124" no`, plus the corrected annotation (two releases skip, not one). Verified vacuity hole: the old cases straddled the gap, so hardcoding the threshold to 0.2.124 inside the decision left the entire suite green. 3. The frozen pin's failure message no longer reads as instructions for the creep edit that disarms Gate B. It now says the fix is almost certainly to put the value back, and points at the reword assertion above it. 4. The operator comment in auto-update-canary.sh regains the reword escape hatch this PR had narrowed away, and no longer says "self-retiring" beside "FROZEN" (review 9). 6. The Rule Lint error text named `check`, one of four accepted forms, so a developer tripping the gate was told to add the wrong thing. It now lists all four and names the two known blind spots. 7. `"ok` tightened to `["']ok[[:space:]]+-`, and `pass` added. Measured on a fixture of eight lines: `echo "okay, the suite is starting"` and a bare `echo "ok"` no longer count, while single-quoted `echo 'ok - ...'` and the `pass` form now do (four more self-tests stop scoring zero). This PR's own diff still scores 2 against origin/main.
Problem
Gate A blocked release v0.2.124 on a binary that was perfectly healthy. The gate was right to block on what it saw; what it saw was its own harness failing.
When it builds the client-API router, the node unconditionally
create_dir_allsstd::env::temp_dir()/freenet/webs(crates/core/src/server/client_api.rs,let contract_web_path =) and panics if that fails.temp_dir()reads onlyTMPDIR, so the path does not follow--data-dir, and the canary isolated HOME, config, data and log dirs but notTMPDIR— so it escaped the sandbox.That directory is vestigial:
grep -rn '"webs"' crates/returns exactly one hit, the creation site. Nothing reads or writes it; web contracts are unpacked elsewhere (default_webapp_cache_dir). Its one surviving effect is the ability to abort startup. Filed separately as #5291 — this PR fixes the canary, not the node.cross-compile.yml:638stages the binary it is about to gate at/tmp/freenet. That is exactly the path the node then tries to create a directory under, socreate_dir_allhitENOTDIRagainst the binary file:The node died (exit 101) before the update task spawned, so Gate A reported "the startup update check never ran" and refused to publish. Correct behaviour on the evidence available to it — the release stayed a draft, the fleet stayed on 0.2.123, and the dev-room alarm fired. But the shipping binary was fine.
Two things made this land at the worst moment: the harness had never run in real CI before it gated a real release, and local validation (4/4 green pre-merge) ran the binary from
target/release/freenet, where the collision cannot occur.Approach
Set
TMPDIRto a directory inside the canary's own workdir, scoped to the subshell that launches the node, so only the throwaway node is affected.Correction to an earlier revision of this description. It claimed "relocating the binary alone is not sufficient." That was wrong, and review disproved it by testing: with the pre-fix script, a clean scratch
TMPDIRcontaining nofreenetentry, and the binary staged outside it, the same artifact returnsrc=0. Relocating the binary incross-compile.ymlalone would have unblocked v0.2.124. The contrary evidence came from a host-specific condition over-generalised —/tmp/freeneton the test host is a directory owned by another user, so that run hit EACCES rather than the file collision.Isolation is still the better fix, for the narrower and true reason: relocation cures the CI symptom while leaving the canary non-self-contained, so on any machine where
$TMPDIR/freenetis owned by someone else — a shared host, or one already running a node under a service account — the node still panics. Isolation removes the dependency on what happens to be in the ambient temp dir.It also closes the second half of this file's own isolation claim. The ports half was corrected earlier (the runs bind real ports and collide with a running node); this is the other half — until now two nodes on one machine shared
/tmp/freenet/webs, so the header's "keeps its files to its own temp directory" was false.Testing
Verified against the shipped v0.2.124 artifact, downloaded from the draft release (
0.2.124 (6d39f89b9207)), staged at$TMPDIR/freenetto reproduce the exact collision. CI staged it at/tmp/freenetliterally; that path is an unwritable directory on the test host, soTMPDIRwas pointed at a scratch dir with the binary inside it — same mechanism, reproducible without root.rc=0—OK: the node compared against '0.2.123', which matches GitHub's latest release.andOK: startup update check ran to completion and parsed GitHub's response.rc=1,node exited with code 101,panicked at client_api.rs:256,Failed to create contract web directory at …/freenet/webs: Not a directory (os error 20)The control reproduces the production failure exactly, so this is a before/after with a working negative case rather than an assertion that it should help.
A separate run — same artifact, non-colliding binary path, default
TMPDIR, on a host where/tmp/freenetis another user's directory — also failed. That is the EACCES case, not the ENOTDIR one, and it is host-specific; see the correction above. Both are startup panics reported by Gate A as "the startup update check never ran", which is the hard-fail branch, not the INDETERMINATE/UNVERIFIED branch the script keeps carefully distinct.Regression coverage. The four suites were all green while this bug shipped, because they drive a bash fake node with no web directory. Added: a behavioural lifecycle case in which the fake node is the regular file staged at
$TMPDIR/freenet, so the ENOTDIR comes from the kernel rather than being simulated — red before the fix, green after. A source-scrape pin is included too, but the behavioural case is the load-bearing one: a pin cannot distinguishexport TMPDIR="$work/tmp"fromexport TMPDIR=/tmp. A companion case asserts the full env handed to the node, closing a sibling gap where deletingexport HOME=also left every suite green.Not fixed here, deliberately
$TMPDIR/freenet/websregardless of--data-dir. That is arguably a product bug — a node's files escaping its configured data dir — but changing it affects every node, not just the canary, and belongs in its own PR with its own review. Filed separately.cargo publish. v0.2.124's crates are on crates.io while its GitHub release is still an unpublished draft, because the gate sits after the one irreversible step. A Gate A block should cost a re-run, not a version number. Filed separately; it needs an ordering change inrelease.yml/cross-compile.yml, not a one-line patch.main— with mechanically identical staging, or it proves nothing — would surface harness faults where they are cheap. Filed separately.Refs #5222, #5236.
[AI-assisted - Claude]