Skip to content

fix(canary): isolate TMPDIR so Gate A stops blocking healthy releases - #5290

Merged
sanity merged 5 commits into
mainfrom
fix/canary-tmpdir-isolation
Aug 12, 2026
Merged

fix(canary): isolate TMPDIR so Gate A stops blocking healthy releases#5290
sanity merged 5 commits into
mainfrom
fix/canary-tmpdir-isolation

Conversation

@sanity

@sanity sanity commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Problem

Gate A blocked release v0.2.124 on a binary that was perfectly healthy. The gate was right to block on what it saw; what it saw was its own harness failing.

When it builds the client-API router, the node unconditionally create_dir_alls std::env::temp_dir()/freenet/webs (crates/core/src/server/client_api.rs, let contract_web_path =) and panics if that fails. temp_dir() reads only TMPDIR, so the path does not follow --data-dir, and the canary isolated HOME, config, data and log dirs but not TMPDIR — so it escaped the sandbox.

That directory is vestigial: grep -rn '"webs"' crates/ returns exactly one hit, the creation site. Nothing reads or writes it; web contracts are unpacked elsewhere (default_webapp_cache_dir). Its one surviving effect is the ability to abort startup. Filed separately as #5291 — this PR fixes the canary, not the node.

cross-compile.yml:638 stages the binary it is about to gate at /tmp/freenet. That is exactly the path the node then tries to create a directory under, so create_dir_all hit ENOTDIR against the binary file:

thread 'main' panicked at crates/core/src/server/client_api.rs:256:13:
Failed to create contract web directory at /tmp/freenet/webs: Not a directory (os error 20)

The node died (exit 101) before the update task spawned, so Gate A reported "the startup update check never ran" and refused to publish. Correct behaviour on the evidence available to it — the release stayed a draft, the fleet stayed on 0.2.123, and the dev-room alarm fired. But the shipping binary was fine.

Two things made this land at the worst moment: the harness had never run in real CI before it gated a real release, and local validation (4/4 green pre-merge) ran the binary from target/release/freenet, where the collision cannot occur.

Approach

Set TMPDIR to a directory inside the canary's own workdir, scoped to the subshell that launches the node, so only the throwaway node is affected.

Correction to an earlier revision of this description. It claimed "relocating the binary alone is not sufficient." That was wrong, and review disproved it by testing: with the pre-fix script, a clean scratch TMPDIR containing no freenet entry, and the binary staged outside it, the same artifact returns rc=0. Relocating the binary in cross-compile.yml alone would have unblocked v0.2.124. The contrary evidence came from a host-specific condition over-generalised — /tmp/freenet on the test host is a directory owned by another user, so that run hit EACCES rather than the file collision.

Isolation is still the better fix, for the narrower and true reason: relocation cures the CI symptom while leaving the canary non-self-contained, so on any machine where $TMPDIR/freenet is owned by someone else — a shared host, or one already running a node under a service account — the node still panics. Isolation removes the dependency on what happens to be in the ambient temp dir.

It also closes the second half of this file's own isolation claim. The ports half was corrected earlier (the runs bind real ports and collide with a running node); this is the other half — until now two nodes on one machine shared /tmp/freenet/webs, so the header's "keeps its files to its own temp directory" was false.

Testing

Verified against the shipped v0.2.124 artifact, downloaded from the draft release (0.2.124 (6d39f89b9207)), staged at $TMPDIR/freenet to reproduce the exact collision. CI staged it at /tmp/freenet literally; that path is an unwritable directory on the test host, so TMPDIR was pointed at a scratch dir with the binary inside it — same mechanism, reproducible without root.

Case Result
Fixed rc=0OK: the node compared against '0.2.123', which matches GitHub's latest release. and OK: startup update check ran to completion and parsed GitHub's response.
Control (fix reverted, same collision) rc=1, node exited with code 101, panicked at client_api.rs:256, Failed to create contract web directory at …/freenet/webs: Not a directory (os error 20)

The control reproduces the production failure exactly, so this is a before/after with a working negative case rather than an assertion that it should help.

A separate run — same artifact, non-colliding binary path, default TMPDIR, on a host where /tmp/freenet is another user's directory — also failed. That is the EACCES case, not the ENOTDIR one, and it is host-specific; see the correction above. Both are startup panics reported by Gate A as "the startup update check never ran", which is the hard-fail branch, not the INDETERMINATE/UNVERIFIED branch the script keeps carefully distinct.

Regression coverage. The four suites were all green while this bug shipped, because they drive a bash fake node with no web directory. Added: a behavioural lifecycle case in which the fake node is the regular file staged at $TMPDIR/freenet, so the ENOTDIR comes from the kernel rather than being simulated — red before the fix, green after. A source-scrape pin is included too, but the behavioural case is the load-bearing one: a pin cannot distinguish export TMPDIR="$work/tmp" from export TMPDIR=/tmp. A companion case asserts the full env handed to the node, closing a sibling gap where deleting export HOME= also left every suite green.

Not fixed here, deliberately

  • The node hardwiring its web dir to $TMPDIR/freenet/webs regardless of --data-dir. That is arguably a product bug — a node's files escaping its configured data dir — but changing it affects every node, not just the canary, and belongs in its own PR with its own review. Filed separately.
  • Gate A runs downstream of cargo publish. v0.2.124's crates are on crates.io while its GitHub release is still an unpublished draft, because the gate sits after the one irreversible step. A Gate A block should cost a re-run, not a version number. Filed separately; it needs an ordering change in release.yml/cross-compile.yml, not a one-line patch.
  • The canary has never run outside a release. Its first live exercise was the release it was gating. A continuous run on main — with mechanically identical staging, or it proves nothing — would surface harness faults where they are cheap. Filed separately.

Refs #5222, #5236.

[AI-assisted - Claude]

Gate A blocked v0.2.124 on a binary that was perfectly healthy.

The node's contract web directory is `std::env::temp_dir()/freenet/webs`
(client_api.rs:255), hardwired -- it does not follow `--data-dir`. The canary
isolated HOME, config, data and log dirs but not TMPDIR, so that path escaped
its sandbox. `cross-compile.yml` stages the binary it is about to gate at
`/tmp/freenet`, which is exactly the path the node then tries to create a
directory under: `create_dir_all` hit ENOTDIR against the binary FILE, the node
panicked (exit 101) before the update task spawned, and Gate A correctly
reported "the startup update check never ran" -- of a harness fault, not a
product one.

Verified against the SHIPPED v0.2.124 artifact, staged at $TMPDIR/freenet to
reproduce the collision:

  fixed    rc=0  "OK: the node compared against '0.2.123', which matches
                  GitHub's latest release."
  control  rc=1  node exited with code 101, panicked at client_api.rs:256:
                 "Failed to create contract web directory at .../freenet/webs:
                  Not a directory (os error 20)"

Relocating the binary alone is NOT sufficient: with the default TMPDIR the same
artifact still came back UNVERIFIED on a host where /tmp/freenet already exists
as another user's directory. Isolating TMPDIR is what fixes it, and it closes
the second half of the header's isolation claim -- the PORTS half was corrected
earlier; two nodes on one machine also shared /tmp/freenet/webs until now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Rule Review: TMPDIR isolation fix — no blocking issues

Rules checked: git-workflow.md
Files reviewed: 5 (.github/workflows/ci.yml, docs/RELEASING.md, scripts/auto-update-canary.sh, scripts/auto-update-canary_lifecycle_test.sh, scripts/auto-update-canary_test.sh)

This is a fix:-scoped PR. It satisfies the regression-test requirement with two behaviorally-distinct lifecycle test cases (case 9: a real ENOTDIR reproduction via a fake node staged at $TMPDIR/freenet; case 10: an environment-dump assertion that reads the value the node actually received) plus a cross-file source pin, and the commit messages document mutation-testing results (fixed vs. reverted-fix control) confirming the tests fail without the fix. All commit subjects use valid conventional-commit prefixes (fix, test, docs, ci).

Warnings

None.

Info

  • .github/workflows/ci.yml:20 — This PR bundles a change to the CI's own "fix PR must include a regression test" detector (adding the ok helper alongside check) into a PR whose primary subject is TMPDIR isolation. It's well-motivated (the new lifecycle tests use ok/bad and would otherwise be invisible to that gate) and documented inline, but it's a distinct concern from the canary fix itself — worth a reviewer's explicit sign-off that this doesn't read as scope creep under the "one logical change per PR" guidance, even though it's arguably necessary for this same PR's CI to pass correctly.

Rule review against .claude/rules/. WARNING findings block merge.

sanity and others added 4 commits August 12, 2026 14:47
…s, pin it

Review of #5290 found the TMPDIR fix correct but its surrounding claims wrong
in three places, plus the fix itself unpinned.

The path is not what the comments said it was. `contract_web_path`
(`client_api.rs`) is VESTIGIAL: `"webs"` has exactly one occurrence in
`crates/`, the `create_dir_all` itself, and unpacked web contracts live under
`default_webapp_cache_dir` instead. So "two nodes on one machine share
/tmp/freenet/webs" was wrong -- they would share an always-empty, never-read
directory. The real hazard is that the mkdir can FAIL and panic the node at
startup: `$TMPDIR/freenet` being a file (the CI case, ENOTDIR) or another
user's directory without a `webs` child (EACCES). Both comments now say that.

Two escapes from the isolation the header claims:

- Gate B leaked. The `freenet update` subshell exported only HOME, and
  `download_and_install` stages the downloaded tarball in
  `tempfile::tempdir()`, so a release tarball landed in the ambient system
  temp dir. It now scopes TMPDIR too. Safe for the swap: `replace_binary`
  copies to a `.freenet.new.tmp` beside the DESTINATION and renames there, so
  the atomic same-filesystem rename never involves TMPDIR.
- The webapp cache escaped via XDG. `default_webapp_cache_dir` resolves
  through `directories::ProjectDirs`, which reads XDG_CACHE_HOME ahead of
  HOME, and `FREENET_WEBAPP_CACHE_DIR` overrides both; `WebappCache::with_root`
  create_dir_all's it on every boot. On a host with XDG_CACHE_HOME set the
  canary wrote outside its workdir. Fails soft (a warn), so it never blocked a
  release -- scoped now so the header's claim is true rather than nearly true.

Nothing pinned the fix: TMPDIR appeared in none of the four suites, so deleting
`export TMPDIR` left everything green and the fault would resurface only at the
next release, as a blocked release blaming the product. Added a cross-file
source-scrape pin asserting TMPDIR is exported before BOTH launch sites --
before the node's `exec` (which replaces the shell, so a later export never
runs) and before `freenet update`. Mutation-proven in both directions.

Also corrected in the collision comment: it reported "UNVERIFIED" as the
outcome of the collision case, which is the verdict for an exhausted rc=2
retry loop, not for this. The collision produces rc=1 via
`assert_detection_healthy`. It now cites the reverted-fix control instead
(fixed rc=0 vs control rc=1, exit 101, "Not a directory (os error 20)"), the
stronger evidence the comment had omitted.

Minor: "Until v0.2.124" -> "Through v0.2.124" (this ships in 0.2.125), and
docs/RELEASING.md now warns that reproducing locally against
./target/release/freenet is precisely the environment where this class of
fault cannot occur -- which is why local validation went 4/4 green while CI
blocked.

Suites: canary 60 (was 59, +1 pin), lifecycle 9, wiring 14, wait_for_binaries
16. shellcheck -x clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
…rect a false claim

Three things, from the testing lens on #5290.

A REAL test, not a source scrape. Lifecycle case 9 boots the canary against a
fake node that IS the regular file staged at `$TMPDIR/freenet` -- the CI
layout -- and models `client_api.rs`'s hardwired
`create_dir_all(temp_dir()/freenet/webs)`. The ENOTDIR comes from the kernel
refusing to mkdir under a file, not from a fixture printing a panic on cue.
Verified both directions: green here, and against origin/main's script it goes
red with "the canary let its own TMPDIR reach the node ... a HEALTHY release
was blocked (exit 1)". Case 10 is its companion: the fake dumps its own
environment and the test asserts all four exports the canary makes. That closes
a sibling gap -- deleting `export HOME=` or `export FREENET_SUPERVISED=1` also
left all four suites green, because nothing observed them. On the origin/main
control exactly the two TMPDIR assertions fail and the other twelve pass, so
the new cases are not over-broad.

The source-scrape pin stays, demoted in its own comment to what it actually is:
the weaker check. It cannot tell `export TMPDIR="$work/tmp"` from
`export TMPDIR=/tmp`. It earns its place on the GATE B half (which cannot be
exercised without downloading a real release) and on ORDER. Its previous
comment claimed no behavioural test was possible -- that is now false and is
corrected rather than left to rot.

A FALSE claim, removed. The comment said "Relocating the binary alone is NOT
sufficient; the isolation is the fix." Tested and it is wrong: pre-fix script,
a scratch TMPDIR containing no `freenet` entry, binary staged outside it ->
rc=0, "the node compared against '0.2.121'". Relocating the staged binary in
cross-compile.yml WOULD have unblocked v0.2.124. The contrary evidence came
from a host where /tmp/freenet already existed as another user's directory --
the EACCES case generalised into a claim about the ENOTDIR case. The comment
now states the true, narrower reason isolation is still the better fix:
relocation leaves the canary sharing the caller's temp dir, so on a machine
already running a node under another user the gate can still panic on a
/tmp/freenet it does not own. A wrong assertion inside an otherwise precise
block is worse than one on its own -- it borrows the credibility of its
neighbours.

`mkdir -p "$work/tmp"` moved out of the node subshell into the workdir setup in
`run_node_until_check`, and guarded. Gate A did not care (the node's own
create_dir_all builds its parents) but Gate B does: `tempfile::tempdir()`
requires TMPDIR to EXIST and will not create it, so the export added for Gate B
needed the directory to exist before the installer runs. Inside the
backgrounded subshell a failed mkdir was also invisible -- no `set -e`, output
to node.out -- so a broken workdir would have surfaced as a mystery
update-path failure instead of as itself.

Suites: canary 60, lifecycle 14 (was 9), wiring 14, wait_for_binaries 16.
shellcheck -x clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
…ting

Cases 9 and 10 look redundant and are not. Rewriting the export to
`TMPDIR=/tmp` leaves case 9 GREEN on any host where /tmp/freenet is a usable
directory (it is one on nova) -- the node's mkdir then succeeds somewhere
useless instead of failing. Case 10 reads the value the node actually got, so
it fails on that mutation unconditionally. Deleting the export fails both, and
the source scrape is green on both.

Recorded so the next reader does not delete one as a duplicate of the other.
Observed, not reasoned: each claim here is a mutation that was applied and run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
The `fix:`-must-add-a-test rule already knew that shell-tooling fixes are
tested by `*_test.sh` self-tests rather than Rust `#[test]`s -- its own comment
says the shell branch exists "so such fixes aren't forced to claim
test-exempt". But it counted only `check ` assertions, and
`auto-update-canary_lifecycle_test.sh` reports through `ok`/`bad`.

So it scored 0 against this PR, which adds a behavioural lifecycle case that
reproduces the fault it fixes and goes red without it (mutation-verified: drop
`export TMPDIR` and the suite fails with two assertions naming #5290). The rule
demanded `test-exempt` for a PR carrying exactly the test the rule wants.

That is the same shape as the bug this PR fixes and as several found reviewing
it today: a gate that cannot see its own subject, reporting on something other
than the thing under test.

Counting `ok` alongside `check`: 5 added assertions detected here, 0 under the
old pattern.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
@sanity
sanity added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit c893b06 Aug 12, 2026
18 checks passed
@sanity
sanity deleted the fix/canary-tmpdir-isolation branch August 12, 2026 21:40
sanity added a commit that referenced this pull request Aug 12, 2026
…lly ships it

The constant names the first release whose binary emits the observed-latest
marker, and Gate B skips its positive equality check for anything below it. It
said 0.2.124 -- but 0.2.124 was never PUBLISHED: Gate A blocked it on the TMPDIR
harness bug (#5290), and a draft release is invisible to `/releases/latest`. So
no release a node can actually reach emits that line until 0.2.125.

Left at 0.2.124, Gate B would have demanded the marker from 0.2.123 -- a binary
never built to log it -- and gone red on a healthy transition, on the very gate
we are relying on to catch a repeat of the 0.2.120/121 auto-update break.

Caught by the constant's own pin (added in #5290) when the bump moved the crate
to 0.2.125. Working as designed: the pin exists because raising the constant
silently disarms Gate B, and it is anchored to the crate version precisely so a
stale value cannot survive a release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012dyjTKM35KGZXjE7jmDTsX
sanity added a commit that referenced this pull request Aug 13, 2026
…gether

Addresses the Full-tier review on #5303. The relation pin this PR added was
wrong in the same way `main`'s was, with the sign flipped, and the review found
a silent failure neither pin could see.

DELETE THE RELATION PIN (review 1b). Two have now been tried and both were
correct in one phase and wrong in another. `constant >= crate` (#5290) rested on
"no published release emits the marker yet" and detonated the moment 0.2.125
published. `constant <= crate` is right after publication and wrong during a
marker REWORD: a marker reworded on main first ships in crate+1, so the correct
constant is crate+1, which the pin calls a failure -- while the value that makes
it green makes Gate B demand the new text from a published binary emitting the
old one, i.e. a post-publish red canary and a Matrix alarm blaming the node.
A `<= crate+1` variant fires on a reword too; the reviewer who proposed it
withdrew it after testing.

The generalisation, recorded in the comment so a third attempt is not made: the
constant tracks RELEASE HISTORY and the crate version tracks THIS TREE, and no
fixed relation between two quantities on different clocks holds across both
publication and reword. The freeze is phase-independent and covers strictly
more -- the relation only ever caught "strictly above crate version", while the
freeze catches every wrong value including the empty string and the most likely
wrong one. It also cannot be tripped by `version.workspace = true`, which broke
the relation pin's Cargo.toml read.

FREEZE THE PAIR (review 1a, the highest-value change). Reword
MARKER_LATEST_SEEN everywhere a developer must touch it and leave the constant
alone, and the whole suite passes -- verified, zero red. The source pin
interpolates `$MARKER_LATEST_SEEN`, so it follows the rename by construction,
and nothing else looks at the marker. A stale constant is the DEFAULT outcome of
a reword rather than an escape from a warning. `MARKER_LATEST_SEEN_FROZEN` now
sits beside the version freeze, so a reword fails in the same edit with the
version constant named.

version_at_least ACCEPTS GARBAGE (review, new finding). Measured: `"not-a-version"
>= "0.2.125"` was TRUE (non-numeric sorts after digits under `sort -V`) and
`"0.2.125" >= ""` was TRUE. Both now refuse, loudly, via `is_dotted_version`.
Pre-release tags are refused rather than ordered, because `sort -V` puts
`0.2.125-rc.1` above `0.2.125` where semver puts it below.

`prev_emits_latest_seen` deliberately goes the OTHER way on malformed input: it
ARMS Gate B rather than skipping. The two functions answer different questions.
"Is a >= b" has no answer for garbage, so the comparator refuses; "should Gate B
assert anything" does, and skipping is the silent direction this file exists to
remove.

Also from the review:
  2. `gate_b_arm_case "0.2.124" no`, plus the corrected annotation (two releases
     skip, not one). Verified vacuity hole: the old cases straddled the gap, so
     hardcoding the threshold to 0.2.124 inside the decision left the entire
     suite green.
  3. The frozen pin's failure message no longer reads as instructions for the
     creep edit that disarms Gate B. It now says the fix is almost certainly to
     put the value back, and points at the reword assertion above it.
  4. The operator comment in auto-update-canary.sh regains the reword escape
     hatch this PR had narrowed away, and no longer says "self-retiring" beside
     "FROZEN" (review 9).
  6. The Rule Lint error text named `check`, one of four accepted forms, so a
     developer tripping the gate was told to add the wrong thing. It now lists
     all four and names the two known blind spots.
  7. `"ok` tightened to `["']ok[[:space:]]+-`, and `pass` added. Measured on a
     fixture of eight lines: `echo "okay, the suite is starting"` and a bare
     `echo "ok"` no longer count, while single-quoted `echo 'ok - ...'` and the
     `pass` form now do (four more self-tests stop scoring zero). This PR's own
     diff still scores 2 against origin/main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant