Keep Harper booting when its storage volume is full or at quota

[kriszyp]

Harper crash-looped at startup on a storage-exhausted volume: every boot rewrites harper-config.yaml and the env-config state snapshot when a HARPER_*_CONFIG variable is set, the sibling temp file was refused with EDQUOT, and the error escaped initConfig — so the container restarted, the same write failed again, and nothing inside it could free space because nothing could start. Boot-path persistence of derived config is now best-effort: the effective configuration is already merged and validated in memory, so ENOSPC/EDQUOT is logged and startup continues on it, while install and user-requested writes (set_configuration and friends) still persist or throw. A refused log append also degrades to stdout instead of throwing, which otherwise made every log statement on a full volume a crash point — including the one reporting the storage problem.

Necessary but, on its own, not sufficient: this gets configuration initialization and logging through an exhausted volume. A full harper start there still needs writable space for the storage engine, so "one process starts and the scheduled cleanup runs" is not proven by this change. #846 remains the reason a hosted node reaches this state.

Fixes #847.

For the human reviewer

1. Boot continues on the in-memory config instead of failing fast. The alternative is to keep aborting and require operator intervention, which is what produced the outage. The trade is a guaranteed-consistent on-disk config for availability — everything below is a consequence of admitting that divergence. Reversible in one function (persistConfigDuringBoot), but if you disagree with the premise, the rest of the PR follows from it.

2. Only ENOSPC/EDQUOT degrade; EROFS, EACCES and EIO still abort boot. A read-only remount produces the identical un-breakable restart loop, so this leaves the same field outage reachable by another route. Narrow now on purpose, trivially widened later by adding codes to one set — I'd rather not widen it on speculation.

3. The two artifacts move together, or not at all — and the confirmed record is never at the mercy of a write. The env-config state records the file's pre-env values and is the only copy of them, so the new state is staged in a sidecar, the config file is written, and the sidecar is then promoted with a rename. A rename needs no free space, which is the whole point: a refused staging write leaves the config file alone, a refused config write drops the sidecar, and in both cases every recorded original survives untouched. A sidecar left by an interrupted commit is cleared on the next boot, which also skips drift detection for that boot — it cannot distinguish a manual user edit from the write that was in flight, and guessing wrong hands those paths to user permanently. A boot that re-derives the same state writes nothing at all. This is the subtlest part of the PR and where I'd look hardest; two earlier shapes of it (snapshot-last, then a pending marker written over the confirmed record) each had their own data-loss path, both caught in review.

4. ensureConfigKeysPresent reports nothing it could not persist, so on an exhausted volume a newly-introduced built-in (waf, secretCustody) stays dormant for that boot and the startup log says so. The rejected alternative was activating it in the main thread's memory: worker threads re-read the config from disk and never run the backfill, so that gives a boot where the log claims WAF is on and every request-handling worker is without it. Dormant-and-honest beats active-in-name-only, but it is a security-component policy call and yours to make.

5. Install still fails hard on a full volume. There is no last-known-good config to fall back on and no process to keep alive, so its snapshot write stays mandatory. Worth a line in the release note: boot tolerates a full volume, install does not.

6. skipIfUnchanged is opt-in inside atomicWriteFile. Skipping means no mtime bump and therefore no watcher event, so it is a watcher-visible semantic that every call site can now switch. Only the two artifacts that are re-derived identically each boot pass it. A separate writeFileIfChanged was the alternative; cheap to move, awkward to un-ship once callers depend on the flag.

7. Log fallbacks bypass the stdio guard, and mirror to stdout unconditionally. installStdioGuard routes console output back into the file logger when logging.file and logging.console are both on, so the obvious console.log fallback recurses until the stack blows; everything now goes through one non-throwing writeToStdioDirectly. Two consequences worth your ruling: the no-descriptor branch used to drop every entry after the first and now emits them, and the mirror does not consult logging.console — so a deployment that deliberately keeps log content out of stdout will see it there while the volume is full. I took diagnostics over silence, but gating the mirror on the console setting is a one-line change if you'd rather. A refused append also now sets a 5s cooldown, so a persistently full volume costs one failed syscall every few seconds instead of one per log line on the request path.

8. The deeper simplification I did not take. All of this subtlety exists because boot rewrites the operator's own harper-config.yaml with env-derived values, which is what makes a separate record of the pre-env values necessary in the first place. Stop doing that — keep the user's file untouched and hold the env overlay separately — and the two-artifact ordering problem disappears entirely rather than being solved. I did not do it here: it changes what operators see when they read the config file, changes restoration semantics, and needs a migration for existing state/YAML pairs, none of which belongs in a P1 availability fix. Worth its own issue if you agree.

Verification

Unit — unitTests/config/storageExhaustion.test.js (new, node:assert against the real modules per AGENTS.md): errno classification including Linux's unmapped -122, the swallow/rethrow boundary, skipIfUnchanged behaviour and the default path, temp-file cleanup when the write itself fails, the two-phase snapshot contract (including that an unchanged boot writes nothing and that a still-pending snapshot is discarded on load), and the rollback primitive. The log fallback is covered for real rather than with stubs: a child process logging to /dev/full — which refuses every write — must survive, land its entry on stdout, and not recurse through the stdio guard. It skips where /dev/full does not exist (macOS) and runs on CI.

npx mocha "unitTests/config/**/*.js" "unitTests/utility/logging/**/*.js"   →  401 passing, 14 pending

Fails on base — with config/ and utility/logging/ restored from origin/main and dist force-rebuilt, the same command gives 374 passing, 14 failing: 13 of the 14 new cases plus the runtime-env-var file's before hook, which cannot stub a function that does not exist on base. (The fourteenth — temp cleanup on an ENOENT write — passes on base, which never creates the temp file in that path.) Restored and rebuilt: green again.

End-to-end, on a genuinely full volume — ~/dev/scripts/harper-full-volume-boot-check.sh: a 20MB HFS+ image with an existing config file, an existing log file and the standard directories, filled to zero free bytes, then initConfig with a HARPER_SET_CONFIG layer that must be merged, plus log lines larger than the volume's remaining slack.

free bytes: 0
RESULT append-premise=ENOSPC                      # the log append genuinely cannot land
Harper cannot write to its log file: ENOSPC: no space left on device, write
2026-08-20T13:28:07.154Z [main/0] [error]: unwritable line 0 …   # via nativeStdWrite, no recursion
RESULT initConfig-returned=true
RESULT effective-http-port=8123                   # env layer applied from memory
RESULT config-file-unchanged=true
RESULT leftover-temp-files=none
RESULT env-state-file-present=false               # no snapshot left ahead of the file
exit=0

Same rig on origin/main exits non-zero at initConfig with ENOSPC … backup/.harper-config-state.json.<pid>.<tid>.<hex>.tmp — the reported wedge.

Not covered by the repeatable suite: the coordinator's failure sequences (which write is refused in which order) and the log-append degradation. Both need a filesystem that refuses writes, which no unit test can arrange portably, and AGENTS.md rules out reaching for stubs to fake it — hence the live rig above. test:unit:main and test:integration:all were not run locally (the shared RocksDB system-database lock on this machine fails them before any test executes); CI covers them.

Outside-review coverage: nine completed rounds — Codex + Gemini + Cursor Composer + Harper-domain adjudication at 6ee5ded, Codex + Gemini at 292edf53, Codex + Gemini + Harper-domain adjudication at 6ededadd, and delta rounds (same reviewers, resumed sessions) through this head. Rounds 1–7 each found a real defect in the previous round's fix, all inside the same ~20 lines of commit ordering: snapshot-last lost the operator's originals; snapshot-first left a snapshot ahead of the file; the rollback unlink destroyed originals accumulated over months; the pending marker needed a write an exhausted volume refuses; one fixed sidecar name raced every worker thread; and the pid liveness check exempted processes but not sibling threads, which share a pid. Round 8 was the first with no major, and its two minors (a test assertion that could pass vacuously, and an observer comparing against another process's half-committed pair) are fixed. That history — six data-loss paths and one boot-abort, none of which my own testing found — is the argument for looking hard at item 3 specifically, and the reason this is still draft.

Complexity: complicated

_{Review-Coverage: authored=codex; ran=claude,gemini; declined=cursor-grok,cursor-composer,domain; rounds=4 @ b7180c9}

_{Human-Review-Need: 3 @ b7180c9}

[gemini-code-assist]

Code Review

This pull request introduces robust handling for storage exhaustion (such as disk full or quota exceeded) during the boot process to prevent crash loops. It adds checks for ENOSPC and EDQUOT errors, allows best-effort configuration persistence during boot, and ensures that log write failures on exhausted volumes fall back to direct stdio writes rather than crashing the process. Additionally, it refactors runtime environment configuration application to support atomic state rollbacks and adds comprehensive unit tests. The review feedback suggests minor TypeScript improvements, such as adding explicit type annotations to function parameters, filtering out potentially undefined platform-specific error numbers to avoid NaN values in a Set, and safely wrapping property accesses on caught errors in try/catch blocks to prevent unexpected crashes.

[kriszyp]

Bot pass triaged: the Windows/EDQUOT one was real and is fixed in 64fe91e — os.constants.errno.EDQUOT is absent on platforms without quotas and -undefined is NaN, so the set held a value matching nothing. The three type-annotation suggestions I declined: configUtils.ts types its module-level helpers loosely throughout (atomicWriteFile(filePath, content, …) and neighbours are untyped), and annotating only the functions this PR adds would make the new code the odd one out rather than the file consistent. Happy to type them if we'd rather start converging the file.

[claude]

Reviewed; no blockers found.

[kriszyp]

Note on the red Unit Test check, since it isn't this PR:

• log writes that cannot land (#847) failed on 34ca3328 with ReferenceError: testRoot — that one was mine, fixed in ac35021a.
• HNSW greedy routing above layer 0 (ROUTING_EF) (unitTests/resources/vectorIndex.test.js:673) is not. I ran test:unit:resources against origin/main on a Linux box: 3 failing there too (table() randomAccessFields directive among them). Main's Unit Test job has failed on three different tests across its three most recent runs — pp_updated49, reloadMarker (harper-pro#495), and this one — so the suite is flaky right now, and nothing in this diff can reach HNSW neighbour sets.

Config and logging suites pass on both macOS and Linux (368 passing), and the new storage-exhaustion file is 16 passing on Linux where /dev/full exists.

[claude]

What: This if (!isMainThread) return; guard is the crux of 6a7bf1a37 (the fix for workers racing to persist config), but no test exercises the worker-thread leg — every existing test runs applyRuntimeEnvVarConfig on the main thread, so a regression here (e.g. an inverted condition, or a check that stops matching after a future refactor) would silently reintroduce the exact multi-thread write race this commit fixes, with nothing to catch it.

Why it matters: this is a "new runtime-shape branch" per the testing guidance — both legs need coverage, and the worker-thread leg (the actual bug fix) currently has none.

Suggested fix: unitTests/config/configUtils-runtimeEnvVars.test.js already uses rewire (configUtils.__get__(...)), so the worker branch can be pinned without adding a new stubbing mechanism: configUtils.__set__('isMainThread', false) for a case asserting saveEnvConfigStateStub/fsWriteFileSyncStub/fsRenameSyncStub are never called when not on the main thread.

[claude]

What: This if (!isMainThread) return; guard is the crux of 6a7bf1a37 (the fix for workers racing to persist config), but no test exercises the worker-thread leg — every existing test runs applyRuntimeEnvVarConfig on the main thread, so a regression here (e.g. an inverted condition, or a check that stops matching after a future refactor) would silently reintroduce the exact multi-thread write race this commit fixes, with nothing to catch it.

Why it matters: this is a "new runtime-shape branch" per the testing guidance — both legs need coverage, and the worker-thread leg (the actual bug fix) currently has none.

Suggested fix: unitTests/config/configUtils-runtimeEnvVars.test.js already uses rewire (configUtils.__get__(...)), so the worker branch can be pinned without adding a new stubbing mechanism: configUtils.__set__('isMainThread', false) for a case asserting saveEnvConfigStateStub/fsWriteFileSyncStub/fsRenameSyncStub are never called when not on the main thread.

[claude]

Suggestion (non-blocking): the new interrupted = true on a live-foreign-sidecar path (fixed in bb4ad1802) has no test asserting its actual consequence — that detectConfigDrift is skipped and state.sources[path] is not reassigned to 'user' when a live foreign sidecar coincides with a file/state mismatch. unitTests/config/storageExhaustion.test.js's "leaves another live process's staged commit alone" only asserts the sidecar file itself survives. Worth extending that test (or adding one) to also drift the file relative to the recorded state and assert the path's source is untouched — that's the exact regression this commit's own message describes fixing, and nothing currently pins it.

[kriszyp]

Second CI-flake data point, since the Unit Test check has now gone red three times on this branch for three different tests, none of them in this diff:

Run	Job	Failing test
64fe91e5, ac35021a	v26	HNSW greedy routing above layer 0 (vectorIndex.test.js:673)
871c51d6	v22	put with immediate expiration on sourced table should expire immediately (apiTests/cache-test.mjs:73)
871c51d6	v24	HNSW greedy routing above layer 0 again

origin/main fails too — I ran test:unit:resources against it on Linux and got 3 failures (table() randomAccessFields directive among them), and main's own Unit Test job has failed on pp_updated49 and reloadMarker (harper-pro#495) in its recent runs. The suite currently goes green or red by luck, which makes "CI green" a stochastic gate for anything merging right now — worth someone's attention independently of this PR.

The one red that was mine (ReferenceError: testRoot) was fixed in ac35021a. This PR's own tests — 21 in storageExhaustion.test.js, including the /dev/full and worker-thread legs — have passed on every run where the job reached them, and 373 pass on Linux at head.

[kriszyp]

Last CI note, so nobody re-derives it: Integration Tests 1/6 (Windows, Node.js v24) is a permanent red on main, not a flake and not this PR. It has failed on all four recent main runs including b51e92af (this branch's base) and on every sha here, with the shard's requests all returning ECONNREFUSED — the server never finishes coming up. That matches #2243 (Windows RocksDB seed takes 83–238s versus ~2s on Linux, so the seed outruns the readiness window).

I checked rather than assumed, since this change does touch startup; the base-branch history settles it.

Net: on this repo today, "all checks green" is unreachable for any PR that runs the Windows shards, and the unit suite is separately stochastic (previous comment). The signals that are meaningful for this PR — config and logging suites, all Linux/Bun/uWS integration shards, the Next.js adapter, and unit on v22/v24/v26 — are green, with 373 passing on Linux at head.