Deflake the unit-test cluster: seed HNSW routing, signal-based waits, quarantine two unreproduced hangs

[kriszyp]

Deflakes the cluster of unit-test failures that took the Unit Test workflow red on 7 of its last 30 main runs (four distinct tests across 2026-08-21 alone), plus the Windows-only risk-query integration failure. Three tests are fixed with demonstrated root causes; two could not be root-caused within this round and are quarantined, each behind a filed, prioritized issue under epic [Epic] CI test flakiness (#1655). No product behavior changes beyond a test seam (details below).

What changed

Fixed (root cause demonstrated):

1. HNSW greedy routing (unitTests/resources/vectorIndex.test.js, resources/indexes/HierarchicalNavigableSmallWorld.ts) — graph levels were drawn from unseeded Math.random(), and greedy-vs-full-ef result equality is only statistically true across random graphs: ~2-3% of random 600-node graphs legitimately route to a different layer-0 entry point and displace the top-10 tail. Added a random property to HierarchicalNavigableSmallWorld (test seam, defaults to Math.random) and the test now pins the graph with a seeded PRNG. Product-side this also clamps the entry-point-creation level to MAX_LEVEL, matching the other assignment site — previously -Math.log(random()) at that one site was unclamped, and a random() returning exactly 0 (possible for both Math.random and any test PRNG) would produce level = Infinity and an infinite loop in the node-initialization for loop; a high finite draw persists an entry point dozens of empty layers above the graph for the life of the index. And it is not just a lottery ticket: with a schema-configured mL (e.g. mL: 5, which the constructor accepts), 13.5% of first-node draws exceed MAX_LEVEL — every such index got a permanently over-tall entry point while later nodes were clamped (review-credit: the harper-domain leg quantified this). The clamp now has its own regression test (HNSW entry-point level clamp), driven with a finite pathological draw so a clamp regression fails fast instead of wedging the runner. Also recorded the statistical nature of greedy-equals-full in DESIGN.md.
2. Txn expiration (unitTests/resources/txn-tracking.test.js) — the test raced a fixed 50ms window (Promise.race([delay(50), result])) against a 40ms sleep + two real DB operations + an expiry that structurally needs two ~20ms monitor ticks. Ten milliseconds of slack loses on a contended runner. Now waits on the actual signals (waitFor), and proves the transaction left trackedTxns while the slow get() was still pending — so removal-by-expiry can't be confused with removal-by-completion.
3. Subscription replay (unitTests/resources/subscriptionReplay.test.js) — !omitCurrent: updates to passed keys arrive via queue still used collect()'s quiet-period timer, the exact race its two sibling tests were already migrated off (their in-file comments document it). Under contention the subscription can go quiet longer than the window while queued updates are in flight, so the final-value assertion reads stale values. Now attaches a listener and waitFors every key's final value, with the timeout falling through to the per-key asserts for precise failure messages. The pre-push review then caught five more tests in the file still carrying the same quiet-period/fixed-delay pattern — including two duplicate-detection tests whose assertions could pass vacuously when the quiet window expired before in-flight deliveries arrived. All five are converted, so the file's collect() quiet-window idiom is fully retired for in-flight-write tests (it remains only where a quiet window is the semantically right tool, e.g. asserting no events arrive).

Quarantined (issue filed, skip links to it):

4. risk-query on Windows (integrationTests/components/risk-query.test.ts) — #2273: deploy_component (restart:true) hung server-side after npm pack on Windows CI — the instance log goes silent and the client fetch dies on undici's 300s headers timeout, cancelling all 9 children at 319s. This is a hang, not runner slowness (npm pack itself finished in <1s; then 5+ minutes of zero log output), so a timeout bump would not help — the suite is skipped on win32 until the deploy hang is fixed. The readiness poll also now carries AbortSignal.timeout(5s) per probe so one hung fetch can't consume the whole budget. Suite verified green on Linux.
5. MQTT non-clean session (unitTests/apiTests/mqtt-test.mjs) — #2274: silent 20s mocha timeout on CI (lmdb pass, Node 26), zero server-side log output, and the hang is provably in one of the steps with no individual timeout (the test's own bounded 15s wait never fired). Not reproduced in 30 contended local full-file runs. Skipped on the lmdb pass only — where the hang was observed — so the rocksdb pass keeps this durable-session coverage (this narrowing was a pre-push review finding; the first cut skipped it everywhere). Hypotheses and a reinstatement path are in the issue. Most of this file's diff is indentation from wrapping the test in the conditional skip; the body is unchanged.

Found but not touched (out of scope): the repro loops surfaced a sixth flaky test, MQTT subscribe to retained record with patch operations — ~10% failure rate under 2-core contention, duplicate retained/live delivery asserted as an uncaught exception. Filed as #2275 with mechanism and repro instructions.

For the human reviewer

• The one product-source file touched is HierarchicalNavigableSmallWorld.ts, and the judgment call is the random instance property as a test seam plus the MAX_LEVEL clamp on the entry-point site. The seam is inert in production (this.random defaults to Math.random; both call sites previously called Math.random() directly). The clamp changes behavior only for random() === 0 or graphs demanding level > 10, i.e. probability ~2^-32 per insert — but the unclamped site could infinite-loop, so it is a genuine (if theoretical) hardening, called out here rather than buried.
• The txn test's new "expired-before-settled" outcome check is deliberately stricter than the old assertion: it fails with a distinct message if the slow get() completes before expiry is observed, which would indicate the timing assumption broke in a new way rather than passing vacuously.
• The subscription-replay fix swallows the waitFor timeout (.catch(() => {})) so the detailed per-key asserts report exactly which key went stale — a deliberate trade of one generic error for precise diagnostics.
• Windows quarantine means Windows loses risk-query coverage entirely until Windows CI: deploy_component (restart:true) hangs after npm pack — risk-query integration suite cancelled at 319s #2273 is fixed; that felt better than a suite that red-flags the whole Integration workflow on a 300s hang, but it is a coverage regression on that platform.
• Review decision ledger, left as author calls for you to override: pin-the-graph vs a tolerance assertion on random graphs (pinned — the statistical property belongs to the DESIGN.md benchmark sweeps, not a unit assertion); the random seam as a public field vs threading through schema options (field — options are persisted schema config, a function doesn't round-trip); MAX_LEVEL clamp bundled here vs its own PR (bundled — one line, named in the commit body, now with its own regression test); quarantine now vs holding for root cause (quarantine — greening main is the task).
• The txn test now also asserts (rocksdb) that the expiry abort surfaces to the get() caller, and drains the slow operation so its 500ms tail can't bleed into the next describe — a round-1 review finding.

Verification

• HNSW: pre-fix 1/40 sequential failures; post-fix 0/120 (4-way parallel). Intermediate result worth knowing: seeding alone appeared to still fail 5/100 — because unit tests load dist/, and the seam wasn't built yet; after npm run build, graph state (level histograms) is byte-identical across runs. Diagnostics also confirmed layer-0 connectivity and identical failure output pre-fix, ruling out ef auto-scale flapping.
• Txn expiration: pre-fix 2/80 failures with 8 workers pinned to 2 cores (same assertion family as CI); post-fix 0/160 under the identical harness.
• Subscription replay: pre-fix not reproducible locally (0/120 at 2-core pinning, 0/72 single-core) — the fix is mechanism-based, matching the two sibling tests that were previously converted off collect() for exactly this race; post-fix 0/72 single-core.
• MQTT: 30 contended full-file runs under lmdb against a CI-style fresh install (harper.js install, DEFAULTS_MODE=dev): target test 0 failures (hence quarantine rather than a speculative fix), sibling patch-operations test 3 failures (filed Flaky unit test: MQTT 'subscribe to retained record with patch operations' — duplicate retained/live delivery fails as uncaught assertion (~10% under contention) #2275).
• Gates: npm run test:unit:main, test:unit:resources (rocksdb and lmdb), full apiTests pass (rocksdb, sandboxed fresh install), integrationTests/components/risk-query.test.ts (9/9 on Linux), lint:required, prettier. test:integration:all not run in full: this branch touches a single integration file (run individually above) and the Integration workflow on main is currently failing for unrelated reasons (17 of its last 30 main runs), so a full local run has no usable baseline.

Refs [Epic] CI test flakiness (#1655)

Complexity: low — test-only changes plus one inert test seam and a theoretical-edge clamp in HNSW level assignment.

— Claude Fable 5

🤖 Generated with Claude Code

https://claude.ai/code/session_0142195pzooHeNejheNfsPDZ

_{Review-Coverage: authored=claude; ran=codex; blocked=gemini(exit-1); declined=cursor-grok,cursor-composer,domain; rounds=4 @ cf75953}

_{Human-Review-Need: 4 @ cf75953}

[gemini-code-assist]

Code Review

This pull request addresses test flakiness and environment-specific hangs by introducing condition-based polling (using a waitFor helper), adding abort timeouts to fetch calls, and quarantining problematic tests on Windows and LMDB. It also introduces a random function seam in the HNSW index to allow seeding the PRNG for deterministic graph generation in tests, and clamps the entry-point level to MAX_LEVEL. The reviewer feedback recommends replacing the remaining fixed delays in subscription replay tests with a sentinel record pattern to ensure all prior events are fully processed before assertions, which aligns with best practices for testing ordered streams.

[claude]

Reviewed; no blockers found. One non-blocking suggestion posted inline.