Fix multi-worker lost counter increments and stale reads when a resequenced write reuses a version

[kriszyp]

Fixes the intermittent QA-431(5): N window(s) with lost counts under multi-worker stress failure in integrationTests/resources/ttl-rate-limiter-concurrent.test.ts — investigated and reproduced as a genuine product defect, not a test timing assumption.

Root cause

A resequenced (out-of-order CRDT) write stores its merged record under the version it merged onto rather than advancing it — the version is the max applied update timestamp and must stay that way for cross-node convergence — so one version can identify two different stored values. Every freshness decision keyed on version equality is then wrong:

1. Lost writes (the QA-431 failure). The commit path read its fold base through the record cache's version vouch (first attempt via the resource-phase read, coordinated-retry reloads via getEntry(key, {transaction})). The vouch answers "is this the latest committed version?" — the wrong question for a snapshot read, and wrong outright for a reused version — so an addTo/patch folded onto the stale pre-merge value and durably overwrote the concurrent increment it had merged over. A patch stores a full record derived from its base, so a stale base also silently resurrects old values of fields the patch didn't mention; a put's stale base corrupts index diffing, blob retention, and residency the same way.
2. Indefinitely stale reads. On a VerificationTable miss, the native layer re-confirms freshness against the version stored in the record itself and republishes it ("soft VT miss" in rocksdb-js). For a reused version this re-vouches the stale holder's cache forever once every worker is warm — observed live as GETs pinned at 41–49/50 until TTL expiry while the store held exactly 50.

Fix

• recordUpdater marks a record stored under a non-advancing version with a VERSION_REUSED metadata flag (versionIsReused() is the single definition both the flag and the park derive from); reads of flagged records are never cached and never vouched.
• New uncachedRead option on PrimaryRocksDatabase.getEntry: a plain transaction-snapshot read — no vouch trust, no VT seeding, no cache publish. Every record-write kind that derives stored state from its base (update, put, delete, invalidate, relocate — marked with an explicit reloadCommitBase flag) reloads that base at commit through it; bulk copy-apply rows and crash-recovery replays keep their pre-read base (one read per row, as before — their convergence contract is the post-copy/replay pass). previousResidency is derived from the reloaded base. Without the delete-path reload, a delete diffing indices from a stale base left phantom index entries for the deleted record.
• The writer parks an unvouchable sentinel in the VerificationTable slot on the commit success path — reader-side parking cannot work because with all workers warm no reader ever decodes the flagged record, and the native soft-miss re-confirm republishes the reused version over a reader-parked sentinel. The park is verified, retried on a bounded backoff while a concurrent write's intent holds the slot, and a final refusal is resolved against the stored head — warn only when the head is still flagged (an aborted competitor would otherwise leave a silent stale-read hole); tie-timestamp delete tombstones mark for parking too. The park path is fully contained so a closing-store throw after durability cannot skip the commit's cleanup.
• Warm cached reads consult the sentinel before trusting version equality.

Verification

• Red on unpatched main: amplified QA-431(5)-shaped repro (same fixture, in-burst GET pressure, taskset to 4 CPUs to emulate CI runners) lost increments in 3–15 windows per 900, with a per-vouch audit proving the losses durable (store short) — plus stale-read windows where the store was correct but every GET served a vouched stale value. The new unit tests fail on unpatched main at the invariant assertions.
• Green with the fix: 900/900 windows exact across 3 amplified runs plus 300/300 across the review rounds, zero stale vouches in the audit, and 8/8 runs of the unmodified QA-431 test (same CPU pinning).
• The repro is committed as integrationTests/resources/ttl-rate-limiter-convergence.test.ts — it classifies each short window as converged-late (read-timing) vs stuck-short (durable loss), so a timing race is never mistaken for a lost write.
• Gates: test:unit:resources 1674 pass / 3 fail and LMDB resources 1352 pass / 2 fail — every failure reproduced on pristine main (pre-existing: randomAccessFields ×2, replayStructures ×1). test:unit:main is unrunnable on the fleet box for any branch (startup crash opening a stale shared data dir; environmental).

For the human reviewer

• Accepted perf trade (the one open review major): the warm-read sentinel probe costs +152 ns/op — warm getEntry measured 373 ns/op on a pristine-main build vs 525 ns/op with the fix (unitTests/resources/cache-probe.bench.js, 4-CPU-pinned). Noise per HTTP request; visible in tight cached-read loops. The rocksdb-js follow-up (native honors a no-vouch bit in the fixed metadata word at value offset 8) folds this into the existing native crossing and reclaims it.
• Residual race (documented, not closable in harper): a warm reader passing its sentinel check just before the writer parks can be re-vouched once by the native soft-miss re-confirm; heals at the next decoding read. Same rocksdb-js follow-up closes it.
• Durable uncacheability: a flagged key stays uncached/unvouched until its next in-order write rewrites it — a caching cliff for resequencing-heavy tables (e.g. replicated counters with clock skew) that no metric currently surfaces.
• Design choice: compensating for "one version, two values" with unvouchability rather than advancing the version on resequenced folds — advancing would fabricate a timestamp no real update carries and break cross-node CRDT convergence.
• Write kinds outside the reload set: publish/message writers and sourcedFrom cache-fill resolves keep their own conflict semantics and are not marked reloadCommitBase; flagged (unadjudicated) in the final review round — worth a follow-up look rather than widening this PR.

Builds on the abandoned branch fix/record-cache-stale-on-reused-version (its read-side machinery is commit 1, validated and extended here).

Refs #1881 (same version-reuse family, index-scan surface).

Signed: Claude Fable 5 (dispatch agent)

🤖 Generated with Claude Code

_{Review-Coverage: authored=claude; ran=codex,gemini; declined=cursor-grok,cursor-composer,domain; rounds=4 @ 426518f}

_{Human-Review-Need: 4 @ 426518f}

[claude]

Reviewed; no blockers found.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to handle version-reuse cache staleness in RocksDB-backed tables. It prevents the Verification Table (VT) from vouching for reused versions (where a resequenced write keeps an existing version, meaning one version could identify multiple different stored values) by marking such records with a VERSION_REUSED metadata flag, parking a VERSION_UNVOUCHABLE sentinel in the VT, and introducing an uncachedRead option to bypass the cache vouch on critical commit paths. The changes also include comprehensive integration tests, unit tests, and a benchmark script to measure the sentinel probe overhead. There are no review comments provided, so I have no feedback to address.