Bound the exclusive 'update-attributes' lock wait and release it structurally - #2252
Open
kriszyp wants to merge 4 commits into
Open
Bound the exclusive 'update-attributes' lock wait and release it structurally#2252kriszyp wants to merge 4 commits into
kriszyp wants to merge 4 commits into
Conversation
…structurally The exclusive 'update-attributes' lock was acquired with an unbounded busy-spin (resources/databases.ts and resources/Table.ts): no deadline, no yield, and re-marshalling the key string on every attempt. A holder that never released — e.g. a crashed schema operation — pinned the spinning worker's core at 100% forever, with nothing logged. The acquire is now a shared helper that spins hot only briefly, then blocks in bounded Atomics.wait intervals, and throws a diagnosable ServerError after the shared 10s LOCK_TIMEOUT, naming the lock and the table in scope. The lock key is pre-encoded once instead of per attempt. Release is now structural instead of hand-placed: dropTable's locked section runs under withUpdateAttributesLock(), and table() wraps every path that can take the lock in a single try/finally, subsuming the three ad-hoc release calls (only one of which was in a finally — the unguarded paths could leak the lock and wedge every later schema update on the database). Fixes #2251 Co-Authored-By: Claude Fable <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UuKkXTR5Y4fLjdSyoy3VkA
Co-Authored-By: Claude Fable <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UuKkXTR5Y4fLjdSyoy3VkA
…er review Co-Authored-By: Claude Fable <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UuKkXTR5Y4fLjdSyoy3VkA
Contributor
There was a problem hiding this comment.
Code Review
This pull request replaces the unbounded synchronous spin lock for the exclusive update-attributes lock with a synchronous bounded wait mechanism using Atomics.wait and a 10-second timeout to prevent worker cores from pinning at 100% CPU when a lock is leaked. It introduces helper functions acquireUpdateAttributesLock, releaseUpdateAttributesLock, and withUpdateAttributesLock in Table.ts, refactors makeTable and table() to use these helpers, and adds comprehensive unit tests. The reviewer suggested using assert.strictEqual instead of assert.equal in the unit tests to align with the repository style guide.
Contributor
|
Reviewed; no blockers found. |
Co-Authored-By: Claude Fable <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UuKkXTR5Y4fLjdSyoy3VkA
kriszyp
marked this pull request as ready for review
August 21, 2026 17:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2251.
The exclusive
update-attributeslock was acquired at two sites with an unbounded busy-spin (while (!rootStore.tryLock('update-attributes')) {}) — no deadline, no yield, and the key string re-marshalled on every attempt. A holder that never released (e.g. a crashed schema operation) pinned the spinning worker's core at 100% forever, silently; the observed incident burned ~8.8 cores for 16–18 hours. Release was also hand-enforced at each escape path intable()(three ad-hoc calls, only one in afinally), so every early return or throw between acquire and release was a fresh chance to wedge a core — one such leak had already been hand-patched (the deletedcatchdocumented it).What changed:
acquireUpdateAttributesLockinTable.ts: fast-pathtryLock, ~2ms hot spin (brief contention keeps today's latency; a release by another thread is observable without yielding), then boundedAtomics.waitsleeps (1→16ms backoff), thenServerErrorat the shared 10sLOCK_TIMEOUT(the existing constant, now exported) with the lock name, thedatabase.tablein scope, and the elapsed time in the message.table()wraps every path that can acquire in a singletry/finallywith an idempotentreleaseLock()(idempotent because the created-concurrently early return must release before its recursive reload — the lock is not reentrant — and a bare second unlock could release another thread's lock). The three hand-placed releases are subsumed; the release point is unchanged, so lock hold time does not grow.dropTable's locked section now runs under awithUpdateAttributesLock(fn)wrapper.Buffer.from('update-attributes'), byte-identical to ordered-binary's string encoding, verified at runtime by the tests).exclusiveLock()is untouched; the structural release covers it identically to before.DESIGN.md's spin-lock trap paragraph updated to the bounded/structural reality.For the human reviewer
completeInterruptedDropover many column families on saturated I/O) can exceed 10s, making waiters throw where the old spin eventually converged. This is the specified contract for this fix (bound + throw); a holder-liveness stamp is a possible future refinement if spurious timeouts appear under load.IndexRebuildingErrorprecedent) would tell replication/clients to retry transient contention; trivial to change if preferred.table()mutates the liveTablemetadata before the lock is taken (pre-existing shape): a deadline throw can leave this worker's in-memory schema ahead of the catalog until reload. Pre-dates this change; the deadline adds a new trigger. Left as-is deliberately — restructuring the mutation order is a larger change.try/finallyvs extractingtable()'s locked region into thewithUpdateAttributesLockcallback form: the lock is acquired lazily at five conditional points mid-flow, so the callback form would force eager acquisition (a contention regression); the wide guard preserves acquisition semantics exactly.Date.now()): an NTP step backwards extends one DDL wait; repo-wide convention, previously infinite.Verification
unitTests/resources/updateAttributesLock.test.js+ a worker-thread holder): uncontended pass-through, helper deadline throw (parameterized 250ms) with lock/table/elapsed in the message, release-on-callback-throw,table()-level structural release (injected catalog failure mid-schema-update → throw → lock free → subsequent schema op succeeds), a realtable()create against a wedged lock throwingServerErrorat ~10.0s and succeeding after unlock, and cross-thread brief contention (worker holds via the string key, main acquires via the Buffer key — pinning the key-encoding equivalence).test:unit:resources(RocksDB and LMDB engines) and lint/format/typecheck clean; the only local failures (randomAccessFields×2,replayStructures, and atest:unit:mainload crash) reproduce identically on pristine origin/main on this machine (stale local test roots/config) and are unrelated.table()deadline test is the closest executable proof of the production path.Complexity: moderate — concurrency-sensitive locking change on the schema/DDL path, but no data-format or hot-path changes and the critical-section boundaries are preserved.
— Claude Fable
🤖 Generated with Claude Code
Review-Coverage: authored=unknown; ran=none; rounds=1 @ c329356
Human-Review-Need: 4 @ c329356