Defect
waitForLocalMutationToSettle (added in c213602, shipped in v0.10.47) is written to bound its wait at 2 seconds: it polls every 2ms, honours ctx.Done(), and carries a 2s timer.
The timer does not bound the function. Each poll iteration calls a blocking s.mu.Lock() outside the select. A permanently wedged mutex holder therefore blocks the call past its own deadline — the timer is only consulted between calls, never during one.
This is the same defect class we already hit once in this incident: a deadline checked between calls does not bound a call. A 5-minute timeout sat unreachable through a 62-minute hang for exactly this reason.
Severity and why this is a follow-up, not a blocker
Not currently reachable on the failing path. The one-shot initial seed never calls this function — PushLocalAndFlushOnce flushes with unlockDuringUpload=false, and only the realtime watcher fast path sets it true. On timeout the event is simply not checkpointed; the WebSocket loop reconnects from the previous cursor and replays it, so the failure mode is delay rather than silent loss.
So: real, bounded in blast radius today, and worth fixing before something else starts calling it.
Requested fix
Acquire the lock through a bounded path so the deadline actually governs the call — e.g. a TryLock loop driven by the same select that watches the timer and ctx.Done(), so every wait state is one the deadline can interrupt.
Please also add a test that holds s.mu for longer than the timeout and asserts the function returns within its bound. Without that test this reintroduces itself silently, since the current code reads as bounded.
Credit: found by the concurrency audit of c213602 during the v0.10.47 hotfix review (AgentWorkforce/cloud#3143), which flagged it rather than smoothing it over.
Defect
waitForLocalMutationToSettle(added in c213602, shipped in v0.10.47) is written to bound its wait at 2 seconds: it polls every 2ms, honoursctx.Done(), and carries a 2s timer.The timer does not bound the function. Each poll iteration calls a blocking
s.mu.Lock()outside theselect. A permanently wedged mutex holder therefore blocks the call past its own deadline — the timer is only consulted between calls, never during one.This is the same defect class we already hit once in this incident: a deadline checked between calls does not bound a call. A 5-minute timeout sat unreachable through a 62-minute hang for exactly this reason.
Severity and why this is a follow-up, not a blocker
Not currently reachable on the failing path. The one-shot initial seed never calls this function —
PushLocalAndFlushOnceflushes withunlockDuringUpload=false, and only the realtime watcher fast path sets it true. On timeout the event is simply not checkpointed; the WebSocket loop reconnects from the previous cursor and replays it, so the failure mode is delay rather than silent loss.So: real, bounded in blast radius today, and worth fixing before something else starts calling it.
Requested fix
Acquire the lock through a bounded path so the deadline actually governs the call — e.g. a
TryLockloop driven by the sameselectthat watches the timer andctx.Done(), so every wait state is one the deadline can interrupt.Please also add a test that holds
s.mufor longer than the timeout and asserts the function returns within its bound. Without that test this reintroduces itself silently, since the current code reads as bounded.Credit: found by the concurrency audit of c213602 during the v0.10.47 hotfix review (AgentWorkforce/cloud#3143), which flagged it rather than smoothing it over.