perf(engine): inline WS/SSE broadcast egress on the dispatch goroutine (+43% io_uring & epoll) - #404
Merged
Merged
Conversation
The io_uring and epoll engines serialise off-event-loop writes (WebSocket frames, SSE events) through a per-loop detachQueue: an off-loop goroutine appends the conn and writes a wakeup eventfd so the event loop drains the queue and flushes each conn's writeBuf. Under a Hub broadcast fan-out the write closure runs once per (message x connection) from the GOMAXPROCS*4 dispatch goroutines, and every call issued its own eventfd wakeup write. At 1024 connections that is a storm of redundant wakeup syscalls — the loop needs only one wakeup per drain cycle, because a single drain already flushes every queued conn. Coalesce with an edge-triggered wakeup: only the enqueue that takes the detach queue empty->non-empty writes the eventfd, gated by `wasEmpty := detachQPending.Swap(1) == 0` under detachQMu. drainDetachQueue clears the flag (Store(0)) under the same lock *before* it swaps the queue out, so a racing enqueue is either captured by the drain's swap or observes pending==0 and re-arms the eventfd — never both missed. No wakeup is dropped. The detachMu-guarded writeBuf mutation is untouched, so the WS-write-vs-flushWrites ordering invariant (celeris#284) is preserved. Applied uniformly to every off-loop enqueue site in both engines (WS write, recv pause/resume backpressure, async-detach setup); loop-thread enqueues are unchanged since they run on the drain thread and never race the flag. Benchmarks (ws-hub-broadcast, 1024 conns, amd64, median of 3 x 12s): io_uring 773,823 -> 918,111 rps (+18.6%) epoll 718,965 -> 838,918 rps (+16.7%) go test -race passes on both engine/iouring and engine/epoll.
Detached WS/SSE writes funneled through the single event-loop thread (append to writeBuf under detachMu, enqueue to the per-loop detachQueue, wake the loop, which did the write). On a broadcast fan-out that serializes 1024 sends across N loop threads (N/thread), while the std engine does them inline on GOMAXPROCS goroutines across all cores — the ~2x broadcast gap. Issue the send inline on the dispatch goroutine when the conn is clean: flushWrites(cs,false) runs inside the same detachMu critical section that already guards orig() and that the loop-thread dirty-flush + closeConn take, so it can neither race the loop's flush nor touch a closed fd; writeBuf is one ordered buffer flushed from writePos, so no reorder. On full drain we skip the loop handoff entirely; on partial/EAGAIN/error we fall through to the existing enqueue path (surfacing OnError on I/O failure first). ws-hub-broadcast @1024 conns (epoll, amd64): +43-48% (638K -> ~930-960K), loop-thread CPU 50%% -> 0.3%% (write moves to the dispatch goroutines). -race clean on engine/epoll + middleware/websocket.
…_ISSUER-safe) Port the epoll inline-egress ceiling-breaker to io_uring. SINGLE_ISSUER forbids the dispatch goroutine from submitting a ring SEND, but a raw unix.Write(2) on the socket fd is legal iff no ring SEND is in-flight for the conn — else the two writes interleave on the wire. detachMu gates it: every ring SEND is submitted under detachMu with cs.sending set, and completeSend now CLEARS cs.sending under detachMu too (was cleared before the lock — a cross-thread read race the inline path would hit). Gated on !cs.fixedFile: under ACCEPT_DIRECT cs.fd is a ring file-table index, not a syscall'able fd, so those conns keep using the ring (same guard hijack uses). Also gates on !zcNotifPending + empty sendBuf/bodyBuf. ws-hub-broadcast @1024 conns (io_uring, amd64, fixed_files=false): +43.2% (663K -> ~950K). -race clean on engine/iouring (102s) + middleware/websocket.
…ne egress Broadcasts 8KB frames (>= sendZCMinBytes, so io_uring uses SEND_ZC) to 32 detached conns while dispatch goroutines issue inline writes, asserting every received frame is byte-intact and per-conn in order. Run under -race in CI this covers the SEND_ZC-completion-vs-inline-write interaction the 64B ws-hub benchmark never reaches. Passes -race x3 on both epoll + io_uring.
FumingPower3925
changed the base branch from
perf/ws-broadcast-wakeup-coalesce
to
main
July 3, 2026 11:21
…panic Three pre-existing (v1.5.6) bugs in the engine WS detach/close path, surfaced by a -race build under aggressive peer-RST-mid-upgrade: 1. WSReady upgrade-completion barrier. The WS middleware installs the detached callbacks (RawWriteFn, pause/resume, idle-deadline, OnDetachClose LAST) on the async goroutine AFTER Detach releases detachMu (celeris#273/#309), so they were not lock-serialised against closeConn reading OnDetachClose / PauseRecv while tearing the conn down on a peer RST. H1State.WSReady is Stored(true, release) as the final wiring step; closeConn Loads it (acquire) before touching those callbacks, so it sees a fully-wired conn or skips WS teardown entirely (conn still closed via fd/read path). Gated in both engines. 2. writeErr atomic.Value inconsistent-type panic (== the validator's I-LIVENESS code=2 crash). OnError stores errors of varying concrete types (errPeerClosed, syscall errors, …); atomic.Value.Store panics on the second differing type. Boxed via storedWriteErr so the stored dynamic type is constant; also skip storing a nil error. -race clean on ./middleware/websocket, ./engine/{epoll,iouring}, ./internal/conn.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Break the WS/SSE broadcast throughput ceiling on the loop engines by issuing the socket send inline on the dispatch goroutine instead of funnelling every detached-conn write through the single event-loop/worker thread — the same thing the
stdengine does (goroutine-per-conn, directwrite(2)).Stacked on #403 (eventfd-wakeup coalesce). Together they are the v1.5.7 WS-broadcast work.
Root cause (measured)
On
ws-hub-broadcastat 1024 conns,stdbeat io_uring/epoll ~2×, and all loop variants clustered at a shared ~500K ceiling (flat at 128 conns — a high-connection cliff). CPU profile @1024c (epoll): the loop thread was 50% of all CPU, of whichflushWrites→unix.writewas 37.5% — every conn'swrite(2)serialized on one thread per core, whilestdspreads them acrossGOMAXPROCSgoroutines.detachQMucontention was ~1% (ruled out). #403 removed the wakeup-syscall storm (+17%) but left this funnel.Fix
epoll: in the detached
guardedwriteFn, after thewriteBufappend, attemptflushWrites(cs, false)inline. It runs inside the samedetachMucritical section the loop-thread dirty-flush andcloseConntake, andwriteBufis one ordered buffer flushed fromwritePos, so it can neither race the loop's flush nor reorder nor touch a closed fd. Full drain → skip the loop handoff entirely; partial/EAGAIN/error → fall through to the existing enqueue path (surfacingOnErroron I/O failure).io_uring (
SINGLE_ISSUER-safe): the ring may only be driven by the worker, so use a rawunix.Write(cs.fd), valid iff no ring SEND is in-flight for the conn (!cs.sending && !cs.zcNotifPending && sendBuf/bodyBuf empty, all read underdetachMu) and the fd is real (!cs.fixedFile— underACCEPT_DIRECT,cs.fdis a ring file-table index, so those conns keep using the ring). Companion hardening:completeSendand thehandleSendSEND_ZC/error branches now clearcs.sending/cs.zcNotifPending/sendBufunderdetachMuso the dispatch-goroutine read is race-free.Results (amd64,
fixed_files=false)Profile (epoll): loop-thread CPU 50.4% → 0.3% — the
write(2)moved onto the dispatch goroutines.Validation
go test -raceclean onengine/epoll,engine/iouring(102s),middleware/websocket— including an 8KB-frame run that exercises the SEND_ZC-completion-vs-inline-write path (the 64B benchmark never crossed the 4096B ZC threshold).OnErroron the inline error path, and io_uring'shandleSendsend-state race.TestNativeEngineHubBroadcastInlineEgress(both engines,-race) asserts no corruption/reorder under concurrent large-frame broadcast.Known limitation
The io_uring inline path is gated on
!cs.fixedFile. On kernels whereACCEPT_DIRECTworks, WS conns keep the ring path (no regression, no inline win there — aFIXED_FD_INSTALLfollow-up could materialize a real fd). The win applies wherever fixed files aren't in use, including the benchmark cluster (kernel 7.0 rejectsACCEPT_DIRECT).Update: pre-existing WS upgrade/close hardening (commit ba3887a)
Validating this branch through the nightly matrix-validate gate surfaced pre-existing (v1.5.6) races + a panic in the engine WS detach/close path — not introduced by the inline-egress change (they reproduce on v1.5.6 without it). Fixed here so the gate passes:
atomic.Valueinconsistent-type panic inConn.writeErr— this was the validator'scode=2/ I-LIVENESS crash.OnErrorstored errors of varying concrete types; boxed viastoredWriteErrsoatomic.Valueholds one type (+ skip nil).WSReadyupgrade-completion barrier (H1State.WSReady) — the WS middleware wires the detached callbacks afterDetachreleasesdetachMu;closeConnread them concurrently on a peer-RST-mid-upgrade.WSReadyisStored (release) as the final wiring step andLoaded (acquire) beforecloseConntouches the callbacks, so it sees a fully-wired conn or skips WS teardown (conn still closed via the fd path). Gated in both engines.PauseRecv/ResumeRecvdrop race — same barrier.Validation: full nightly matrix-validate (every refapp × engine, both archs,
-race+checkptr, incl. the I-CONN leak-balance properties) is GREEN;go test -raceclean onwebsocket+ both engines +conn.Known separate item: a deliberately brutal RST-mid-upgrade
-racerepro still shows a different pre-existing race cluster — aContextrecycled while a detached async handler holds it. It does not trip the validator (v1.5.6 passed it) and is unrelated to v1.5.7; filed as follow-up hardening.