fix(broadcast): publish slots in sequence - #145
Draft
tisonkun wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Correctness failure
The previous
tail_cnt.fetch_addwas a reservation cursor, but receivers treated it as a committed cursor. A producer could reserve sequencenand stall before taking the slot lock while a later producer reserved and publishedn + 1. After a ring wrap, the older producer could then overwrite the newer slot. Before the wrap, receivers could observe an unwritten hole;recv().awaitwould repeatedly retry becausetail != head, and a panic between reservation and publication left that hole permanent.This patch makes
State::taila committed-prefix boundary: after observing tailn, every sequence beforenhas a complete slot. Concurrent sends are ordered by the state mutex. A receiver acquires its slot read lock while the same state lock still protects the tail snapshot, then releases the state lock before cloning the value. Wakers are still invoked after all internal locks are released.Why a mutex in this draft
Tokio keeps its broadcast tail and receiver wait list under one mutex and holds that serialization point through slot publication: Tokio broadcast source. This patch uses the same correctness shape without attempting a performance redesign.
LMAX Disruptor is useful context, but its multi-producer sequencer deliberately separates claiming from publication. It uses a reservation cursor plus a per-slot availability generation, scans for the highest contiguous published sequence, and prevents producers from wrapping past gating consumers: user guide, MultiProducerSequencer. MEA broadcast overflow intentionally has no producer gating, so copying only the claim cursor would retain the stale-producer overwrite problem. A per-slot sequencer would also need an explicit linearizability rule for
try_recvwhen a later send completes before an earlier claim publishes. That optimization is intentionally deferred.Channel direction for 0.7
Producer/consumer cardinality is only one axis. In particular, an MPMC work queue delivers each item to one competing receiver, while an MPMC broadcast delivers each item to every receiver. The public taxonomy should lead with delivery semantics and capacity policy, not generate a public type for every topology combination.
send,try_recvrecv/Futuretry_send,try_recvsend().await,recv().awaitsend,try_recvrecv().awaitsend,try_recvrecv().awaitsend,try_recvrecv().awaittry_send,try_recvsend().await,recv().awaitsend, inspectchanged().awaitRecommendations:
oneshot,mpsc,broadcast, and a futurewatchas the public families. Treat SPSC as a possible optimized implementation/factory for queue or broadcast, not a new family. Add SPMC/MPMC competing-consumer queues only when there is a concrete use case; cloneable receivers materially change fairness, cancellation, and close semantics.try_send/try_recvfor immediate attempts,poll_send/poll_recvinternally where waiting exists, and asyncsend/recvas thin wrappers. Do not add blocking methods to MEA;pollster::block_oncan park a synchronous caller as already documented in the README. OptionalStream/Sinkadapters can be a separate interoperability feature.broadcast::overflow,broadcast::unbounded, andbroadcast::backpressureshould not be one generic implementation with policy branches. The same applies to bounded versus unbounded queues.Proposed public grouping
For the 0.7 breaking release, a coherent grouping would be:
The matching Cargo features can be additive umbrella features with leaf features:
Leaf features should encode their internal dependencies, while umbrella features only group them. Keeping
fullas the default preserves current out-of-box behavior; moving existing APIs behind opt-in features is therefore reserved for 0.7. CI should cover no-default, each umbrella, default, and all-features rather than every exponential leaf combination. This follows Cargo's additive feature model: https://doc.rust-lang.org/cargo/reference/features.htmlWhether 0.7 should physically move all paths under these four modules or use the grouping only for documentation and Cargo features is a separate migration choice. My recommendation is to move the paths in 0.7 if we want the grouping at all, rather than maintain duplicate root and grouped APIs indefinitely.
Scope
This PR only establishes the broadcast correctness baseline. It does not add channel families, change public paths, add Cargo features, or claim a throughput improvement.
Validation
cargo test -p mea broadcast::overflow -- --nocapturecargo x lintcargo x test --no-capturecargo +1.85.0 test --workspace --no-default-features