Skip to content

fix(broadcast): publish slots in sequence - #145

Draft
tisonkun wants to merge 1 commit into
mainfrom
codex/fix-broadcast-publication-order
Draft

fix(broadcast): publish slots in sequence#145
tisonkun wants to merge 1 commit into
mainfrom
codex/fix-broadcast-publication-order

Conversation

@tisonkun

@tisonkun tisonkun commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • serialize broadcast slot publication under one state mutex
  • keep the tail cursor and receiver waiter set in that state so checking the tail and registering a waiter is atomic
  • advance the tail only after the selected slot is fully written
  • add a regression test proving that a panic while replacing a slot cannot publish a permanent hole

Correctness failure

The previous tail_cnt.fetch_add was a reservation cursor, but receivers treated it as a committed cursor. A producer could reserve sequence n and stall before taking the slot lock while a later producer reserved and published n + 1. After a ring wrap, the older producer could then overwrite the newer slot. Before the wrap, receivers could observe an unwritten hole; recv().await would repeatedly retry because tail != head, and a panic between reservation and publication left that hole permanent.

This patch makes State::tail a committed-prefix boundary: after observing tail n, every sequence before n has a complete slot. Concurrent sends are ordered by the state mutex. A receiver acquires its slot read lock while the same state lock still protects the tail snapshot, then releases the state lock before cloning the value. Wakers are still invoked after all internal locks are released.

Why a mutex in this draft

Tokio keeps its broadcast tail and receiver wait list under one mutex and holds that serialization point through slot publication: Tokio broadcast source. This patch uses the same correctness shape without attempting a performance redesign.

LMAX Disruptor is useful context, but its multi-producer sequencer deliberately separates claiming from publication. It uses a reservation cursor plus a per-slot availability generation, scans for the highest contiguous published sequence, and prevents producers from wrapping past gating consumers: user guide, MultiProducerSequencer. MEA broadcast overflow intentionally has no producer gating, so copying only the claim cursor would retain the stale-producer overwrite problem. A per-slot sequencer would also need an explicit linearizability rule for try_recv when a later send completes before an earlier claim publishes. That optimization is intentionally deferred.

Channel direction for 0.7

Producer/consumer cardinality is only one axis. In particular, an MPMC work queue delivers each item to one competing receiver, while an MPMC broadcast delivers each item to every receiver. The public taxonomy should lead with delivery semantics and capacity policy, not generate a public type for every topology combination.

Family Topology Delivery Capacity policy Immediate API Waiting API
oneshot SPSC one value once one slot send, try_recv recv / Future
mpsc bounded MPSC each value once bounded backpressure try_send, try_recv send().await, recv().await
mpsc unbounded MPSC each value once memory-bounded in practice send, try_recv recv().await
broadcast overflow MPMC multicast every receiver overwrite oldest send, try_recv recv().await
broadcast unbounded MPMC multicast every receiver grow and reclaim by slowest cursor send, try_recv recv().await
broadcast backpressure MPMC multicast every receiver gate on slowest receiver try_send, try_recv send().await, recv().await
watch, later MPMC state distribution latest value only coalesce send, inspect changed().await

Recommendations:

  1. Keep oneshot, mpsc, broadcast, and a future watch as the public families. Treat SPSC as a possible optimized implementation/factory for queue or broadcast, not a new family. Add SPMC/MPMC competing-consumer queues only when there is a concrete use case; cloneable receivers materially change fairness, cancellation, and close semantics.
  2. Use one nonblocking state machine per operation: try_send / try_recv for immediate attempts, poll_send / poll_recv internally where waiting exists, and async send / recv as thin wrappers. Do not add blocking methods to MEA; pollster::block_on can park a synchronous caller as already documented in the README. Optional Stream / Sink adapters can be a separate interoperability feature.
  3. Keep policy-specific implementations separate where their state is genuinely different. broadcast::overflow, broadcast::unbounded, and broadcast::backpressure should not be one generic implementation with policy branches. The same applies to bounded versus unbounded queues.
  4. Use Disruptor-style single- and multi-producer sequencers only after benchmarks show the state mutex is material. For backpressured multicast, gating sequences map naturally. For overflow multicast, a correct lock-free design also needs stale-writer rejection and a defined contiguous-publication contract.

Proposed public grouping

For the 0.7 breaking release, a coherent grouping would be:

mea::sync::{Barrier, Condvar, Latch, Mutex, Once, OnceCell, OnceMap, RwLock, Semaphore, WaitGroup}
mea::channel::{oneshot, mpsc, broadcast}
mea::coordination::{admission, shutdown, singleflight}
mea::atomic::{AtomicBox, AtomicOptionBox}

The matching Cargo features can be additive umbrella features with leaf features:

[features]
default = ["full"]
full = ["sync", "channel", "coordination", "atomic"]
sync = ["barrier", "condvar", "latch", "mutex", "once", "rwlock", "semaphore", "waitgroup"]
channel = ["oneshot", "mpsc", "broadcast"]
coordination = ["admission", "shutdown", "singleflight"]
atomic = ["atomicbox"]

Leaf features should encode their internal dependencies, while umbrella features only group them. Keeping full as the default preserves current out-of-box behavior; moving existing APIs behind opt-in features is therefore reserved for 0.7. CI should cover no-default, each umbrella, default, and all-features rather than every exponential leaf combination. This follows Cargo's additive feature model: https://doc.rust-lang.org/cargo/reference/features.html

Whether 0.7 should physically move all paths under these four modules or use the grouping only for documentation and Cargo features is a separate migration choice. My recommendation is to move the paths in 0.7 if we want the grouping at all, rather than maintain duplicate root and grouped APIs indefinitely.

Scope

This PR only establishes the broadcast correctness baseline. It does not add channel families, change public paths, add Cargo features, or claim a throughput improvement.

Validation

  • cargo test -p mea broadcast::overflow -- --nocapture
  • cargo x lint
  • cargo x test --no-capture
  • cargo +1.85.0 test --workspace --no-default-features

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant