feat: adopt reth pool maintenance and persist the txpool across restarts

[bronxyz]

Summary

• Replace the hand-rolled canonical-state pool update with reth's maintain_transaction_pool_future, and hold locally submitted transactions to the same fee and eviction rules as external ones (no_local_exemptions). In-flight marks release on a dedicated canonical-stream task, with the TTL sweep and membership reconcile driven from the 30s engine gap-check tick so a burst cannot starve release.
• Delete the bypass validator and the orphan-batch re-introduction: sealed-but-uncommitted transactions now stay pending and marked in flight, so nothing is re-collected from NodeBatchesCache and bypass-validated back in.
• Persist every worker pool (pending + queued transactions and their marks) on graceful shutdown and reload it on boot, so sealed-but-uncommitted transactions, which no peer can re-supply, survive a restart. A safety-net snapshot runs before the unbounded engine-drain await; both writes are tmp-then-rename so the post-drain snapshot wins on the healthy path.

Stack 2/9 of the txpool in-flight tracker and observer-forwarder series.

Surface areas touched

• Consensus protocol (primary / worker / network / state-sync)
• Execution / EVM
• JSON-RPC (eth_*, rayls_*, faucet)
• Middleware (orchestrator / processor / bridge)
• Infrastructure (types / storage / config / network-cli)
• On-chain contracts (rayls-contracts/)
• Operations (etc/, scripts, Docker, compose)
• CI / build (.github/workflows/, Makefile)
• Documentation only (doc/, in-crate READMEs, root docs)
• Tests only

Breaking / compatibility

None on the wire. Behavior change: local transactions lose their fee/eviction exemption, and a node now writes a txpool backup file under its datadir on graceful shutdown. A transaction executed during the drain is rejected nonce-too-low on reload and its mark reconciled away.

Test plan

• atomic_write_replaces_existing_file, streaming_reader_accepts_legacy_json_array in txn_pool/backup.rs.
• Builder integration tests (tests/it/build_batches.rs) pass against the reth maintenance task.
• make check on the stack tip; CI on this branch.

[procdump]

do we need to persist queued/pending txs upon graceful restarts - if we delay the start up of the stopped validator could this lead to double spending problem because the user erroneously retried with a new nonce? - also in the case of being an observer its snapshot may be used for spinning a new one inheriting the persisted txs - could this lead to an issue?

[bronxyz]

The txpool and evm won't allow for double spending to occur.

1. When we restart - we are going to enter the CvvInactive state, which doesn't produce any batches. Any tx that has been part of a block will be dropped by the pool maintenance. All remaining ones are still eligible to be included in batches when they become pending.
2. The pool snapshot must be used for local purposes only, so snapshots must not include this local pool backup for restoring purposes.

[github-actions]

Claude finished @bronxyz's task in 2m 52s —— View job

---

PR Review Update: feature/txpool-reth-pool-maintenance

• Load tools
• Read CLAUDE.md for repo guidelines
• Diff vs base branch (origin/feature/txpool-in-flight-tracker-core)
• Check whether previous red/yellow issues were addressed
• Read changed files
• Post updated review

---

Summary: This sync appears to be a rebase onto the updated base branch. The substantive new addition is the spawn_engine_update_task change in engine.rs, which wires sweep_due and reconcile_in_flight into the 30 s gap-check tick — correctly addressing the burst-starvation concern raised in the previous review. The two red issues and two yellow issues from the previous review remain open.

---

What's new vs. the previous review

engine.rs — gap-check tick now drives TTL sweep and forward-mark reconcile

crates/middleware/orchestrator/src/epoch_manager/engine.rs:66–77

_ = gap_check_interval.tick() => {
    consensus_bus.batch_tracker().check_gaps();
    let anchor = consensus_bus.executed_anchor().borrow().number;
    for pool in engine.get_all_worker_transaction_pools().await {
        pool.in_flight().sweep_due(anchor);
        pool.reconcile_in_flight();
    }
}

This is the right fix for the burst-starvation concern: forward marks that survive a block burst now get a guaranteed release path every 30 s regardless of whether the canonical-stream task is keeping up. The is_empty() early-return in reconcile_in_flight (line 221 of txn_pool.rs) makes the tick cheap when no marks are live. The O(N-pending) cost still applies once any mark exists, but at 30 s frequency it is not a hot-path concern — only the per-block canonical-stream call is.

One note: pool.in_flight() clones the InFlightTracker (which is Arc-backed) before calling sweep_due on it. That is correct — the clone shares the same underlying state — but the caller's variable is a clone, not a reference. This is fine as long as sweep_due mutates through the Arc interior (which it should, given InFlightTracker: Clone). Worth a quick sanity-check that sweep_due takes &self and uses interior mutability rather than &mut self.

---

Open issues from previous review

🔴 max_tx_lifetime: Duration::from_mins(5) — still unresolved

crates/execution/evm/src/txn_pool.rs:176

MaintainPoolConfig {
    max_tx_lifetime: Duration::from_mins(5),
    no_local_exemptions: true,
    ..Default::default()
},

Reth's default is 3 hours. Five minutes means any transaction submitted during a moderate consensus hiccup or epoch boundary will silently evict before it can be sealed. This interacts with TODO(issue 114) on line 296: the pool seals at MIN_PROTOCOL_BASE_FEE rather than an accurate next-block base fee, so fee-based eviction is already miscalibrated — layering a 5-minute lifetime on top compounds the risk. This either needs a justified comment explaining why 5 min is correct for this network's block times, or should be changed to the reth default. Fix this →

🔴 Safety-net snapshot silently drops JoinError — still unresolved

crates/middleware/orchestrator/src/epoch_manager/core.rs:422–431

{
    let pools = engine.get_all_worker_transaction_pools().await;
    let _ = tokio::task::spawn_blocking(move || {
        for pool in &pools {
            pool.save_backup();
            pool.save_mark_backup();
        }
    })
    .await;   // JoinError (blocking task panic) silently discarded
}

The post-drain snapshot at lines 448–458 wraps an identical pattern in AssertUnwindSafe(...).catch_unwind() and logs on error. The pre-drain safety-net does not. A panic in the blocking task here propagates as a JoinError that is silently swallowed, leaving the node to proceed into the unbounded engine_done_rx.await with no backup and no trace in the logs. Minimum fix: match the post-drain pattern. Fix this →

🟡 Stale orphan_batches comments — still unresolved

crates/consensus/worker/src/batch-builder/src/lib.rs:268–269

// those batches don't make it into this epoch's blocks and are rescued next epoch by
// orphan_batches, so nothing is lost.

crates/consensus/primary/src/proposer/recovery.rs:64–66

// skip NodeBatchesCache cleanup for a header whose subdag reaches
// the epoch boundary — the subscriber drops its output, so orphan_batches must still find
// its batches to rescue them.

orphan_batches is gone. Both comments describe a mechanism that no longer exists. The second one in recovery.rs is load-bearing: the dropped=true guard it explains may still be correct (preserving NodeBatchesCache entries while those transactions are pending-in-flight), but the reasoning now points to a deleted function and will mislead the next reader deciding whether to keep or remove the guard. Fix these →

🟡 reconcile_in_flight still materializes the full pending pool on every canonical commit

crates/execution/evm/src/txn_pool.rs:221–229

pub fn reconcile_in_flight(&self) {
    if self.in_flight_tracker.is_empty() {
        return;
    }
    let pending: B256Set =
        self.pool.pending_transactions().iter().map(|tx| *tx.hash()).collect();
    let released = self.in_flight_tracker.release_mined(&pending);
    ...
}

The is_empty() early-return keeps the 30 s gap-check tick cheap in the steady state. However, during active batch building (the common case), the canonical-stream subscription fires on every block and each call allocates a full Vec<Arc<PoolTxn>> + B256Set over all pending transactions. An O(marks) approach — iterating the tracker's own mark set and calling self.pool.get(hash) per mark — would scale with active batch count rather than pool size. Fix this →

🟢 Minor: load_mark_backup computes mark_backup_path() twice — still unresolved

crates/execution/evm/src/txn_pool/backup.rs:197–209

---

Positive observations

The TTL-sweep + forward-mark reconcile wired into the 30 s gap-check tick (new in this sync) is the right architectural answer to burst-starvation. The is_empty() guard makes it free in the steady state. The pool-per-tick fetch (get_all_worker_transaction_pools().await) is the correct pattern given workers are created after the node-scoped task starts.