Add durable execution-queue overflow (opt-in) - #335
Draft
harry-anderson wants to merge 4 commits into
Draft
Conversation
Persist execution-queue messages that would otherwise be dropped, so they survive a queue-full burst or an ungraceful shutdown and are replayed on a later boot instead of being lost. Gated behind an optional [executor.queue_overflow] config block; absent means the feature is off and behaviour is unchanged. Requires a persistent storage backend (hard error at startup otherwise). - Spill-on-full: the "Queue Full! log dropped!" path hands the message to a bounded async channel; a dedicated task does the storage write off the webhook thread so slow storage cannot block ingress. GET-mode messages are excluded since they cannot be replayed. - Same-role reload: webhook-role pods claim and reinject batches, backing off when the queue is full. Claiming is atomic (delete-returns-value) so multiple replicas are safe against a shared backend. - Timeboxed spill-on-shutdown: the executor-drain join is bounded; on timeout (the wedged-executor case) still-queued messages are force-persisted. Messages are gzip-compressed and keyed by creation time for chronological replay, with a per-item size guard, an approximate persisted-count cap, and a max-age reaper. Cross-pod recovery only applies with a shared backend such as DynamoDB (documented on the config field).
obelisk
requested review from
Copilot and
michelemin
and removed request for
michelemin
July 2, 2026 22:23
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in “durable overflow” path for the executor’s bounded execution queue: when the queue is full (or shutdown is forced), eligible messages are persisted to a storage namespace and later replayed on boot by webhook-role instances, reducing loss during bursts or ungraceful termination.
Changes:
- Adds
[executor.queue_overflow]config and startup validation requiring a persistent storage backend. - Introduces
executor::overflow::OverflowStorefor persist/claim/reinject of overflowed messages (gzip-compressed, chronological keys, max-age drop). - Implements spill-on-full (bounded async channel to offload storage writes), same-role reload polling, and a timeboxed shutdown drain that persists still-queued messages.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| runtime/plaid/src/storage/mod.rs | Exposes Storage::is_persistent() to support startup validation of the feature. |
| runtime/plaid/src/executor/overflow.rs | New durable overflow store implementation (persist + reload/claim logic, compression, tests). |
| runtime/plaid/src/executor/mod.rs | Exports the new overflow module. |
| runtime/plaid/src/config.rs | Adds [executor.queue_overflow] config struct and documentation. |
| runtime/plaid/src/bin/plaid.rs | Wires spill channel, reload task, persistent-backend validation, and shutdown drain behavior into the Plaid binary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }; | ||
| // We removed a row that we had counted; reflect that regardless of what | ||
| // happens next (reinjected, reaped, or dropped as corrupt). | ||
| self.count.fetch_sub(1, Ordering::Relaxed); |
Comment on lines
+240
to
+243
| Err(TrySendError::Disconnected(_)) => { | ||
| error!("Executor channel disconnected while reloading overflow {key}"); | ||
| break; | ||
| } |
| maybe_msg = rx.recv() => { | ||
| match maybe_msg { | ||
| // None means all senders dropped; nothing left to persist. | ||
| Some(message) => { let _ = store.persist(&message).await; } |
| // executor-drain step at the end of main(), which is the real | ||
| // backstop for un-enqueued messages during shutdown. | ||
| while let Ok(message) = rx.try_recv() { | ||
| let _ = store.persist(&message).await; |
Comment on lines
+831
to
+834
| let join_handle = tokio::task::spawn_blocking(move || executor_threads.join()); | ||
| match tokio::time::timeout(EXECUTOR_DRAIN_DEADLINE, join_handle).await { | ||
| Ok(_) => info!("Executor threads drained cleanly"), | ||
| Err(_) => { |
Comment on lines
+165
to
+166
| /// How many overflow messages to claim and reinject per reload poll. Keeps a single | ||
| /// poll from loading an unbounded namespace into memory. Defaults to 256. |
Redesign claim as ready→inflight lease so a crash mid-reload no longer drops messages after delete. Add saturating count accounting, age reaper, base64 wire format (smaller items), forced-spill on shutdown, detached executor join on drain timeout, and broader unit tests for the failure modes found in adversarial QA.
Resolve conflicts with 429-on-full and Prometheus metrics: - Keep durable spill on queue-full; return 429 only if spill also fails - Export both metrics and overflow executor modules
Address adversarial QA v2 findings for reliability under spikes: - Honest HTTP admission: 202 when accepted onto spill, 429 under pressure - Concurrent spill workers (configurable) instead of a single serial writer - Spill high-watermark + recent-failure backpressure before channel hard-fill - Bounded list_keys_limited (oldest-first) for reload/reap/reclaim - Reinject high-watermark to avoid claim→restore thrash on near-full queues - Age reaper on a slower cadence than reload - Binary v2 envelope (legacy v1/json still decode); faster gzip - Prometheus overflow metrics (depth, persist outcomes, reload events) - Route interval/SQS/GitHub/Okta/websocket through send_or_spill - Higher default spill capacity (4096) and concurrency (8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Persist execution-queue messages that would otherwise be dropped, so they survive a queue-full burst or an ungraceful shutdown and are replayed on a later boot instead of being lost.
The feature is gated behind an optional
[executor.queue_overflow]config block. When absent (the default) behaviour is unchanged. It requires a persistent storage backend and fails fast at startup otherwise.What it does
Queue Full! log dropped!path now hands the message to a bounded async channel; a dedicated task performs the storage write off the webhook thread, so slow storage cannot block ingress. GET-mode messages (which carry a response channel and cannot be replayed) are excluded.Messages are gzip-compressed and keyed by creation time for chronological replay, with a per-item size guard, an approximate persisted-count cap, and a max-age reaper.
Config
Notes / limits
Test plan
cargo check/cargo clippyclean (no new warnings)