Add automatic batch deprioritization for large submissions - #417
Open
helq wants to merge 6 commits into
Open
Conversation
Add a submitted_at field to Event::RawMboxSubmitted, stamped with the server's current time when a raw mbox is submitted via POST /api/submit (Inject variant). This timestamp is propagated through metadata.received_date and used as the patchset date instead of the email's Date: header. This prevents stale mbox timestamps (which can be arbitrarily far in the past) from skewing queue ordering. The original email date is preserved in the messages table for display purposes. The Thread and Remote submission paths are unaffected as they already use server-side timestamps via create_fetching_patchset(). Signed-off-by: Elkin Cruz <elkin@google.com>
Define PriorityRule and CompiledPriorityRule structs for regex-based patchset priority classification. Add a custom serde deserializer (deserialize_indexed_vec) to handle both TOML array and env-var indexed-map representations. Add the priority_rules field to ReviewSettings with serde(default) so existing configs are unaffected. Signed-off-by: Elkin Cruz <elkin@google.com>
Add a priority INTEGER DEFAULT 500 column to the patchsets table and a composite index idx_patchsets_status_priority_date on (status, priority DESC, date ASC) for efficient priority-ordered queries. The column defaults to 500 so existing patchsets are unaffected. Signed-off-by: Elkin Cruz <elkin@google.com>
Add migration to create the priority column and composite index. Introduce create_patchset_with_priority() which threads an explicit priority through all INSERT/UPDATE paths, with MIN(priority, ?) semantics to preserve manual deprioritization. Refactor create_patchset() to delegate with default None. Change get_pending_patchsets() ordering to priority DESC, date ASC. Add calculate_priority() for evaluating compiled regex rules against subjects (last match wins). Signed-off-by: Elkin Cruz <elkin@google.com>
Compile priority_rules from settings at startup and thread them through the DB worker into process_parsed_article(). Use calculate_priority() to compute priority from the patchset subject before calling create_patchset_with_priority(). API callers pass None for priority to create_fetching_patchset(). Signed-off-by: Elkin Cruz <elkin@google.com>
When a patchset transitions to Pending, count how many sibling
patchsets share a similar timestamp (within a configurable window,
default 30s). If the count exceeds configured thresholds, cap
the priority of all patchsets in that window using MIN(priority, cap)
to avoid blocking the review queue with massive batch syncs.
Three new config options in [review]:
- batch_window_secs: time window for sibling detection (default 30)
- batch_tiers: list of {min_size, priority} threshold tiers
The deprioritization is applied retroactively to all patchsets in
the window on each new arrival, ensuring early arrivals in a batch
get deprioritized once the batch size becomes apparent. The MIN()
semantics ensure priority is never raised, preserving any manual
or rule-based deprioritization already in place.
Signed-off-by: Elkin Cruz <elkin@google.com>
helq
force-pushed
the
upstream/batch-deprior
branch
from
August 21, 2026 18:38
791b5fc to
d4b1f5a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On top of #415
We receive large requests into our sashiko instance (upon thousands of patchset reviews in a single minute). To keep sashiko servicing reviews that don't come in the batch, this change lowers the priority of patchsets that arrive within the same window of time (30 seconds). How many requests and how large the window should be completely depend on our use case. We use:
When a patchset transitions to Pending, count how many sibling
patchsets share a similar timestamp (within a configurable window,
default 30s). If the count exceeds configured thresholds, cap
the priority of all patchsets in that window using MIN(priority, cap)
to avoid blocking the review queue with massive batch syncs.
Three new config options in [review]:
The deprioritization is applied retroactively to all patchsets in
the window on each new arrival, ensuring early arrivals in a batch
get deprioritized once the batch size becomes apparent. The MIN()
semantics ensure priority is never raised, preserving any manual
or rule-based deprioritization already in place.