fix(feed): stop a dead follow from stalling feedQuery for minutes - #436
Merged
Conversation
The dashboard's merged feed took minutes to load whenever one followed
xite's saved feed SQL referenced a table its database no longer has (a
follow whose SQL outlived the xite's schema). Two compounding causes:
- db_query classified "no such table"/"no such column" as transient and
retried them for the full 60s budget. That classification exists for
the rebuild window where tables are dropped and recreated under a
stable handle - but on a settled database the name never appears, so
every feedQuery call burned the whole budget on the dead query.
- feedQuery ran each followed xite's queries serially, so that one 60s
stall blocked every other xite's rows behind it.
Fix both ends:
- settle_stable_db_result now retries a missing table/column only while
a clone or db rebuild is actually in flight for the xite (the same
busy signal await_db_quiet polls); on a quiet database it answers with
the error immediately. This is race-safe: a finished rebuild bumps
db_generation, so a query that raced one lands in the receipt-changed
retry branch, never this fail-fast path. SQLite contention errors
("locked"/"busy") retry exactly as before.
- feedQuery runs all followed feeds' queries concurrently, each under
its own 10s deadline; a timed-out or failed query is skipped instead
of stalling the merged view.
Measured on a live node with a dead "New conversations" follow on a
xite missing its `message` table: feedQuery 60.2s -> 0.36s cold /
0.01s warm, identical rows.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The dashboard's middle feed panel (Latest/Post/Comment/Topic/Mention) took minutes to load while the left menu was instant. Measured over the dashboard's websocket: a single
feedQuerycall took 60.2s.Root cause chain:
message) that the followed xite's database no longer has — the follow's SQL snapshot outlived the xite's schema.db_queryclassifiesno such table/no such columnas transient and retries for the full 60-second budget at 200ms intervals. That classification exists for the rebuild window (tables dropped and recreated under a stable handle), but on a settled database the name will never appear — so every call burned the entire budget (~300 retries of log spam per call).feedQueryran each followed xite's queries serially, so the one dead query blocked every other xite's rows behind it, and each dashboard tab interaction re-paid the stall.Fix
A — fail fast on permanently missing schema (
state.rs)is_missing_schema_error()anddb_rebuild_or_clone_in_flight()(the same busy signalawait_db_quietpolls:is_cloning+db_rebuilds_in_flight).settle_stable_db_resultnow retries a missing table/column only while a clone/rebuild is actually redoing the xite's schema; on a quiet database it answers with the error immediately. Race-safe: a finished rebuild bumpsdb_generation, so a query that raced one gets a changed receipt and lands in the rerun branch, never this fail-fast path. Contention errors (locked/busy) retry exactly as before.B — concurrent feed queries with a per-query deadline (
command.rs)feedQuerycollects all followed feeds' SQL first, then runs every query concurrently (join_all), each under its own 10stokio::time::timeout. A timed-out or failed query is skipped, so one sick xite can never serialize-stall the whole merged view.Verification
cargo check -p epix-uiclean; 28 related tests pass, includingfeed_query_aggregates_followed_xites(exercises the rewritten handler) and thedb_queryreceipt tests.feedQuery60.2s → 0.36s cold / 0.01s warm, identical rows returned. Log now shows a singleschema settled, not retryingline per call instead of ~300 retry lines.