Stop re-reading the whole xite tree at startup - #437
Merged
Conversation
Revalidating an extern object re-read and re-verified every byte of its file against the stored outboard. The only cheap check was the length, so an unchanged file cost a full read. Startup registers every served xite, and that registration runs while `set_edx_store` holds the global activation gate for write. `dbQuery` needs that gate for read (via `recheck_db_query_receipt`), so for as long as registration runs, every dbQuery on every xite blocks. On a node holding a large xite that is not a slow start, it is an outage. The media box holds a 234 GB xite: it read 411 GB in the 25 minutes after a restart, and for that whole window app pages sat on their loading screen because their first query never returned. Small nodes never see it, which is why the gateway looks fine. Record the (len, mtime) a file had when its bytes last verified, in a new local `extern_stamp` table, and skip the re-read while both still match. A missing, malformed or different stamp falls back to the full check exactly as before, so this only ever trades a re-read for a stat, never verification for trust. The table is local like `extern` itself: never signed, never transmitted, safe to discard. Every write to the extern mapping clears the stamp, so a relocated or retired object can never be judged by a stamp describing another file. Only a completed end-to-end verification writes one back. The first boot after this change still does one full pass, because no stamps exist yet; it records them as it goes, and later boots are stats.
clippy's never_loop is deny-by-default, so this failed `cargo clippy --all-targets` while the build and tests passed. The body reads one announce and breaks with it, so the loop was only ever wrapping a single await.
Startup registration read the tree twice. The stamp change removed revalidate's pass; this removes the other one. `register_entry` calls `ObjId::of_file` on every non-bundleable file a manifest declares, hashing it end to end purely to check the result equals the id the manifest already states, and only then adopts it. On the second and every later boot that file is already adopted as an extern at that exact path, and the revalidate loop just above has re-proven it, so the hash rediscovers an id the store is holding right there. Ask the store instead. `is_extern_at` answers whether an id is adopted, complete, and backed by exactly this path; anything less - different path, incomplete, not an extern, outside the xite root - answers false and falls through to the original hash-and-adopt, so a file that really did change is still caught by revalidate retiring it first. Measured on the media box (232 GB xite): the pre-fix binary read 411 GB in the 25 minutes after a restart, and dbQuery was still blocked at 240s. With only the stamp fix a boot still read ~3 GB/min through this path.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



A node holding a large xite was unusable for tens of minutes after every
restart. Not slow to sync - every app page hung on its loading screen,
because
dbQuerydid not return at all.What was happening
set_edx_storetakes the globalxite_activation_gatefor write andholds it while it registers every served xite.
dbQueryneeds that samegate for read, via
recheck_db_query_receipt, on every iteration ofits retry loop. So for as long as registration runs, every
dbQueryonevery xite blocks - the retry budget never even starts counting, because
the await is on the lock.
Registration read the tree twice:
revalidateon every extern object. The only cheap check was the filelength; everything else was a full BLAKE3 re-read of every byte.
register_entrycallsObjId::of_fileon every declared file,hashing it end to end purely to rediscover an id the manifest already
states - and only then adopting it, which on any boot after the first
is a no-op.
Measured on a node with a 232 GB xite: 411 GB read in the 25 minutes
after a restart, with
siteInfoanswering in 3 ms andbad_files: 0the whole time while
dbQueryhad not returned after 240 seconds. Agateway with 18 small xites finishes registration in seconds and looks
perfectly healthy, which is what makes this easy to misread as an
application bug.
The fix
Pass 1 - record the
(len, mtime)a file had when its bytes lastverified, in a new local
extern_stamptable, and skip the re-read whileboth still match. Absent, malformed or different falls back to the full
check, so this trades a re-read for a
stat, never verification fortrust. Every write to the extern mapping clears the stamp, so a relocated
or retired object can never be judged by a stamp describing another file;
only a completed verification writes one back.
Pass 2 -
is_extern_atasks the store whether an id is alreadyadopted, complete, and backed by exactly this path. Registration uses it
to skip the rediscovery hash. Anything less - different path, incomplete,
not an extern, outside the xite root - answers false and falls through to
the original hash-and-adopt. A file that genuinely changed is still
caught, because revalidate retires it before this point.
The
extern_stamptable is local likeexternitself: never signed,never transmitted, safe to discard.
Result on the affected node
Same 3515 objects from 23 xites registered either way.
dbQuerynowanswers in 1-16 ms during startup instead of blocking, and the page
that hung on six loading skeletons renders identically to the gateway.
The first boot after upgrading still does one full pass, because no
stamps exist yet; it records them as it goes.
Tests
Two new ones.
revalidate_skips_rereading_an_unchanged_externcorruptsthe bytes but restores the mtime - answering "complete" is only possible
without reading - then moves only the mtime and asserts it is rejected.
is_extern_at_answers_only_for_the_exact_adopted_filecovers the wrongpath, an unknown object, and a path outside the xite root.
Also drops a
never_loopinepix-reticulum/tests/mesh.rsthat failedcargo clippy --all-targets(deny-by-default) while the build passed.