Skip to content

Stop re-reading the whole xite tree at startup - #437

Merged
MudDev merged 3 commits into
mainfrom
fix/extern-revalidate-stamp
Aug 27, 2026
Merged

Stop re-reading the whole xite tree at startup#437
MudDev merged 3 commits into
mainfrom
fix/extern-revalidate-stamp

Conversation

@MudDev

@MudDev MudDev commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

A node holding a large xite was unusable for tens of minutes after every
restart. Not slow to sync - every app page hung on its loading screen,
because dbQuery did not return at all.

What was happening

set_edx_store takes the global xite_activation_gate for write and
holds it while it registers every served xite. dbQuery needs that same
gate for read, via recheck_db_query_receipt, on every iteration of
its retry loop. So for as long as registration runs, every dbQuery on
every xite blocks - the retry budget never even starts counting, because
the await is on the lock.

Registration read the tree twice:

  1. revalidate on every extern object. The only cheap check was the file
    length; everything else was a full BLAKE3 re-read of every byte.
  2. register_entry calls ObjId::of_file on every declared file,
    hashing it end to end purely to rediscover an id the manifest already
    states - and only then adopting it, which on any boot after the first
    is a no-op.

Measured on a node with a 232 GB xite: 411 GB read in the 25 minutes
after a restart
, with siteInfo answering in 3 ms and bad_files: 0
the whole time while dbQuery had not returned after 240 seconds. A
gateway with 18 small xites finishes registration in seconds and looks
perfectly healthy, which is what makes this easy to misread as an
application bug.

The fix

Pass 1 - record the (len, mtime) a file had when its bytes last
verified, in a new local extern_stamp table, and skip the re-read while
both still match. Absent, malformed or different falls back to the full
check, so this trades a re-read for a stat, never verification for
trust. Every write to the extern mapping clears the stamp, so a relocated
or retired object can never be judged by a stamp describing another file;
only a completed verification writes one back.

Pass 2 - is_extern_at asks the store whether an id is already
adopted, complete, and backed by exactly this path. Registration uses it
to skip the rediscovery hash. Anything less - different path, incomplete,
not an extern, outside the xite root - answers false and falls through to
the original hash-and-adopt. A file that genuinely changed is still
caught, because revalidate retires it before this point.

The extern_stamp table is local like extern itself: never signed,
never transmitted, safe to discard.

Result on the affected node

time read
before 34m 22s 497 GB
after 1m 53s 1.54 GB

Same 3515 objects from 23 xites registered either way. dbQuery now
answers in 1-16 ms during startup instead of blocking, and the page
that hung on six loading skeletons renders identically to the gateway.

The first boot after upgrading still does one full pass, because no
stamps exist yet; it records them as it goes.

Tests

Two new ones. revalidate_skips_rereading_an_unchanged_extern corrupts
the bytes but restores the mtime - answering "complete" is only possible
without reading - then moves only the mtime and asserts it is rejected.
is_extern_at_answers_only_for_the_exact_adopted_file covers the wrong
path, an unknown object, and a path outside the xite root.

Also drops a never_loop in epix-reticulum/tests/mesh.rs that failed
cargo clippy --all-targets (deny-by-default) while the build passed.

MudDev added 3 commits August 26, 2026 22:18
Revalidating an extern object re-read and re-verified every byte of its
file against the stored outboard. The only cheap check was the length, so
an unchanged file cost a full read.

Startup registers every served xite, and that registration runs while
`set_edx_store` holds the global activation gate for write. `dbQuery`
needs that gate for read (via `recheck_db_query_receipt`), so for as long
as registration runs, every dbQuery on every xite blocks.

On a node holding a large xite that is not a slow start, it is an outage.
The media box holds a 234 GB xite: it read 411 GB in the 25 minutes after
a restart, and for that whole window app pages sat on their loading
screen because their first query never returned. Small nodes never see
it, which is why the gateway looks fine.

Record the (len, mtime) a file had when its bytes last verified, in a new
local `extern_stamp` table, and skip the re-read while both still match.
A missing, malformed or different stamp falls back to the full check
exactly as before, so this only ever trades a re-read for a stat, never
verification for trust. The table is local like `extern` itself: never
signed, never transmitted, safe to discard.

Every write to the extern mapping clears the stamp, so a relocated or
retired object can never be judged by a stamp describing another file.
Only a completed end-to-end verification writes one back.

The first boot after this change still does one full pass, because no
stamps exist yet; it records them as it goes, and later boots are stats.
clippy's never_loop is deny-by-default, so this failed
`cargo clippy --all-targets` while the build and tests passed.
The body reads one announce and breaks with it, so the loop was
only ever wrapping a single await.
Startup registration read the tree twice. The stamp change removed
revalidate's pass; this removes the other one.

`register_entry` calls `ObjId::of_file` on every non-bundleable file a
manifest declares, hashing it end to end purely to check the result
equals the id the manifest already states, and only then adopts it. On
the second and every later boot that file is already adopted as an extern
at that exact path, and the revalidate loop just above has re-proven it,
so the hash rediscovers an id the store is holding right there.

Ask the store instead. `is_extern_at` answers whether an id is adopted,
complete, and backed by exactly this path; anything less - different
path, incomplete, not an extern, outside the xite root - answers false
and falls through to the original hash-and-adopt, so a file that really
did change is still caught by revalidate retiring it first.

Measured on the media box (232 GB xite): the pre-fix binary read 411 GB
in the 25 minutes after a restart, and dbQuery was still blocked at 240s.
With only the stamp fix a boot still read ~3 GB/min through this path.
@sonarqubecloud

Copy link
Copy Markdown

@MudDev
MudDev merged commit a62985b into main Aug 27, 2026
12 checks passed
@MudDev
MudDev deleted the fix/extern-revalidate-stamp branch August 27, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant