Skip to content

perf: patch the snapshot on drift, cache describe by version - #45

Merged
vreshch merged 1 commit into
masterfrom
perf/snapshot-and-describe
Aug 22, 2026
Merged

perf: patch the snapshot on drift, cache describe by version#45
vreshch merged 1 commit into
masterfrom
perf/snapshot-and-describe

Conversation

@vreshch

@vreshch vreshch commented Aug 22, 2026

Copy link
Copy Markdown
Member

Stacked on #44 (feature/read-many) - the perf fixture and the store touch the same files, and the property test bulk-reads through readMany. Review after #44, merge after #44.

Internal only: no contract change, no new verb, no output shape moves.

S-D items 3 and 6 of the simplification plan.

a. Snapshot patch-on-drift

detectDrift already computed the changed-path diff on every push and then dropped the snapshot, so the next query walked the entire commit history again. It now patches:

  • the diff (--name-status --no-renames) is the authority on what exists at the new version
  • a log of the range only (old..new) is the authority on when each path changed - a wider set than the diff, because a note edited and put back has no diff at all yet did age
  • rebuild is still the fallback: a non-fast-forward move (force-push, reset) or a changed path git cannot attribute to a commit (a merge) drops the snapshot as before

One visible side effect, deliberate: rename detection is off, so an external rename now reports the removed path AND the added one. Observers used to hear only the arrival.

b. describe() cached by version

Two spawns once per version, then free - the store-level version of the derived-cache pattern already used for stats. A returned card is a copy, so a caller's edit cannot become the cache.

c. Write path 7 -> 4/5: NOT done, and why

Measured on this machine, 60 writes, per stage of the 7:

stage ms/write share
cat-file --batch (existing doc) 1.7 13%
hash-object 1.8 13%
read-tree + update-index + write-tree 5.9 45%
commit-tree 1.9 15%
update-ref (CAS) 1.9 14%

The tree trio is the prize (~4ms/write if collapsed to one spawn), and each way to collapse it fails the "provably identical" bar:

  • mktree from the parent tree only builds ONE level. A nested path needs ls-tree + mktree per level (worse for work/deep/note.md, better only for root-level notes) and it must re-emit every sibling entry with its exact mode - a mistake silently drops files.
  • Encoding trees in JS (one cat-file --batch for the parent trees, one hash-object -t tree --stdin-paths) gets to 5 spawns but re-implements git's tree binary format, entry sort order and mode preservation. validateBareRepoTree exists precisely because repos do contain symlinks and odd modes; a bug here corrupts a user's repository.
  • git fast-import would do blob + commit + ref in one process, but it force-updates the ref (our compare-and-swap lost-update proof would be replaced by fast-import's own non-fast-forward refusal) and writes a packfile per invocation, on a store that already never gc's.

~4ms on the write verb, against corrupting a repo, on the item the plan itself ranked last. Not shipping it. Instead the cost is now pinned by a test that asserts the exact 7-spawn sequence (and that a no-op write stops after the first), so any future attempt is visible in the diff rather than hidden in the numbers.

Conformance-first

The kit carries every claim; both stores pass it.

  • ConformanceTarget.reopen (new, optional): another instance over the same storage. New proof - a store that followed the changes answers exactly like one that just opened: same list at three shapes, same search, same describe, same read.
  • Following an external change costs the change, not a rebuild (counted): drift is at most 3 round trips, and the listing after it costs zero.
  • describe is computed once per version, then free (counted), and still moves with a write.
  • The bare-git target's mutateExternally now cycles add -> rewrite -> delete, so the kit's external proofs see every kind of drift instead of one append.

New test/stores/snapshot-patch.test.ts:

  • property test over random push sequences (add / rewrite / restore-original / delete / rename-in-one-commit / force-push backwards), with and without eager refresh(): the followed store and a fresh one must be equal on list x3, search, describe and readMany. 80 runs locally, FUZZ_DIFF_RUNS in CI.
  • a divergent force-push rebuilds even when the diff looks patchable - the case the ancestry check exists for.
  • spawn shape: a fast-forward push is exactly diff, merge-base, log; a rewind shows ls-tree (rebuild).

Both guards were mutation-checked: dropping the ancestry check or dating from the diff instead of the range log makes them fail (the property test found the restore case on its own).

Perf (harness, same machine, PERF_SCALE=1000, PERF_COMMITS=4000)

The gate gained the commit-count axis the review asked for: a fast-import churn fixture (4000 commits over 200 notes; 20000 nightly). The drift row is priced against a rebuild measured in the same run, so it stays honest on a slow runner.

metric before after
external push -> fresh answer @4000 commits 45.7ms (rebuild) 7.8ms (rebuild in that run: 46ms)
external push -> fresh answer @20000 commits ~203ms 7.6ms
describe warm avg 5.3ms 0.1ms
cold first list @4000 commits 42.7ms 42.7ms (unchanged - the once-per-version walk stays)

Verify: type-check + lint + format + 260 tests + coverage + build + dist smoke green locally; nightly-scale run (20000 commits, 25 differential sequences) green too.

No version bump - the coordinator owns the release.

@vreshch
vreshch marked this pull request as ready for review August 22, 2026 19:43
@vreshch
vreshch changed the base branch from feature/read-many to master August 22, 2026 21:14
An incoming push already computes the changed-path diff, then threw the
snapshot away and made the next query walk the whole history again. It now
patches instead: the diff says what exists, a log of the range says when it
changed, and only a force-push or an unattributable change rebuilds.

describe() joins the derived pattern - two spawns once per version, then free,
so a page of vault cards costs one computation instead of one per card.

The perf gate gains the commit-count axis it was missing (a 4000-commit
fast-import fixture, 20000 nightly) and prices the patch against a rebuild
measured in the same run. The write path is left alone and pinned at 7 spawns
with its own budget test - see the PR body for why.
@vreshch
vreshch force-pushed the perf/snapshot-and-describe branch from 66d2951 to 38789e4 Compare August 22, 2026 21:15
@vreshch
vreshch merged commit dd153e1 into master Aug 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant