feat: WAL time-travel snapshots - #31
Open
mvanhorn wants to merge 6 commits into
Open
Conversation
`materialize_at` — replay the newest checkpoint at or before a sequence plus the log entries through it, install the pack set that was live there — lived in `walgit-cli/src/wal_cmd.rs`, so nothing but the CLI could rewind. Move it to `walgit-wal::snapshot` (the crate both the CLI and the server already share) and give it the shape a caller other than a terminal needs: * it returns a `Snapshot` (at_seq, head_seq, from_seq, replayed entries, git dir, refs bounded at `REFS_MAX`, ref count, pack set with per-pack provenance) instead of printing; * narration goes to an `Arc<dyn Fn(String)>` sink, so the CLI prints and a server task turns the lines into notices; * errors are `WalError`, not `anyhow`, and no `unwrap` remains on the path; * the whole replay runs on the bulk runtime — downloads, 32 MiB copies and `git index-pack` waits must not sit on a caller's request workers (AGENTS §5, principle VI). `snapshot_at` is the second entry point over the same replay: it owns the `<cache.dir>/snapshots/<owner>/<name>/<seq>/` layout, rejects a zero or past-the-head sequence before touching the bucket, refuses to build a second pack set on an instance whose cache budget cannot hold the live one, and is idempotent — `snapshot.json` is written last, so a tree carrying it is complete and a tree without it is a partial run that gets rebuilt. The existing cold-rewind test moves with the code and grew the assertions the returned `Snapshot` makes possible (pack provenance, replay start, refs); a second test covers the `snapshot_at` layout, idempotency and seq validation. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
The rewind that `walgit wal materialize --at-seq` does is now reachable over
HTTP on the instance that answers:
* `POST …/ops/snapshot?at_seq=<n>` is a non-mutating op (`mutating: false`) that
runs `walgit_wal::snapshot_at` as a task, narrated like every other op, and
returns `{built, snapshot}`. Missing, unparsable, zero and past-the-head
sequences are refused with the reason. Nothing is published and no live ref
moves: the answer carries `head_seq`, which is where the serving copy stays.
* `GET …/api/snapshot/{seq}` reads that rewind back with the overview's auth
(read). `no-store`, because a snapshot is a local cache like every other byte
on disk; `404` naming the op when this host has not built one.
`ops::run` now takes the narrator as an owned `Arc` and derives the borrowed
`Log` from it, because the WAL crate's replay outlives the call and runs on
another runtime.
New test `tests/snapshot.rs` (wired into `just test`): two pushes, rewind to
seq 1, the result's refs are the first commit, the rewound copy is a real
repository that does not contain the second commit, `info/refs` and the local
copy are still at seq 2, the GET round-trips, a second run rebuilds nothing,
and each bad `at_seq` is refused without leaving a tree behind.
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
* WAL tab: the `snapshot` op's `at_seq` is a value, not a flag, so the ops table renders a number field (defaulting to the head) next to it plus a link to `…/api/snapshot/<n>`. `VALUE_PARAMS` replaces the hard-coded `strategy` check. Everything else — running, streaming, the recent-runs table — is the existing `runOp`/SSE path. * SDK: `repo.snapshot(seq)` and the `Snapshot`/`SnapshotRef`/`SnapshotPack` types (the UI stays an adapter over the SDK, D20). * `web/API.md`: the endpoint with its shape, the op that builds it, the 404 and the ref cap; `snapshot` added to the task kinds and the conformance list. * `docs/CONTRACT.md`: the `walgit_wal::snapshot` surface and the two routes; `WalError` gained the `Invalid`/`TooLarge` variants it already had in code. * `docs/ROUNDTRIPS.md`: a budget row for the rewind (operator-driven, off the critical path) — freshness GET, checkpoint + refs, the segments through the cut, and per missing pack a striped pack ∥ idx ∥ side-files; an already materialized sequence costs nothing past the freshness GET. * `AGENTS.md` §2.4: the provenance line now names the HTTP form of the rewind. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Two follow-ups on the lift: the local-pack copy builds `pack-<checksum>.<ext>` itself instead of going through `file_name()` (the name it wants is the one the lines below already build), and `walgit wal materialize` prints `ref_count`, not the length of the capped inline list. The WAL tab's at_seq field also stops nesting its explanation and the JSON link inside the input's own label. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
mvanhorn
marked this pull request as ready for review
August 28, 2026 17:19
Author
|
HyperFrames walkthrough of this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Walkthrough
HyperFrames walkthrough of this PR.
mp4
What this does
The WAL already keeps every state a repository has ever been in, and
walgit wal materialize --at-seqalready rebuilds one. That rewind was reachable only from a terminal with bucket credentials, because the code lived incrates/walgit-cli/src/wal_cmd.rs. This exposes the same replay over HTTP.The replay moved into
walgit-wal(crates/walgit-wal/src/snapshot.rs), the crate the CLI and the server already share — not copy-pasted intoops.rs. On the way it gained the shape a non-terminal caller needs: it returns aSnapshot(at_seq,head_seq,from_seq, replayedentries,git_dir,refs/ref_count,packswith per-packsource) instead of printing; narration goes to anArc<dyn Fn(String)>sink so the CLI prints and a server task emits notices; errors areWalErrorrather thananyhow; and nounwrapis left on the path. The whole replay runs on the bulk runtime (sync::on_bulk_runtime) so downloads, 32 MiB copies andgit index-packwaits never sit on a caller's request workers (principle VI).POST /{owner}/{repo}/api/ops/snapshot?at_seq=<n>— a new OPS entry,id = "snapshot",params = ["at_seq"],mutating: false. It runswalgit_wal::snapshot_atas an ordinary task (narrated, attachable, interrupted by the D31 drain like every other unit) and writes to<cache.dir>/snapshots/<owner>/<name>/<seq>/.snapshot.jsonis written last, so a tree carrying it is complete and one without it is a partial run that gets rebuilt: the op is idempotent and answers{built: bool, snapshot: {…}}.GET /{owner}/{repo}/api/snapshot/{seq}— reads that rewind back with the overview's auth (require_read),no-store,404naming the op when this host has not built one. An unparsable{seq}is a 400 from the route itself.WAL tab: the ops table renders
at_seqas a number field (defaulting to the head) plus a link to the JSON, because it is a value parameter and not a flag. Running, streaming and the recent-runs table are the existingrunOp/SSE path, unchanged.SDK + docs:
repo.snapshot(seq)and theSnapshottypes inweb/sdk/repos.ts; the endpoint, the op, the 404 and the ref cap inweb/API.md(plussnapshotin the task-kinds list and the conformance checklist); thewalgit_wal::snapshotsurface indocs/CONTRACT.md; a budget row indocs/ROUNDTRIPS.md; the provenance line inAGENTS.md§2.4.What it does not do
head_seqso a caller can see where the serving copy still is, and the test asserts it (refs endpoint, manifest head, applied seq and the local copy'srefs/heads/mainall stay at seq 2 after a rewind to seq 1).rematerializeis untouched. It has noat_seqand still means "catch this instance's copy up to the head".cache.dirthat an operator creates deliberately, andsnapshot_atrefuses to build one on an instance whose cache budget cannot hold even the live pack set (WalError::TooLarge→ 503 with the existing message). Bounding them across sequences would be a separate change.refsin the JSON is capped at 1000 entries withref_countalongside, so a 466 k-ref repository does not turn a request path into a 30 MB document. The rebuilt copy atgit_dirhas all of them.File pointers
crates/walgit-wal/src/snapshot.rsmaterialize_at), the cache-dir form (snapshot_at),snapshot_dir/read_snapshot,Snapshot/SnapshotRef/SnapshotPack, both unit testscrates/walgit-wal/src/lib.rspub mod snapshot+ re-exportscrates/walgit-cli/src/wal_cmd.rsWalAction::Materializenow calls the shared function and prints the summary; 348 lines of replay and its test left the CLIcrates/walgit-server/src/ops.rssnapshotOpSpec and arm;runtakes the narrator as an ownedArcand derives the borrowedLogfrom itcrates/walgit-server/src/web/ui.rssnapshot/{seq}route and handlercrates/walgit-server/src/web/v1.rs/api/v1discoverycrates/walgit-server/tests/snapshot.rsjust test)web/sdk/repos.ts,web/src/pages/OverviewPage.tsx,web/src/styles.cssat_seqfield, its input styleweb/API.md,web/sdk/README.md,docs/CONTRACT.md,docs/ROUNDTRIPS.md,AGENTS.mdRound trips
No hot path changes;
docs/ROUNDTRIPS.md§2 gained one row for the rewind, which is operator-driven: one conditional manifest GET (the refs sync that validatesat_seqagainst the head), then the checkpoint and its refs (2 GETs when one is at or before the cut), the log segments through the cut, and per pack not already local a striped pack ∥ idx ∥ advertised side-files. A sequence already materialized on the host costs nothing past the freshness GET. The sim suite's request budgets are unchanged and pass.Tests
crates/walgit-wal/src/snapshot.rs: the existing cold-rewind test moved with the code (not deleted) and grew the assertions the returnedSnapshotmakes possible — pack provenance (storevslocal), the replay start, the refs, and that the writer's copy kept its packs and its head. A second test covers thesnapshot_atlayout, idempotency (samebuilt_aton the second call), rebuild of a tree whose marker is gone, and rejection of seq 0 and past-the-head sequences without leaving a tree behind.crates/walgit-server/tests/snapshot.rs: two pushes (head seq 2) →POST ops/snapshot?at_seq=1→ the result's refs are the first commit,head_seqis 2, the rewound copy is a real repository at the first commit that does not contain the second,info/refs+ manifest + applied seq + the local copy are all still at seq 2,GET snapshot/1round-trips,GET snapshot/2is a 404 naming the op,GET snapshot/nopeis a 400, the second op run rebuilds nothing, and each badat_seq(missing, empty,abc,0,99) is refused with its reason.Run here on the pinned toolchain:
cargo fmt --checkclean;cargo test --workspace --lib --bins,cargo test -p walgit-store -p walgit-git -p walgit-wal -p walgit-bundle --tests, thewalgit-serverfast tier (now including--test snapshot), and--test simall green;cargo build --workspace --all-targetshas no rustc warnings;webpnpm run build(oxlint--deny-warnings+tsc --noEmit+ both vite builds) passes.Two pre-existing failures on this branch are also present on
mainat the same commit and are unrelated:e2e::lfs_roundtrip_when_available(verified failing on6d8fa54in a clean worktree here) ande2e::fetch_from_front_that_serves_the_base_remotely(the flake AGENTS.md documents; passes alone).cargo clippy -- -D warningsis also red onmainwith the pinned 1.97.1 toolchain — newer pedantic lints inwalgit-proto,walgit-configandwalgit-server/build.rs, ~1800 workspace-wide. The files this PR touches contribute none:crates/walgit-wal/src/snapshot.rsis clippy-clean, andcrates/walgit-server/tests/snapshot.rshas 6 findings of the classes every sibling test file already carries (indexing may panicetc.: 70 insim.rs, 61 inweb_api.rs). Fixing the pre-existing set would be a separate change.