Skip to content

feat: WAL time-travel snapshots - #31

Open
mvanhorn wants to merge 6 commits into
tobi:mainfrom
mvanhorn:cursor/wal-time-travel-snapshots-a974
Open

feat: WAL time-travel snapshots#31
mvanhorn wants to merge 6 commits into
tobi:mainfrom
mvanhorn:cursor/wal-time-travel-snapshots-a974

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Aug 28, 2026

Copy link
Copy Markdown

Walkthrough

HyperFrames walkthrough of this PR.

Walkthrough

mp4

What this does

The WAL already keeps every state a repository has ever been in, and walgit wal materialize --at-seq already rebuilds one. That rewind was reachable only from a terminal with bucket credentials, because the code lived in crates/walgit-cli/src/wal_cmd.rs. This exposes the same replay over HTTP.

  1. The replay moved into walgit-wal (crates/walgit-wal/src/snapshot.rs), the crate the CLI and the server already share — not copy-pasted into ops.rs. On the way it gained the shape a non-terminal caller needs: it returns a Snapshot (at_seq, head_seq, from_seq, replayed entries, git_dir, refs/ref_count, packs with per-pack source) instead of printing; narration goes to an Arc<dyn Fn(String)> sink so the CLI prints and a server task emits notices; errors are WalError rather than anyhow; and no unwrap is left on the path. The whole replay runs on the bulk runtime (sync::on_bulk_runtime) so downloads, 32 MiB copies and git index-pack waits never sit on a caller's request workers (principle VI).

  2. POST /{owner}/{repo}/api/ops/snapshot?at_seq=<n> — a new OPS entry, id = "snapshot", params = ["at_seq"], mutating: false. It runs walgit_wal::snapshot_at as an ordinary task (narrated, attachable, interrupted by the D31 drain like every other unit) and writes to <cache.dir>/snapshots/<owner>/<name>/<seq>/. snapshot.json is written last, so a tree carrying it is complete and one without it is a partial run that gets rebuilt: the op is idempotent and answers {built: bool, snapshot: {…}}.

  3. GET /{owner}/{repo}/api/snapshot/{seq} — reads that rewind back with the overview's auth (require_read), no-store, 404 naming the op when this host has not built one. An unparsable {seq} is a 400 from the route itself.

  4. WAL tab: the ops table renders at_seq as a number field (defaulting to the head) plus a link to the JSON, because it is a value parameter and not a flag. Running, streaming and the recent-runs table are the existing runOp/SSE path, unchanged.

  5. SDK + docs: repo.snapshot(seq) and the Snapshot types in web/sdk/repos.ts; the endpoint, the op, the 404 and the ref cap in web/API.md (plus snapshot in the task-kinds list and the conformance checklist); the walgit_wal::snapshot surface in docs/CONTRACT.md; a budget row in docs/ROUNDTRIPS.md; the provenance line in AGENTS.md §2.4.

What it does not do

  • Live refs and the serving copy never move. Nothing is published, no WAL entry is written, no lease is taken. The answer carries head_seq so a caller can see where the serving copy still is, and the test asserts it (refs endpoint, manifest head, applied seq and the local copy's refs/heads/main all stay at seq 2 after a rewind to seq 1).
  • rematerialize is untouched. It has no at_seq and still means "catch this instance's copy up to the head".
  • No tar/archive endpoint (no such helper exists in the tree), no bundle-uri historical families, no smart-HTTP clone of a past sequence, no new store backend, no new auth path, no new webhook event, no new config key.
  • No retention for snapshot trees: they are a cache under cache.dir that an operator creates deliberately, and snapshot_at refuses to build one on an instance whose cache budget cannot hold even the live pack set (WalError::TooLarge → 503 with the existing message). Bounding them across sequences would be a separate change.
  • refs in the JSON is capped at 1000 entries with ref_count alongside, so a 466 k-ref repository does not turn a request path into a 30 MB document. The rebuilt copy at git_dir has all of them.

File pointers

Where What
crates/walgit-wal/src/snapshot.rs the replay (materialize_at), the cache-dir form (snapshot_at), snapshot_dir/read_snapshot, Snapshot/SnapshotRef/SnapshotPack, both unit tests
crates/walgit-wal/src/lib.rs pub mod snapshot + re-exports
crates/walgit-cli/src/wal_cmd.rs WalAction::Materialize now calls the shared function and prints the summary; 348 lines of replay and its test left the CLI
crates/walgit-server/src/ops.rs the snapshot OpSpec and arm; run takes the narrator as an owned Arc and derives the borrowed Log from it
crates/walgit-server/src/web/ui.rs the snapshot/{seq} route and handler
crates/walgit-server/src/web/v1.rs the route in /api/v1 discovery
crates/walgit-server/tests/snapshot.rs the server test (wired into just test)
web/sdk/repos.ts, web/src/pages/OverviewPage.tsx, web/src/styles.css SDK method + types, the at_seq field, its input style
web/API.md, web/sdk/README.md, docs/CONTRACT.md, docs/ROUNDTRIPS.md, AGENTS.md the contracts

Round trips

No hot path changes; docs/ROUNDTRIPS.md §2 gained one row for the rewind, which is operator-driven: one conditional manifest GET (the refs sync that validates at_seq against the head), then the checkpoint and its refs (2 GETs when one is at or before the cut), the log segments through the cut, and per pack not already local a striped pack ∥ idx ∥ advertised side-files. A sequence already materialized on the host costs nothing past the freshness GET. The sim suite's request budgets are unchanged and pass.

Tests

  • crates/walgit-wal/src/snapshot.rs: the existing cold-rewind test moved with the code (not deleted) and grew the assertions the returned Snapshot makes possible — pack provenance (store vs local), the replay start, the refs, and that the writer's copy kept its packs and its head. A second test covers the snapshot_at layout, idempotency (same built_at on the second call), rebuild of a tree whose marker is gone, and rejection of seq 0 and past-the-head sequences without leaving a tree behind.
  • crates/walgit-server/tests/snapshot.rs: two pushes (head seq 2) → POST ops/snapshot?at_seq=1 → the result's refs are the first commit, head_seq is 2, the rewound copy is a real repository at the first commit that does not contain the second, info/refs + manifest + applied seq + the local copy are all still at seq 2, GET snapshot/1 round-trips, GET snapshot/2 is a 404 naming the op, GET snapshot/nope is a 400, the second op run rebuilds nothing, and each bad at_seq (missing, empty, abc, 0, 99) is refused with its reason.

Run here on the pinned toolchain: cargo fmt --check clean; cargo test --workspace --lib --bins, cargo test -p walgit-store -p walgit-git -p walgit-wal -p walgit-bundle --tests, the walgit-server fast tier (now including --test snapshot), and --test sim all green; cargo build --workspace --all-targets has no rustc warnings; web pnpm run build (oxlint --deny-warnings + tsc --noEmit + both vite builds) passes.

Two pre-existing failures on this branch are also present on main at the same commit and are unrelated: e2e::lfs_roundtrip_when_available (verified failing on 6d8fa54 in a clean worktree here) and e2e::fetch_from_front_that_serves_the_base_remotely (the flake AGENTS.md documents; passes alone). cargo clippy -- -D warnings is also red on main with the pinned 1.97.1 toolchain — newer pedantic lints in walgit-proto, walgit-config and walgit-server/build.rs, ~1800 workspace-wide. The files this PR touches contribute none: crates/walgit-wal/src/snapshot.rs is clippy-clean, and crates/walgit-server/tests/snapshot.rs has 6 findings of the classes every sibling test file already carries (indexing may panic etc.: 70 in sim.rs, 61 in web_api.rs). Fixing the pre-existing set would be a separate change.

cursoragent and others added 6 commits August 28, 2026 16:53
`materialize_at` — replay the newest checkpoint at or before a sequence plus
the log entries through it, install the pack set that was live there — lived in
`walgit-cli/src/wal_cmd.rs`, so nothing but the CLI could rewind. Move it to
`walgit-wal::snapshot` (the crate both the CLI and the server already share)
and give it the shape a caller other than a terminal needs:

* it returns a `Snapshot` (at_seq, head_seq, from_seq, replayed entries, git
  dir, refs bounded at `REFS_MAX`, ref count, pack set with per-pack provenance)
  instead of printing;
* narration goes to an `Arc<dyn Fn(String)>` sink, so the CLI prints and a
  server task turns the lines into notices;
* errors are `WalError`, not `anyhow`, and no `unwrap` remains on the path;
* the whole replay runs on the bulk runtime — downloads, 32 MiB copies and
  `git index-pack` waits must not sit on a caller's request workers
  (AGENTS §5, principle VI).

`snapshot_at` is the second entry point over the same replay: it owns the
`<cache.dir>/snapshots/<owner>/<name>/<seq>/` layout, rejects a zero or
past-the-head sequence before touching the bucket, refuses to build a second
pack set on an instance whose cache budget cannot hold the live one, and is
idempotent — `snapshot.json` is written last, so a tree carrying it is complete
and a tree without it is a partial run that gets rebuilt.

The existing cold-rewind test moves with the code and grew the assertions the
returned `Snapshot` makes possible (pack provenance, replay start, refs); a
second test covers the `snapshot_at` layout, idempotency and seq validation.

Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
The rewind that `walgit wal materialize --at-seq` does is now reachable over
HTTP on the instance that answers:

* `POST …/ops/snapshot?at_seq=<n>` is a non-mutating op (`mutating: false`) that
  runs `walgit_wal::snapshot_at` as a task, narrated like every other op, and
  returns `{built, snapshot}`. Missing, unparsable, zero and past-the-head
  sequences are refused with the reason. Nothing is published and no live ref
  moves: the answer carries `head_seq`, which is where the serving copy stays.
* `GET …/api/snapshot/{seq}` reads that rewind back with the overview's auth
  (read). `no-store`, because a snapshot is a local cache like every other byte
  on disk; `404` naming the op when this host has not built one.

`ops::run` now takes the narrator as an owned `Arc` and derives the borrowed
`Log` from it, because the WAL crate's replay outlives the call and runs on
another runtime.

New test `tests/snapshot.rs` (wired into `just test`): two pushes, rewind to
seq 1, the result's refs are the first commit, the rewound copy is a real
repository that does not contain the second commit, `info/refs` and the local
copy are still at seq 2, the GET round-trips, a second run rebuilds nothing,
and each bad `at_seq` is refused without leaving a tree behind.

Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
* WAL tab: the `snapshot` op's `at_seq` is a value, not a flag, so the ops table
  renders a number field (defaulting to the head) next to it plus a link to
  `…/api/snapshot/<n>`. `VALUE_PARAMS` replaces the hard-coded `strategy` check.
  Everything else — running, streaming, the recent-runs table — is the existing
  `runOp`/SSE path.
* SDK: `repo.snapshot(seq)` and the `Snapshot`/`SnapshotRef`/`SnapshotPack`
  types (the UI stays an adapter over the SDK, D20).
* `web/API.md`: the endpoint with its shape, the op that builds it, the 404 and
  the ref cap; `snapshot` added to the task kinds and the conformance list.
* `docs/CONTRACT.md`: the `walgit_wal::snapshot` surface and the two routes;
  `WalError` gained the `Invalid`/`TooLarge` variants it already had in code.
* `docs/ROUNDTRIPS.md`: a budget row for the rewind (operator-driven, off the
  critical path) — freshness GET, checkpoint + refs, the segments through the
  cut, and per missing pack a striped pack ∥ idx ∥ side-files; an already
  materialized sequence costs nothing past the freshness GET.
* `AGENTS.md` §2.4: the provenance line now names the HTTP form of the rewind.

Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Two follow-ups on the lift: the local-pack copy builds `pack-<checksum>.<ext>`
itself instead of going through `file_name()` (the name it wants is the one the
lines below already build), and `walgit wal materialize` prints `ref_count`, not
the length of the capped inline list. The WAL tab's at_seq field also stops
nesting its explanation and the JSON link inside the input's own label.

Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
@mvanhorn
mvanhorn marked this pull request as ready for review August 28, 2026 17:19
@mvanhorn

mvanhorn commented Aug 28, 2026

Copy link
Copy Markdown
Author

HyperFrames walkthrough of this PR.

Walkthrough

mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants