Add worktree and branch graph views - #654
Open
zzet wants to merge 175 commits into
Open
Conversation
Enumerate a repository's worktrees with git worktree list --porcelain -z so paths with spaces or newlines parse correctly, resolve git and common directories through plumbing, and derive each linked worktree's administrative name. Failed enumeration returns a typed error so callers cannot mistake it for worktree removal; an inaccessible root is reported per record without failing the inventory. Also sample HEAD state (attached, detached, unborn) and record volume identity plus nearest-ancestor evidence for later availability classification.
Route GetOutEdges, GetInEdges, both batched variants, and AllEdges through one baseEdgeVisible predicate so every reader agrees on which base edges survive an overlay: an edge to a re-emitted target stays, an edge to a hidden or tombstoned target is dropped. GetRepoNodes no longer returns the base copy of an ID the overlay re-emitted. Stats and RepoStats fold overlay node and edge deltas instead of reporting base-only totals; per-kind and per-language breakdowns stay base-derived and say so. The query engine rehydrates bundle candidates through the active reader and skips the base edge pre-seed when an overlay is installed, so replaced payloads and deleted symbols cannot resurface from cached bundles; the no-overlay fast path is unchanged. Worktree instance tests now assert result and metadata counts, not just name shape.
New package with the core vocabulary for request-pinned graph views: canonical sha256 fingerprints over length-prefixed encodings for repo and workspace view identities, capability states with required/optional evaluation, stable error codes usable with errors.Is and errors.AsType, strict full-ref and commit OID selector validation, and a refcounted generation lease manager with context-aware drain and no polling.
Bump the go.mod directive to 1.27.0; every workflow reads the Go version from go.mod, so CI follows. Move the ci test matrix and its coverage upload guard to 1.27, the goreleaser-cross image and eval-build container to the matching tag, and the golangci-lint pin to 2.13.1 (config schema unchanged, zero new findings). Reformat the two files the 1.27 gofmt flags for return-statement continuation indentation.
Add the control-plane catalog to the SQLite store: repository families, checkouts with incarnation and dual availability/removal clocks, tracking intents with per-source provenance, single-slot intent transitions, path evidence, dedicated graphs with a partial unique primary per family, view generations and layers, checkout routes, ref views with coalesced builds, and a cleanup journal with no foreign keys. Guarded transitions run as single compare-and-swap transactions: checkout state updates check incarnation, route flips check route_epoch, primary flips check the family primary_epoch, and publishing a generation only moves building to ready. Deleting a routed or based generation is refused in Go and backstopped by foreign keys, which every connection enforces. Schema version moves to 13 with an idempotent in-place migration.
New package with three implementations of one ContentSource interface. FilesystemSource confines every access to its root with os.Root plus lexical pre-checks, so path traversal and symlink escapes fail closed. GitTreeSource enumerates a tree with NUL-delimited ls-tree and reads blobs through one long-lived cat-file batch child per source, preferring the NUL-framed batch-command dialect and falling back to newline framing fed only regex-validated hex object ids; children run with lazy fetch disabled and prompts off, and a missing object is a typed error distinct from an absent path. LayeredSource routes reads by an ownership predicate and merges walks deterministically.
New package that reconciles a repository family against its worktree inventory. Identity is matched by family and admin name, allocated with a guarded insert so racing passes cannot mint duplicates, and reused across inaccessibility. Removal classification is conservative: a failed inventory or any permission, IO, unknown-volume, or mismatched-ancestor condition classifies as inaccessible; only validated inventory omission or a prunable record backed by same-volume path evidence counts as removal. Availability and removal run on independent durable clocks that survive restart. Cleanup runs as journaled sagas (forget checkout, retire primary closure, forget family) that resume idempotently after a crash, guard on incarnation and primary epoch, and verify a zero-row postcondition before the journal deletes itself last. Catalog gains the listing, guarded-allocation, observation, and delete accessors the sagas need.
Add EdgesByKind and NodesByKind to the reader interface, matching the store signatures, and implement them on the overlaid view with the same visibility rules as the bulk readers: base rows filtered by the shared edge-visibility predicate, layer rows added on top, exactly one copy per re-emitted id with the layer payload winning.
Route request-serving graph reads in tool, resource, and prompt handlers through the request reader instead of the base store, so an installed editor overlay is reflected across navigation, analyze, review, quality, and export surfaces. Resource and prompt handlers now get the overlay view installed at all; previously only tools did. Read-only entry points in analysis, analyzer, review, wiki, and docs widen from the mutable store to the reader, and capability assertions run against the request reader so an overlay degrades to the exact composed fallback instead of serving stale base payloads. Lifecycle, enrichment writes, cache administration, and index-health reporting stay on the base store by design. Thirty overlay regression tests pin the new behavior.
Schema v14 adds view_gen INTEGER NOT NULL DEFAULT 0 to edges, both in the fresh-store DDL and as an idempotent in-place migration probed through pragma_table_xinfo. No index or query changes; every existing row lands at generation zero.
Rename the store struct to an unexported core and make the public handle embed it alongside a pinned view generation. Open returns the owning handle; AtGeneration derives a handle sharing the same core, and only the owner tears down pools, statements, and the checkpoint loop on Close. Zero-value handles keep their documented never-panic behavior with an explicit nil-core gate. No SQL changes.
Widen the remaining read-only entry points to the reader: diff mapping and joining, PR risk, impact (with a store-gated reach acceleration that falls back to live edge walks under an overlay), review runs, change verification, guard and architecture evaluation, rule families, wakeup digests, hotspots, and the rerank graph-completion retriever. Thread the request reader at every remaining call site, align the two change verification sites on the request engine, and mark the deliberate base-store reads: index-writing engines (dataflow, callpath, reach), enrichment writers, simulation base sides, and indexer-stamped metadata each carry a one-line statement of why base is correct. Handler-driven regression tests replace the two compile-only ones and cover every newly widened surface; each was verified to fail with the wiring reverted.
A file node's id is its bare path, which the file-prefix helper maps to an empty string, so overlay ownership checks never matched file nodes: point lookups returned the base copy while batched lookups returned the layer copy, kind and bulk readers returned both copies, and a tombstoned file node survived. Ownership now resolves a separator-free id as the file whose path equals the id, checked against the layer's covered-path set, across point, batched, bulk, bounded-adjacency, and shadow-source paths. Two tests that only passed through the duplicate are corrected, and a file-node visibility matrix pins the behavior; every new case fails against the old ownership check.
Layer nodes carried only a repo prefix, so in a workspace whose slug differs from the prefix, scope-narrowed reads dropped every buffer symbol and text-search attribution lost matches in covered files. The layer builder now resolves one identity donor per staged file - the base file node, else the smallest-id base sibling with a slug, else the repo indexer's own binding - and stamps workspace and project slugs the same fill-when-empty way the indexer does. Regression tests cover the donor and fallback arms and a scope-narrowed read keeping overlay nodes; each fails without the stamp.
Schema v15 rebuilds the eighteen sidecar tables with a leading view_gen primary-key column, driven by one shared registry that feeds both the fresh-store DDL and the in-place migration so the two shapes cannot drift. Scoped reads, writes, and deletes bind the handle's pinned generation; upsert conflict targets follow the new keys; repository purge, orphan scans, and prefix rekeys stay deliberately unscoped across generations and say so. Symbol and content FTS docid maps carry the generation while docids stay globally unique, and the raw index-state side door pins generation zero explicitly. Sidecar indexes keep their names, gain the leading generation column where they serve scoped reads, and are created after migrations run. A derived-handle smoke test proves generation-one writes are invisible to the base handle.
Schema v16 rebuilds the two core tables so the primary key of nodes is (id, view_gen) and the edge identity constraint is the five-column coordinate plus view_gen, with the generation trailing so every existing index prefix, seek, and free ordering is preserved plan-for-plan; all plan fences pass unmodified. The edge autoincrement sequence survives the rebuild, generated and promoted columns are recreated by the shared ensure helpers, and core index creation moves after migrations next to the sidecar indexes. Node and edge inserts bind the handle's generation and conflict targets follow the new keys; the JSONB batch path is renumbered and stays equality-gated against the placeholder writer. Point lookups and edge-existence checks are generation-exact, so a derived handle reads its own rows and generation zero point reads match the base handle; broader read scoping arrives with the leading-key flip in a later change.
Every read and write over nodes and edges now binds the handle's pinned generation as a residual predicate: prepared lookups, adjacency, aggregates with paired-generation endpoint joins, the BFS expansion, the dead-code and integrity probes, unresolved-frontier pages, keyset and high-water cursors, projections, and the symbol and content search joins through their generation-carrying docid maps. Repository purge, prefix rekey, orphan scans, and schema migrations stay deliberately generation-unscoped administration. Two partial indexes serve sparse generation enumeration without touching base plans, the bundle cache and builtin-stub memoization key by generation, and schema v17 records the additions. A leak fence seeds divergent corpora at two generations and drives every store read method and capability interface, asserting each handle sees only its own generation; writer families get a disturb table proving generation-one mutations leave generation zero byte-identical.
A single lifecycle component now owns registration, untrack, reload, janitor sweeps, startup seeding, and implicit auto-index observation. Both track surfaces converge on one registration helper, so the CLI path gains session invalidation and the MCP path gains watcher attachment; untrack revokes intent sources with an all-or-nothing preflight and runs the journaled forget saga with watcher detach before eviction; reload registers additions, records pending transitions instead of deleting silently, and follows the watcher diff; the janitor replaces the stat-based worktree scan with evidence-classified reconciliation over families from both the corpus and the configuration, persisting config after cleanup; startup seeds catalog identities for existing installs without re-indexing and never resets durable clocks. Implicit auto-indexing records family and checkout rows without minting tracking intent and stays out of the persisted config across restarts.
Parse git status porcelain v2 NUL records into typed entries: staged and unstaged changes, deletions, untracked files, renames decomposed into a delete plus an add carrying the old path, unmerged paths, and submodule flags. Kind classification reads the mode columns before the status codes, since a bare chmod reports identical blobs on both sides, and a symlink transition wins over a plain mode flip. The snapshot fingerprint hashes a length-prefixed encoding of head, entries, and lstat size and mtime evidence, so consecutive identical samples match and any content, mode, or staging change differs; observation runs with optional locks disabled so sampling never writes the index.
Schema v18 adds the four sparse-ownership tables: file masks with replace or delete modes, node tombstones, edge-source replacement markers, and per-producer completeness, all keyed by a leading generation with no default so a base-generation mask is a constraint error rather than a silent claim. The lifecycle composes them: begin creates a building catalog row and hands back a pinned handle that reuses the existing write paths; publish seals the generation, drains in-flight writers, validates mask consistency and producer states, records rollups, and flips building to ready under the guarded CAS, restoring an unknown seal on any failure so a lost race can never reopen writes to a published generation; routing flips a named route slot with the epoch CAS, leaving a losing generation ready but unrouted; retire refuses routed, based, or leased generations, then sweeps every generation-keyed row family in bounded self-shrinking chunks, deleting FTS documents and their docid maps in the same chunk transaction so multi-chunk corpora cannot leak. The end-to-end test builds a divergent overlay generation, routes it, reads both corpora through their handles, retires it, and proves the base byte-identical with every generation row gone.
…sibility Extract the read surface the overlaid view consumes into an interface so any layer implementation composes through the same verified semantics; the in-memory layer implements it unchanged and the consistency matrix becomes a conformance harness any implementation can run. Node visibility now uses one ownership predicate across point, bulk, name, file, and count readers, so a tombstone on an identity in an uncovered file hides the node everywhere instead of only in batched reads, and a detached re-emitted identity resolves to the layer payload on every path. Node and repo count deltas charge hidden detached identities in one batched base read, keeping counts consistent with iteration.
A generation layer implements the overlay contract from a handle pinned to one published generation plus its ownership masks: replace and delete file masks cover paths, delete reports removal, node tombstones remove identities without claiming files, and edge-source markers own the outgoing sets of unchanged dependents. Membership probes prefetch whole at construction and point reads memoize for the layer's immutable lifetime. Composition nests the overlaid view bottom to top, and the materializer builds a checkout's stack from its route slots, leases every referenced generation until close, and unions producer completeness upward. A differential test proves the composed stack equal to a flat index of the same final tree across nodes, both edge directions, file lists, kind iteration, and counts, including the tombstone and unchanged-dependent lanes; retire refuses while a materialized view holds its lease.
An indexer can carry a content source; when set, file reads, sniff prefixes, module manifests, contract sources, embed chunking, and the walk all answer from the source instead of the working tree, and the read-version receipt degrades to a single snapshot version since the source is immutable. Oversize entries surface from the source walk and mint the same size-skip stubs as the filesystem walk. Per-directory ignore files and the untracked-asset gate are filesystem concepts and go inert under a source, with the layered config excludes still applied. The os-backed paths are unchanged, guarded by an equivalence test between a source-backed and a filesystem index of the same fixture.
A sparse generation builder runs the production pipeline against a handle pinned to a fresh generation: it computes the affected closure of the changed set against the base corpus with a capped reverse-reference walk, widens the file set so cross-file resolution binds inside the generation, indexes through an isolated indexer instance sharing only process-wide admissions, emits replace and delete masks with node tombstones where retargeting removes base identities, declares per producer completeness honestly, and publishes. The commit builder feeds it a NUL-safe two-tree diff with renames decomposed and reads content from the target tree; the dirty builder maps the porcelain snapshot, lets the disk demote claims git staging still carries, and re-samples the fingerprint before publish, marking a torn build superseded with a typed retryable error. Acceptance proves a built commit layer composed over base equals a flat index of the target tree, resolver output included, and the dirty layer equals a flat index of the working tree.
A capped, parameterized listing over view generations filters by state, checkout, graph, and owner kind, ordered newest first so a layer always precedes the generation it sits on; the single-row read shares the scan with the listing so the projections cannot drift.
One coordinator per automatic checkout debounces watcher, poller, and reconcile signals, samples head and dirty state, and reconciles the route: a commit layer is adopted by identity when a generation for the same tree and fingerprints exists, so switching a branch away and back flips routes with zero rebuilds; the dirty layer rebuilds only when the sampled fingerprint or its underlying commit generation changed, retries a torn build once from a fresh sample, and leaves the previous route intact on repeated supersede. Moving the commit slot while a dirty generation is routed clears both slots in one compare-and-swap so no request ever composes a working-tree layer over a different tree. Replaced, superseded, and torn generations are offered for retirement newest first, a dying coordinator drains its backlog to the lifecycle, and the janitor sweep additionally collects crash-orphaned generations from the catalog, skipping checkouts with live coordinators; the store guards stay the only deletion authority. Coordinators come up when a family reconciles, at registration, and at boot seeding.
A request resolves its view before handlers run: an explicit selector wins over the session binding, which wins over the base corpus. A session inside a live, fully routed automatic checkout gets the composed generation stack as its request reader with the buffer overlay composed on top; the worktree and base selectors validate scope before readiness so out-of-scope probes learn nothing; ref and commit selectors validate then report themselves not yet buildable; a bound checkout whose route is building serves the base with exact false and a fallback reason, never silently. Request leases come from the lifecycle manager, so a sweep cannot retire a generation an in-flight request is reading; leases release on every exit path. Mutating tools refuse routed views, since edits must land in the checkout that owns the path. Riders extend the freshness block with requested and actual view, exactness, and fallback reason.
The engine accepts the ordered generation sources a routed view carries and asks each generation's own corpus alongside the base, then composes: higher-layer ownership masks lower hits using the layer contract predicates, the highest layer wins per identity, sources interleave by rank position since scores from different corpora are not comparable, and every survivor rehydrates through the composed reader in one batched read. Every candidate lane follows: bundles, the channel fallback, the exact-name splice, both substring fallbacks, and both content-search call sites. The vector lane is inert under a composed view rather than serving base vectors as view vectors. The base path stays pointer identical with zero added allocations, pinned by tests; a symbol existing only in a sparse generation is found, and a deleted base symbol never surfaces even by exact name.
A session whose working directory sits inside a live automatic checkout previously resolved the unresolved-workspace sentinel, so scope-narrowed reads emptied the very results its routed view served. Scope resolution now recognizes the checkout through the same catalog binding view selection uses and recomputes the binding from the family primary's own root, making the worktree session's scope identical to a primary-root session: same workspace and project slugs, same ceiling, no revision dimension. Unknown paths, dedicated checkouts, unavailable states, and stores without a catalog keep the fail-closed sentinel; the binding rides the existing session cache and invalidation. A regression drives a scope-narrowed search from the worktree and finds the generation-only symbol while the deleted one stays hidden.
A layer hides a base edge when it claims the file the edge was recorded in, not whenever it claims the file the edge's source lives in: re-deriving a file re-derives the edges written in that file and no others, while a symbol's callers hold their edges in their own files. The whole-adjacency claim narrows to identities re-emitted outside any covered file and to explicit edge-source markers, outgoing readers merge base-surviving and layer rows instead of short-circuiting, and count deltas price each hidden edge individually. The conformance matrix encodes the corrected rule for both layer implementations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Gortex previously indexed one corpus per tracked repository and mutated it in place on every branch switch, and a sibling git worktree either got a full duplicate index or nothing. This PR gives every checkout and ref its own coherent, queryable view:
gortex repos(families,explain-view,forget,reconcile,set-primary) — there is nogortex checkoutscommand; the reference is indocs/cli.md.How it works
Storage. The SQLite payload schema is re-keyed by a view generation (schema v14–v18): nodes and edges carry the generation as the trailing identity column so every existing index and query plan stays byte-identical, sidecars lead with it, and two partial indexes serve sparse enumeration. Every statement binds the handle's pinned generation or is explicitly repository-wide administration. A leak fence seeds divergent corpora at two generations and drives every store read method and capability interface, proving isolation; a disturb table proves writer families cannot touch a neighbor generation.
Composition. The in-memory overlay's verified precedence semantics were extracted into a layer contract; persisted generations implement it over ownership masks (file replace/delete, node tombstones, edge-source markers), and views nest bottom-to-top: base, commit layer, dirty layer, editor buffers. A materializer builds a checkout's stack from its route, holds leases so retirement cannot pull a generation out from under a reader, and unions producer completeness upward.
Building. Sparse generations are produced by the production parse/resolve pipeline running against a pinned handle over a content source (checkout filesystem behind an os.Root-confined reader, or a git tree behind a long-lived cat-file batch child that can never fetch). The build closure iterates to a fixed point: reverse dependents of the change, forward dependencies, references the change introduces, and manifests unconditionally; cap truncation surfaces as producer incompleteness rather than silent divergence. A property-based oracle generates mutation scripts and asserts the composed view equals a flat index of the same tree — nodes, both edge directions, files, counts, and resolver output — across randomized seeds plus soak volume.
Serving. A request resolves its view before handlers run: explicit selector over session binding over base. Sessions inside a worktree bind the family scope, search enumerates candidates from every generation in the stack with mask-then-dedup-then-interleave composition, text search answers from the checkout's own root, and per-checkout language servers run under a global cap. Operations evaluate their required capabilities against the served view's completeness and answer with typed errors or degradation annotations — base-scoped engine results under a non-base view are labeled, never passed off as view-scoped. Mutating tools refuse non-filesystem views. Hook probes and the control socket resolve views the same way, with fail-open preserved for undiscovered worktrees and daemon outages.
Observability. A bounded-cardinality metrics registry and log fields at every lifecycle seam make four questions answerable: which layers served a query, why a view is stale, what evidence classified a checkout, and why a generation still exists. The daemon status payload gains an additive views census.
Compatibility
Verification
Every round landed through an implement/adversarial-review/fix cycle. Gates on the final tree: full-module build, vet, golangci-lint clean; the complete test suite under
-race; the equivalence oracle at default and soak volume; plan-fence tests proving no query regressed to a table scan; and the cross-generation leak fences. The 39 commits are ordered for review: storage first, then composition, builders, lifecycle, serving, and surfaces.Closes #663