invertedstore: land the segment-based index component (standalone, unwired)

.github/workflows/ci.yml

@@ -66,6 +66,11 @@ jobs:
           # those tests' deterministic seams already exercise the race per-iteration.
           # Linux runs the FULL suite (no -short) via the coverage gate above.
           set -o pipefail
+          # GitHub's default bash runs with `set -e`; without disabling it the failing
+          # `go test | tee | grep` pipeline aborts the script BEFORE the diagnostic dump
+          # below, so a macOS/Windows failure never prints its assertion detail. Capture
+          # the status explicitly and dump on failure instead.
+          set +e
           go test -short -v -timeout 15m ./... 2>&1 | tee vitest.log \
             | grep -E '^(ok|FAIL|--- (PASS|FAIL):|panic:)' | grep -vE '^--- PASS:.*\(0\.00s\)$'
           status=${PIPESTATUS[0]}

AGENTS.md

@@ -3,43 +3,41 @@
 Mandatory working principles for any AI agent (Claude Code, etc.) operating in this
 repository. They override default behavior. **Read them at the start of every session.**
 
-## 0. Every change happens in a git worktree — no exceptions
-
-**Before making ANY change to this repository — code, tests, docs, config, even a
-one-line edit or an edit to this file — work inside a dedicated git worktree, never
-the primary `main` checkout.** There are no exceptions and no "too small to bother"
-cases.
-
-- At the start of any task that will modify files, create/enter a worktree FIRST
-  (native `EnterWorktree`, else `git worktree add` under `.claude/worktrees/`).
-- Never edit files in the primary working tree. If you have already started there,
-  move the changes into a worktree (e.g. `git diff > /tmp/p.patch`, apply it in the
-  worktree) and `git restore` the main checkout to clean before continuing.
-- This isolates in-flight work, keeps `main` pristine, and makes every change
-  reviewable as its own branch.
-
-## 1. Code changes follow spec → review → task breakdown → review → implementation — no exceptions
-
-**Never jump straight to editing code.** Every change to code (and the tests/config that
-accompany it) goes through this flow, in order. Each step produces a **written artifact**,
-and each **review** gate must be explicitly approved before the next step starts:
-
-1. **Spec** — write WHAT changes and WHY: problem, goals / non-goals, design, the
-   interfaces & files affected, durability / compatibility impact, risks, and how it
-   will be verified.
-2. **Review** — the spec is reviewed and approved before any decomposition.
-3. **Task breakdown** — decompose the approved spec into concrete, ordered,
-   independently-verifiable tasks.
-4. **Review** — the task breakdown is reviewed and approved.
-5. **Implementation (SDD)** — implement strictly per the approved spec and tasks. If
-   reality diverges from the spec, STOP and amend the spec (back through review) — do
-   not improvise in code.
-
-A plan sketched in chat is NOT a spec. Measurement / exploration spikes are allowed
-*before* the spec (to inform it), but production code changes wait for an approved spec
-**and** task breakdown. There is no "too small to spec" exception.
-
-## 2. Infrastructure: ship any real benefit, however small
+## 0. NEVER write code directly — the SDD flow is mandatory, no exceptions
+
+For any code change, however small or "obvious", you MUST follow this pipeline IN ORDER and
+NEVER skip a stage:
+
+1. **Spec** — write the design/spec first (chunked per Principle 3).
+2. **Multi-agent review** — dispatch MULTIPLE independent review agents to cross-review the
+   spec; fix every blocker/major before proceeding.
+3. **Task breakdown** — decompose into bite-sized TDD tasks.
+4. **Multi-agent cross-review** — multiple agents cross-review the task breakdown; fix issues.
+5. **Implementation — driven by a WORKFLOW, never by hand.** TDD (red → green), **ONE item at a
+   time**, orchestrated through the Workflow tool (multi-agent): the coordinator MUST NOT hand-edit
+   product code in the main loop — every code edit happens inside a workflow subagent. For each item
+   the workflow: writes the failing test → runs it red → implements → runs it green → runs the gates,
+   then dispatches MULTIPLE independent review agents and LOOPS (fix → re-review) until that item
+   returns zero blocker/major; only THEN commits it and moves to the next item. Never batch several
+   items before reviewing. If you catch yourself opening an editor on product code outside a workflow,
+   STOP — author the workflow instead.
+
+**The review stages are a LOOP, not a single pass — re-review until clean.** Whenever you fix
+findings from a review (stage 2, 4, or 5), you MUST dispatch a FRESH round of multiple independent
+review agents on the REVISED artifact and repeat — your own edits are unverified until a new review
+round confirms them, and a fix routinely introduces a new blocker (e.g. a deadlock fix that
+reintroduces the deadlock elsewhere). Keep iterating rounds until a full round returns **zero
+Blocking and zero Major** findings. Do NOT advance to the next stage, and do NOT report the artifact
+as done, after merely *applying* fixes — applied-but-not-re-reviewed is not done. Record each round's
+findings + resolutions in the artifact so the convergence is auditable.
+
+Do NOT jump straight to editing code, not even for a "quick prototype", a "let me just
+measure it" spike, or a one-line fix. Prototyping a change before the spec/review is still
+"writing code directly" and is forbidden. Measurement that requires new/changed product code
+follows the same flow. If you catch yourself opening an editor before the spec is written and
+reviewed, STOP and go back to stage 1.
+
+## 1. Infrastructure: ship any real benefit, however small
 
 This work is infrastructure. If a change produces a **real, correct benefit — even a
 tiny one — do it.** Do not skip a sound improvement because the measured win looks
@@ -51,7 +49,7 @@ lever" or an easier alternative.
 - Don't substitute a different, larger-scope change for the obvious small one.
 - Do the complete job: sweep **all** the safe cases, not just the big ones.
 
-## 3. Verify at the source — no substitute environment or method
+## 2. Verify at the source — no substitute environment or method
 
 **Verify a problem, and its fix, WHERE the problem actually occurs.** Do not use a
 proxy environment or a substitute method and then draw conclusions from it.
@@ -63,14 +61,13 @@ proxy environment or a substitute method and then draw conclusions from it.
 - Don't be clever or presumptuous. Go to where the issue is, reproduce it there, and
   validate the fix there.
 - Do not propose unrequested "alternative approaches" in place of verifying the real
-  thing at its source. Don't fragment a small in-flight change into a separate,
-  deferred branch/PR to avoid doing it now — make it part of the work you are already
-  doing, in your current worktree (Principle 0).
+  thing at its source. Don't spin up a new worktree/PR for a small in-flight change —
+  make it directly in the branch you are already working in.
 
-When the two meet: make the real infrastructure improvement (Principle 2) **and** prove
-it in the real failing environment (Principle 3) — never in a convenient substitute.
+When the two meet: make the real infrastructure improvement (Principle 1) **and** prove
+it in the real failing environment (Principle 2) — never in a convenient substitute.
 
-## 4. Author large files incrementally — chunk, don't dump
+## 3. Author large files incrementally — chunk, don't dump
 
 When creating a large file (a plan, spec, design doc, or sizable code file), **do not
 emit the whole thing in one giant write.** Build it up in chunks: create the file with
@@ -82,3 +79,27 @@ its header/skeleton first, then append one section at a time.
   after the entire artifact lands, and produces a cleaner edit history.
 - Applies to generated docs and plans especially, but to any long file: prefer a
   sequence of focused appends over one monolithic write.
+
+## 4. A perf demo must be format-identical to the real implementation
+
+When you measure a design with a demo/prototype/spike, **the demo's on-disk format and
+data path must be EXACTLY what the real code will implement.** No simplified, packed,
+"good-enough", or approximate version is acceptable as a source of numbers.
+
+- **The disk format is the contract.** The byte layout the demo writes MUST be the byte
+  layout the production code writes — same blocks, same indexes, same chunking, same
+  encodings. A simplified layout produces simplified (i.e. wrong, usually optimistic)
+  numbers — disk size, memory, read amplification all change with the format.
+- **Every feature is measured through the demo, not estimated.** If a feature exists in
+  the design (the forward map, tombstones, compression, merge, large-value chunking),
+  it must be present and exercised in the demo before any number that involves it is
+  reported. "Inverted-only", "merge handled separately", "forward estimated" etc. are
+  self-deception — the deployed system always pays those costs, so the measurement must
+  too.
+- The CODE may be rough (messy, unfactored, demo-quality) — that is fine. The FORMAT and
+  the set of features exercised may **not** be rough or partial.
+- If a measurement was taken on a simplified path, it does not count. Rebuild the demo to
+  the real format and re-measure.
+
+This is the data-integrity counterpart to Principle 2: Principle 2 says measure in the
+real *environment*; Principle 3 says measure with the real *format and feature set*.

core/invertedstore/README.md

@@ -0,0 +1,123 @@
+# invertedstore — status, measurements & findings
+
+The segment-based (LSM-like) inverted index: the **go-forward** replacement for the pebble-backed
+`core/invertedindex`. Full as-built architecture: [`docs/design/invertedstore-design.md`](../../docs/design/invertedstore-design.md).
+This README is the durable record of **measured data + key conclusions** (so the next iteration does
+not re-derive them); the forward-looking redesign lives in
+[`docs/design/invertedstore-luceneization-exploration.md`](../../docs/design/invertedstore-luceneization-exploration.md)
+and [`...-implementation-plan.md`](../../docs/design/invertedstore-luceneization-implementation-plan.md).
+
+## Status (2026-06)
+
+**Built and component-complete, but NOT integrated.** The production server runs on the pebble-backed
+`invertedindex`; `invertedstore` is a **standalone, deliberately unwired** component. It is held back
+because it is **not yet mature at scale** (see "Known scale gaps" below), and — since `core/invertedindex`
+is a published package other projects depend on — **no swap seam / interface is added to it** for an
+unproven replacement. The integration approach will be designed only once invertedstore is stable. The
+component is exercised by its own tests + the `core/cmd/idxbench` A/B harness (a dev-only, uncommitted
+tool). Form: single mpsc-worker-owned head buffer → atomically-published immutable sealed segments +
+a MANIFEST; size-tiered background merge; single-mutator invariant; lock-free refcounted reader
+snapshots.
+
+## Measurements (lx corpus: 94,559 docs; `/workspace` xfs; default config CapBytes 16 MiB / Fanout 4 / L0 snappy, merged zstd)
+
+### Build / steady state (current tree, post C.4 fix)
+| metric | value |
+|---|---|
+| build (AutoMerge on) | **42.6 s** |
+| disk (settled) | **234.9 MiB** |
+| peak build RSS | **393 MiB** |
+| search | ~9.0 ms/q over 198 queries (hits 2,414,505) |
+| final live segments | 3 (1×L1 + 2×L2) |
+
+### Store vs pebble `invertedindex` (A/B, same corpus; store re-confirmed this session, pebble from the prior A/B)
+| | store | pebble | |
+|---|---|---|---|
+| build | ~42.6 s | ~64 s | store **~1.5× faster** |
+| disk | 234.9 MiB | ~643 MiB | store **~2.7× smaller** |
+| peak RSS | 393 MiB | ~610 MiB | store lower |
+| search | — | — | store **~4× SLOWER** (the known weak axis: read-amp = scan every live segment) |
+
+### Spill / merge cadence
+- A spill fires when the head's byte estimate reaches **CapBytes (16 MiB)** → an L0 segment ≈ **4.6 MiB on
+  disk** (snappy, ~3.4× compression). The lx corpus produces **~56–69** L0 spills.
+- With AutoMerge on (Fanout 4): **~23 tiered merge passes** collapse the 56 spills to **3** segments — a
+  merge roughly every **2–3 spills / ~1.7 s**, run concurrently (off-worker) with ingest. **No covering
+  merge fires on a pure-add build** (dead-fraction ≈ 0; covering only triggers at ≥ 0.25 or DeleteTable).
+
+### Fanout / write-amplification sweep (`idxbench -fanout`, write_bytes = real disk writes)
+| config | build | disk written (amp) | final segs |
+|---|---|---|---|
+| AutoMerge **off** (pure build) | 30.1 s | 322 MiB (**~1×**) | 69 (all L0) |
+| Fanout **4** (default) | 40.8 s | 725 MiB (**2.25×**) | 3 |
+| Fanout **8** | 50.8 s *(run-to-run outlier)* | 766 MiB (2.4×) | 5 |
+| Fanout **16** | 35.2 s *(reproduced)* | 540 MiB (**1.7×**) | 8 |
+
+→ Merges are **overlapped** with ingest, so AutoMerge adds modest wall-time (+~10 s vs pure build) but
+**~2.25× write amplification**; larger Fanout trades fewer/cheaper merges (less write-amp) for more
+residual segments (worse search). The pure-build vs merged disk gap (322→234.9 MiB) is the merge's
+recompress (L0 snappy → merged zstd) + dedup.
+
+## Merge strategy (and where it sits vs Lucene / RocksDB)
+
+**Size-tiered** (Cassandra STCS-like), NOT leveled (LevelDB/pebble): a level with ≥ Fanout (4) segments
+is k-way merged into ONE next-level segment; segments within a level are full-keyspace overlapping sorted
+runs, so a query scans **every** live segment newest→oldest (read-amp = segment count — the source of the
+~4× search gap vs pebble's leveled 1-file-per-level). A **covering** merge (all live segments → one,
+triggered at dead-fraction ≥ 0.25 or DeleteTable) is the escape hatch that reclaims tombstones/dead-table
+keys and collapses read-amp — the analog of Lucene `forceMerge(1)`.
+
+This is the **same family as Lucene's `TieredMergePolicy`** (immutable segments, size-tiered, forceMerge):
+deliberately write-optimized (low write-amp, cheap build) trading read-amp — aligned with the priority
+order **build ≫ mem > search**. Differences from Lucene: (1) reconciliation is **per-(keyword,docid)
+newest-wins** because our docid is a reused-with-new-content external id, not Lucene's append-only
+segment-local docid + liveDocs bitset; (2) selection is the crude "whole level ≥ Fanout" (no Lucene
+score-based sizing); (3) **no max-segment-size cap** (Lucene's `maxMergedSegmentBytes`); (4) no concurrent
+merges; (5) FST term index was measured **slower** than the sorted-keyword dict and is rejected.
+
+## Key findings (corrected ground-truth — respect these in any redesign)
+
+- **docid is a monotonic sequential int64 from idtable (`nextId++`), STABLE-PER-KEY, never recycled.** The
+  MD5 is the content/path *key* that maps into the id. Re-indexing the same file reuses the same id with a
+  NEW keyword set — so a plain Lucene deleted-docid bitset is insufficient (the live id can't just be
+  marked dead).
+- **`[I]` keys sort before `[F]`** in a segment, so during a merge a posting is emitted before its doc's
+  forward record (its version) is seen → any merge-time version filter needs a **resident** version table,
+  not inline resolution.
+- **Search resolves deletes INLINE** via the inverted value's `dels` half (newest-wins) and **never reads
+  the forward map**. The term-id ordinal coupling (forward stores ordinals into the segment's sorted
+  inverted dict) is purely a forward concern and is the source of ~120 lines of merge remap/ordSentinel
+  complexity + the "tiered merge can't drop a key" constraint.
+- **Merge is streaming across blocks/segments (one decompressed block per cursor) but NOT within a single
+  keyword:** a hot keyword's whole posting list is materialized (the reconciliation map + `readExternal`
+  reads the whole blob) — in BOTH merge and search. This is the dominant **scale OOM vector**.
+
+## The C.4 regression (a recorded footgun)
+
+`clear()` on a Go map does **not** release its bucket capacity. C.4 (commit 581383a) hoisted the
+per-keyword `adds`/`dels` reconciliation maps out of the merge loop and `clear()`+reused them; once a
+high-cardinality keyword grew a map, every later small key's `for d := range adds` scanned the retained
+(mostly empty) buckets → **O(numKeys × peakBuckets)**, regressing the lx build **6.5× (46→277 s)**. Fix
+(977fa05): revert to a **fresh map per key** → build 42.6 s, RSS −19% (the giant reused map no longer
+stays resident). Lesson: never `clear()`+reuse a Go map across keys of wildly varying size.
+
+## Known scale gaps (lx is a TEST corpus; real targets are orders of magnitude larger)
+
+The above numbers are on a 234 MiB test corpus; none of these gaps shows there. For a general engine they
+are correctness/scalability floors — see the [exploration](../../docs/design/invertedstore-luceneization-exploration.md)
++ [implementation plan](../../docs/design/invertedstore-luceneization-implementation-plan.md):
+
+1. **Hot-keyword OOM** — merge AND search materialize a whole keyword's posting list. Fixes: streaming
+   per-keyword reconciliation (bounds the cross-source union, no reindex) → then **chunked/block postings**
+   (fully df-independent, in the one reindex).
+2. **No max-segment-size cap** — merge grows unbounded; needs `MaxMergedSegmentBytes` with
+   newest-contiguous-by-id subset selection (an OLD subset would invert newest-wins).
+3. **Deletion is a trade** — per-keyword del-postings, reclaimed only by a full-index covering rewrite at
+   25% garbage. A per-doc forward-version tombstone makes deletes O(1) on the write side BUT adds a
+   search-time liveness filter; the two cannot both be free.
+4. **O(segments) MANIFEST** rewritten on every install + **O(docs) `recomputeLive`** on Open — worsen as
+   #2 multiplies segment count.
+
+Roadmap (no large corpus to validate yet): D0 synthetic-stress harness → streaming merge → streaming
+search → max-seg cap → ONE StorageVersion reindex (forward/inverted split + per-doc delete + chunked
+postings + keyword-range skip).

core/invertedstore/apply_fastpath_test.go

@@ -0,0 +1,40 @@
+package invertedstore
+
+import "testing"
+
+// A warm 1-op edit (drop a keyword) MUST still diff against the forward and tombstone the dropped
+// keyword — the fast path must not skip the diff. (Guards that len(ops)==1 still reads `old`.)
+func TestApplyFastPath_WarmEditTombstonesDroppedKeyword(t *testing.T) {
+	s, tid := newForwardSkipStore(t, Options{CapBytes: 1 << 20})
+	s.Update(tid, 1, []string{"alpha", "beta"})
+	s.spillForTest(tid)                      // seal so the next edit reads the forward from a segment
+	s.Update(tid, 1, []string{"alpha"})      // drop "beta"
+	s.q.RunFunc(func() error { return nil }) // drain
+	// "beta" must no longer resolve to docid 1.
+	if got := searchDocidsForTest(t, s, tid, "beta"); len(got) != 0 {
+		t.Fatalf("beta still maps to %v after the warm 1-op edit dropped it", got)
+	}
+	if got := searchDocidsForTest(t, s, tid, "alpha"); len(got) != 1 || got[0] != 1 {
+		t.Fatalf("alpha should still map to {1}, got %v", got)
+	}
+}
+
+func TestApplyFastPath_TakenForOneOpNotMultiOp(t *testing.T) {
+	s, tid := newForwardSkipStore(t, Options{CapBytes: 1 << 20})
+	var fast int
+	applyFastPathTaken = func() { fast++ }
+	t.Cleanup(func() { applyFastPathTaken = nil })
+
+	s.Update(tid, 1, []string{"a"}) // 1-op → fast path
+	s.q.RunFunc(func() error { return nil })
+	if fast != 1 {
+		t.Fatalf("1-op apply took the fast path %d times, want 1", fast)
+	}
+	b := s.NewBatch()
+	b.Update(tid, 2, []string{"b"}).Update(tid, 3, []string{"c"}) // 2-op → multi-op loop
+	b.Commit()
+	s.q.RunFunc(func() error { return nil })
+	if fast != 1 {
+		t.Fatalf("multi-op batch took the 1-op fast path (fast=%d, want still 1)", fast)
+	}
+}