full-history: hot-ingest p99 + memory campaign — packed rows, sorted-run tier, spill-and-merge cold build, zero-decompression freeze

[tamirms]

DRAFT STATUS / MERGEABILITY: rebased onto feature/full-history after #917 merged. The SDK views-walk redesign was abandoned — it was prototyped and measured, and the API cost (a generated surface that grows quadratically, or runtime ordering knowledge demanded of callers) was not worth the gain. In its place the branch pins a small go-stellar-sdk branch of four ingest/-only commits, under review as stellar/go-stellar-sdk#5972 — three local optimizations plus one narrow API change (the events products return trimmed element views rather than [][]byte). That pin is a pseudo-version of an unmerged branch, so this PR still waits on an SDK merge — but on a small, self-contained PR rather than a redesign. The branch also pins stellar/streamhash to a one-method branch (stellar/streamhash#13, Index.MaxBlockKeys() — exporting the format-level per-block key ceiling the events reorder heap derives its backstop from); a one-line pin bump follows that merge. Everything else in the stack stands alone.

Background (if you haven't followed the full-history work)

The v2 daemon (internal/rpcv2) stores the chain's complete history locally, split into fixed ranges of ledgers called chunks:

• The chunk currently being filled lives in a per-chunk RocksDB — the hot tier. Live ingestion appends every closed ledger to it, and queries for recent ledgers are served from it.
• Once a chunk's last ledger commits, the chunk is frozen: its contents are rewritten into immutable, mmap-friendly artifact files (a ledgers pack, a tx-hash lookup file, an events pack + events index) — the cold tier — which serve that ledger range forever after. Historical chunks can also be built directly from raw ledgers without a hot DB; that's the backfill/walk path. Freeze and walk must produce byte-identical artifacts.
• The events index maps a query term (contract ID, topic, …) to the set of matching event IDs inside one chunk.

Ingestion has a hard real-time constraint: the network closes a ledger roughly every ~600 ms, and the same machine concurrently runs freezes and index builds, so per-ledger ingest latency needs a lot of headroom under worst-case load. Throughout this PR, "stress load" means a synthetic worst-case dataset with ~6,000 transactions per ledger at a 600 ms close cadence, on one 8-vCPU box.

Why this PR exists

Measured on that stress load, three things were broken or unsustainable:

1. Hot ingest missed its latency budget. Per-ledger ingest was p50 124 ms / p99 303 ms against a ≤100 ms p99 target. Profiling showed the tail was garbage collection (54% of CPU, ~32 MB of allocation garbage per ledger) and the commit cost was RocksDB memtable key count (~18–30k keys per ledger, one per term-event pair).
2. Building a chunk's cold events index needed the whole index in RAM. The build kept a map of every term's full ID set in memory: 24.1 GB peak RSS for one stress chunk. One run OOM-killed a 32 GB machine.
3. Freezing a chunk decompressed and re-compressed everything. ~33% of freeze CPU was zstd re-encoding data that was already compressed on disk, and the freeze's unthrottled I/O measurably degraded live ingest running on the same machine.

What changed (12 theme-level commits; the numbered items below map onto them)

The branch is eleven commits, one per campaign theme: (1) bench+diagnostics, (2) hot-latency wave 1, (3) events-index memory [items 3-4], (4) zero-decompression freeze + window residue [items 5-6], (5) hot-latency wave 2 [items 7-10], (6) delta postings + un-materialized query — the cold index.pack format change, (7) events-index follow-ons (one-pass bloom/fence + hardening), (8) the storage-layer consolidation series, (9) docs [item 14 + rulings], (10) the go-stellar-sdk pin, (11) forking the ledger encode before the caller's walk, (12) the events reorder-heap improvement — inline small bodies, artifact-derived backstop [item 15]. Each squashed commit's message carries its constituents' full narratives as titled sections, so the per-item reasoning below survives in the history itself.

Commit (6) is deliberately isolated so it can be cherry-picked. Cold artifacts are permanent — there is no cheap migration for a chunk already frozen — so the index.pack record-layout change is the one piece that must be right in the first v2 release. It was split out of what had been a three-theme "events index follow-ons" commit; commit (7) holds the remainder (one-pass bloom/fence construction, the hardening pass, the EVR1→EVR2 run-file tag), all of which is hot-tier only and therefore has an upgrade path: discard the hot DB and let it rebuild. Commit (7) contains no query-path code — its functions are all write/seal/merge — so (6) stands alone with the full read path intact. The split was verified tree-identical to the pre-split history at both boundaries.

Commits (10) and (11) replace what was previously a single "views-walk port" commit. (10) exists only after the rebase: #917 hoisted ExtractLedgerTxParts into the caller, and this branch's zstd fork/join started inside IngestLedger — each correct alone, but composed, the encode no longer overlapped the walk it was supposed to hide behind. Every test stayed green and byte-identity held; only the phase trace showed it.

1. Bench + diagnostics — the harness every number in this PR comes from: a per-ledger CSV trace decomposing ingest into phases, and an opt-in RocksDB statistics dump.

2. Hot ingest latency — four fixes: reuse zstd encode buffers (they were the single biggest allocator); write the events index as one packed row per ledger instead of one RocksDB key per term-event (removes tens of thousands of memtable keys per ledger); run ledger compression concurrently with the other per-ledger work instead of serially; and stop cloning roaring bitmaps on every dense-term update (new IDs accumulate in a small immutable "tail" that readers merge on demand; a real data race in the reader/writer protocol was found and fixed here, with a regression test that fails without the fix).

3. Cold index build without the RAM — instead of an in-memory map of all terms, (term, event-ID) pairs are written to sorted, checksummed spill files and merged in a streaming pass. Memory becomes O(buffer) instead of O(unique terms). The output is verified byte-identical to the old in-memory builder, which is kept solely as the test oracle.

4. A bounded in-memory index for the hot tier (the largest commit). The hot chunk's in-memory events index used to grow without bound as the chunk filled. It's replaced by: a window of the last 256 ledgers' packed rows, older data sealed into checksummed run files on disk (with bloom filters to keep lookups cheap, and periodic merging so no query ever touches more than 8 files), and a small in-memory overlay only for very hot terms. Crash recovery replays from RocksDB, anchored on a manifest. One invariant worth knowing: a not-yet-validated startup writes nothing durable — sealing is only enabled after the open-time consistency check passes, so a failed open can always be retried against unchanged state.

5. Zero-decompression freeze — the freeze now copies already-compressed data as-is: ledgers as verbatim zstd frames, tx-hashes by streaming the (already sorted) RocksDB column family, events by merging on-disk state. No ledger stream input at all (the function signature makes decompression impossible, not just avoided). Byte-identity with the walk path is enforced by tests for all three artifact types.

6. Scale checks + residue — a harness proving the tx-hash index build holds flat memory at terminal-window scale (7.5 B keys), plus write-smoothing on index output (BytesPerSync) so artifact writes don't collide with the live WAL's fsync when co-located.
   7.–10. Hot ingest latency, second wave (four commits). After the first wave, the commit phase was still dominated by the tx-hash index: ~6,000 random 32-byte RocksDB keys per ledger, measured at ~11ms of memtable-insert CPU alone. These commits:

   • (7) tx-hash packed-row engine — a standalone storage engine for the hot tx-hash tier: one sorted row of hashes per ledger in a small in-memory window, sealed every 256 ledgers into checksummed, hash-sorted run files routed by bloom filters — one aligned pread per lookup. Nothing wired yet; reviewable in isolation.
   • (8) the flip — ingest writes one seq-keyed row per ledger instead of ~6,000 hash-keyed Puts (commit p50 20.5→7.9ms); the freeze merges sealed runs with the tail instead of scanning a 60M-key column family (tx-hash freeze arm 1m25→17.9s); cold bytes stay byte-identical (gated). Point lookups measured 2–3× faster in every class at full-chunk density, zero mismatches over 800k verified lookups (misses gain most: bloom short-circuit vs the full RocksDB read path). One trade: opening a full chunk drain-verifies its sealed runs, ~7.5s once per process start.
   • (9) events marshal rewrite — the per-ledger term map replaced by flat (term, id) pair arenas + a 256-bucket MSD sort over keys only (~30,000 write-path allocations/ledger → ~1; events phase p50 8.0→5.9ms). The retired map path survives as test-only reference for byte-identity gates. Constant-key terms (event type, topic count — a closed nine-key alphabet) never enter the hash-and-sort pipeline at all: their IDs collect in per-key ascending lanes and merge into the runs at their byte-order positions (rows byte-identical, differential-gated). Without the lanes, every event emits the same few keys, so each such term’s pairs collapse into one sort bucket and the bucket sort re-derives an order the arena already has — worth 1.4 ms/ledger at stress density.
   • (10) multithreaded zstd encode, config-selected (default 2 workers) — after the above, the single-threaded ~21ms ledger encode became the pre-commit floor. A zstd_encode_workers config field (TOML [storage], default 2) enables libzstd's internal multithreading — still one standard, deterministic, frame-compatible frame (join-wait 7.4→1.9ms; total p50 33.4→29.2). Reviewer note: the setting is format-affecting (an MT frame's bytes differ from single-threaded), so ONE resolved value threads to BOTH the hot encoder and the walk/backfill cold writer — the commit fixes a latent hazard where the walk side hardcoded single-threaded encoding, which would have broken freeze-vs-walk byte identity the moment the default became 2; the identity gate now exercises MT frames on both sides.

7. events: per-transaction payload shaping — the whole-ledger payload materialization becomes a per-transaction shaper (one pass, cursor order preserved exactly; the two-pass shape it replaces is pinned as the test oracle). Still on the current SDK.

8. ingest: raw-bytes threading — the hot and cold ingest loops thread the raw ledger bytes down instead of re-deriving them from views, and one extraction pass feeds both the tx-hash rows and the event payloads. Still on the current SDK.

9. SDK pin + the encode fork (two commits) — the views/Walk/streaming SDK redesign this item used to describe was prototyped, measured, and then abandoned: it bought ~4 ms of extract but required either a generated API surface growing quadratically in type count or runtime ordering knowledge from every caller, and it carried a hand-mirrored locate layer duplicating generated code. What ships instead is a pin to tamirms/extract-easy-wins (ingest: size each element once on the extract path, and return element views (+ stress-density benchmark) go-stellar-sdk#5972) — four ingest/-only commits. Three are local optimizations; the fourth is a deliberate, narrow API change: the events products now hand back trimmed element views ([]xdr.TransactionEventView, [][]xdr.ContractEventView) instead of [][]byte, because element views ARE their wire bytes, so []byte(elem) is free while a consumer reading a field is spared re-wrapping. Our one affected call site (the getEvents stage filter) gets shorter for it.

   The second commit forks the ledger zstd encode before the caller's ExtractLedgerTxParts walk. It exists only post-rebase: v2 ingestion: one walk per ledger — feed and rebuild the getFeeStats windows (#881, #888) #917 hoisted that walk into the caller while this branch's fork/join started inside IngestLedger, so composed, the encode no longer overlapped the walk it was designed to hide behind. Worth 3.5 ms, and invisible to the test suite — every test stayed green and byte-identity held; only the phase trace showed it.

   Measured cost of the trade, 1,000 paced ledgers at stress density, both arms run back-to-back on the same box: extract p50 9.56 → 13.38 ms (+3.82), ingest_total p50 26.80 → 29.74 ms (+11.0%), p99 33.07 → 37.61 ms, peak RSS 1.42 → 1.52 GB. The entire delta is the extract phase — ledgers, txhash, events, and apply are flat, and commit improved 0.32 ms. That is the price of dropping metaFacts/the offsets arena, paid deliberately for an SDK branch that carries no maintenance debt. Headroom against the ≤100 ms p99 target remains ~2.7×.

10. Handoff doc — a design/state document and probe scripts for whoever continues this work. No code. Includes the newest finding: the residual mid-tail latency variance was traced to SMT-sibling scheduling collisions (the ingest thread and zstd workers landing on hyperthread twins) — eliminated entirely by one-thread-per-physical-core placement (total p90 37.8→29.2ms, p99 39.6→33.7ms, zero code); deployment guidance and the owed co-located validation are documented there.

11. Events reorder heap: inline small bodies + artifact-derived backstop (commit 12; co-authored with @marwen-abid, absorbing the review outcome of events: bound cold index reordering independently of posting size #931). Pass B of the streaming index build must emit records in MPHF slot order while its input arrives in key order; the two disagree only within one MPHF block, so strays wait in a bounded reorder heap. This commit splits the buffered representation by size: bodies at or under inlineBodyMax ride the heap inline (the bytes are already in hand from the sequential scan), larger ones are buffered as (offset, length) references into the scratch file and re-read positionally at emit. The threshold is derived from the delta codec's ceiling so every delta-coded body inlines by construction and moves with any codec retune; only multi-MiB roaring bodies take the reference path. An all-reference design was measured first and rejected: ~99.7% of records stray (within-block slot order is effectively random), and one pread per strayed record cost +28.8% on the freeze's events arm at stress density. Measured end state (interleaved adjacent freeze cells, same box): events arm −3.1% vs the copy-everything baseline, lowest peak RSS of the variants tried; on a giant-body dataset (soroswap-shaped, ~few terms × multi-MiB postings) the reference path prices at zero. The heap's corruption backstop also stops being a transcribed constant: it derives from the opened index.hash via streamhash's new Index.MaxBlockKeys() (index: export MaxBlockKeys, the format-level per-block key ceiling streamhash#13), and the bound is exact — a block's minimum-slot key always takes the fast path, so a legal heap never holds more than MaxBlockKeys−1 records and the guard fires on the first provably-impossible push. Hardening from two adversarial review passes rides along: one shared fd for the CRC-verified scan and the emit-time re-reads (same inode by construction), Finish beside Create in both builders, a zero-alloc CRC fold (~3M allocations per build removed), and the byte-identity gate now straddles inlineBodyMax with premise-checked bodies on both sides.

Consolidation series (commit 8, squashed from an 11-commit working series; every item designed by an adversarially-verified review panel and condition-reviewed at implementation). In landing order: a dirent-durability fix — the events engine could crash with the manifest durably naming a run whose directory entry was never journaled (txhash documented and paid this barrier; events silently violated it) — with the ordering pinned by test in both engines; the run-routing bloom extracted to one package (was a verbatim ~35-line duplicate); WriteColdBin becomes a loop over the freeze's stream writer (freeze-vs-walk .bin parity now structural, not test-maintained); housekeeping (five dead-code deletions, commit/close verb alignment); the seal-publish protocol extracted once into stores/internal/runset (publish, dispose-on-failed-publish, orphan sweep, seq resume — the twin dispose branches c0991de patched twice are one implementation, manifest value bytes now golden-pinned); runspill gets one write path at the final name (the tree's last os.Rename and the .tmp two-state mental model deleted; WriteRun and the cross-goroutine buffer hand-back deleted); txhash sealed runs build routing in the write pass (no more re-reading ~55MB per seal beside live ingest) and gain one streaming reader — the fold surfaced that the freeze path silently lacked the warmup's seq-range check (gate added, proven to fail pre-fix); the seal and freeze merge heaps consolidate to one cached-key heap (the two hot-loop heaps stay separate, bench-gated, by measurement); the events freeze consumes manifest-listed sealed runs + the un-sealed tail instead of re-deriving the whole chunk (the txhash freeze shape; crash-state identity pinned by a new byte-identity gate mirroring txhash's, including poisoned-orphan and mid-merge states); a rulings ledger in the handoff; and a review-pass commit from the integrated four-dimension review (zero correctness defects found; the confirmed quality items, each adversarially verified).

Results (same stress load and machine before/after)

metric	before	after
per-ledger ingest p99, 1k-ledger run (default GOGC)	303 ms	~47 ms (63–66 after wave 1)
per-ledger ingest p50	124 ms	~34 ms (49 after wave 1)
ingest p99 across a full 10k-ledger chunk	783 ms and degrading as the chunk filled	64 ms, flat across the whole chunk (wave-1 measurement)
tx-hash point lookup, hit / miss p50	21.5 / 21.0 µs	8.9 / 7.1 µs
cold events index build, peak RSS	24.1 GB	5.4 GB
full cold chunk build wall time	12 m	8 m 36 s
freeze wall time	10 m	~6 m (tx-hash arm 1m25 → 18s)
live ingest p99 while a freeze runs on the same box	~142 ms	window-conditional ~124 ms, blended ~99 ms; back to solo the moment the freeze ends

Two corrections to earlier revisions of this section. The STELLAR_RPC_ZSTD_WORKERS=2 env knob is gone — two encode workers is now simply the default (ledger.DefaultZstdEncodeWorkers = 2), so its gain is already in the numbers above rather than being opt-in. And the "pending go-stellar-sdk change (fused single-pass xdr-view traversal)" was abandoned, not shipped — see item 13 for what replaced it and what that cost.

Current-base measurement (feature/full-history @ 9c195b5, which indexes two additional terms per event since #904), 1,000 paced ledgers at stress density, two runs: p50 32.92 / 33.06 ms, p99 40.51 / 40.64 ms, peak RSS 1.52 GB. The table above predates the rebases and the SDK swap; where they disagree, this line is the current number.

How to review

Each of the twelve commits builds and passes its tests standalone. Reviewing commit-by-commit remains the intended path — the squashed commits' messages carry one titled section per original concern, and the focus notes below name the concern they apply to:

• Commit 2: the concurrency contracts. Pooled encode buffers rely on RocksDB's write batch copying synchronously. The bitmap tail-delta relies on "published bitmaps are never mutated again, not even copy-on-write bookkeeping" — the writer clones only its private copy. TestTailDelta_WriterReaderRace_MultiContainer pins the race this prevents.
• Commit 3: the spill-file format (CRC framing, ascending-ID delta encoding — events.DecodeAscendingIDs is deliberately the only implementation of that validation), and the scratch-directory contract: scratch has no record anywhere, so both builders derive the same deterministic scratch path per chunk and wipe it on entry — that convention is the only thing preventing multi-GB leftovers after a crash.
• Commit 4: the seal/merge/recovery state machine, and the startup-writes-nothing-durable invariant (TestWarmup_FailedOpenLeavesDurableStateUntouched fails without it). The ≤8-file merge cap is measurement-backed (multi-term queries against unmerged files blew the query budget) — please don't simplify it away.
• Commit 5: correctness of copying compressed bytes verbatim — frame-compatibility gating (zstd.FrameHeaderValid), the density check that keeps pack positions equal to ledger sequence, big-endian→little-endian re-encode of the tx-hash values, and crash/retry semantics (all artifact writes follow the existing mark → write → fsync → flip protocol).
• One deliberate trade to be aware of (commit 5): freeze no longer decompresses, so payload verification moves from freeze-time to first cold read, which fails loudly. RocksDB block checksums still cover the copy, and the compressed frame's own checksum travels with it.
• Commit 7: the run-file format (36-byte hash‖seq records, CRC64, drain-verified opens) and the lookup path (window binary search → bloom → page ladder → one pread → full-hash verify, including the page-boundary two-page rule for prefix ties). The engine is born with sealing disarmed — same "an unvalidated open writes nothing durable" rule as the events tier.
• Commit 8: the dense per-ledger row chain (warmup trips loudly on the old format or a gap), fee-bump transactions contributing both outer and inner hashes (fullhistory: index fee-bump transactions by both hashes #862), the freeze's k-way merge emitting duplicate hashes verbatim so cold bytes stay identical to the walk path (composition-gated), and read-only opens disabling tx-hash queries structurally (the freeze never queries).
• Commit 9: the sort correctness gates — the bucketed keys-only sort is property-tested to produce the exact permutation of a stable bytes.Compare sort, and the retired map-based path remains in tests as the byte-identity oracle.
• Commit 10: zstd.WithWorkers panics loudly if libzstd lacks multithreading support rather than silently running single-threaded; the frame stays standard and FrameHeaderValid-compatible, so the cold-inherits-hot verbatim-copy contract of commit 5 is untouched.
• Commit 12: the block contract and its fenceposts. The inline/reference discriminator is pinned by the byte-identity gate's premise-checked straddle corpus (bodies on both sides of inlineBodyMax in one build); the MaxBlockKeys−1 bound rests on the proof that a block's minimum-slot key always fast-paths (heap provably drains at every block boundary); and the count backstop deliberately derives from the format ceiling, not the artifact's own block data — the backstop must not trust the content it is guarding against.

Results across hardware classes

The campaign box was an m6id.2xlarge (8 vCPU / 4 physical cores, 474GB instance NVMe). The full suite was independently reproduced on a c6id.8xlarge (32 vCPU / 16 cores, 1.9TB instance NVMe) — same dataset, same branch, default knobs. The c6id hot, freeze, and cold cells were re-validated at the current tip (post delta-postings); the co-location cells date from two tips earlier (their interference channels are device/bandwidth-level, and the paths they stress are unchanged). The c6id column reflects the branch tip including the consolidation series; the m6id column predates it:

cell	m6id.2xlarge (campaign box)	c6id.8xlarge
hot solo, full 10k chunk p50 / p99	29.3 / 39.9ms (pinned: 27.9 / 34.3)	27.2 / 33.2ms, 0 of 10,000 ledgers >100ms (the consolidation series' one-pass seal routing removed a ~55MB background re-read per seal; the prior tip's 36.5 stall-cluster reading resolved to the campaign's best p99)
hot beside freeze, one disk: window p99	124–153ms across runs (blended 98–122) — the write-collision tail is intensity-sensitive	39.6ms (blended 37.9 — ≈+3ms over solo)
hot beside full txindex rebuild: p99	~152ms	48.3ms (pooled, n=578; zero >100ms)
hot + freeze + full rebuild simultaneously (the coverage-lag worst case)	not measured	p99 53.3ms, max 79.5ms, zero >100ms (n=281; effects compose ~additively: +3 freeze, +11 rebuild, +16 stacked)
freeze wall / peak RSS	5m59.6s / 6.4GB	4m39s–4m40s / 6.9–7.8GB (×2 runs, 2s apart — the events arm drops 269s→194s (−26%) once the freeze merges the hot tier's pre-sorted sealed runs instead of re-deriving the chunk; wall −20% vs the pre-series 5m49–6m04s band, and run-to-run variance collapses with the map-fold gone)
cold chunk build wall / RSS	7m44s (2 workers) / 5.4GB — unchanged by the final two commits (their code is outside the cold-build path); a warm-source-cache re-run measured 5m51, so treat cold walls as cache-sensitive	6m48s (32 workers) / 5.5GB (7m01s at the pre-delta-postings tip)
terminal-window index build (≈6–8B keys)	5m12s, ~24M keys/s, RSS ~200MB	2m47s, 35.9M keys/s, RSS 351MB
cold events index.pack, one chunk (delta postings)	−41% postings bytes, measured on a real pubnet chunk (commit basis)	1.498GB → 0.776GB (−48%) at stress density; events.pack byte-identical; freeze-vs-walk identity verified by cmp on both files

Full-chunk hot RSS: steady ~1.4–2GB (the delta-postings tip retired a per-publish materialization; c6id steady-state 1.54→1.40GB); merge-generation peaks ~4.6GB (campaign box) / 4.8–4.9GB (c6id, paced) since the one-pass bloom/fence construction (the 5.1–5.2GB figures on both boxes predate that change) — see the handoff doc's RAM anchor.

Current-base re-measurement (2026-08-25, c6id.8xlarge)

Every regime measurable on this box class, re-run on the shipping branch at its
current base (feature/full-history @ 9c195b5, which indexes two additional
per-event terms since #904; the constant-key lanes in the wave-2 commit absorb
most of their cost). Same dataset, same knobs. These supersede the two earlier
tables where they disagree:

cell	last documented	2026-08-25	note
1k paced solo p50 / p99	29.74 / 37.61	32.92–33.06 / 40.51–40.64 (2 runs)	—
hot solo, full 10k chunk p50 / p99	28.96–29.39 / 35.22–35.48	32.18 / 39.92	0 of 10,000 >100 ms, max 53 ms (was 1 of 20,000, one at 144 ms)
full-chunk merge-generation peak RSS	5.02–5.09 GB	4.84 GB	back inside the pre-SDK-trade 4.8–4.9 band
hot beside freeze: window p99	39.6	47.6 / 49.2 (2 runs), max 59.2	zero >100 ms; pre/post windows 37.5–39.7 = solo band
freeze wall, colocated / in stack	4m39s–5m33s era	4m00s / 4m40s	faster
hot beside full txindex rebuild: p99	51.51	57.71 (n=290, max 60.0)	zero >100 ms; post-window 38.6 = full recovery; colocated build wall 2m54s
hot + freeze + rebuild (worst case)	53.28, max 66.35	56.79, max 59.49 (n=293)	zero >100 ms; ≈ rebuild-only p99 — contention saturates, effects still don't compound
terminal-window index build	2m47s / 35.9 M keys/s / 351 MB	3m25s / 29.3 M keys/s / 339 MB	flat-memory property holds

Where p50/p99 sit above the prior rows, the delta decomposes into the two
extra #904 terms' irreducible post-lanes cost (~0.6 ms) and base/era drift
measured with zero code difference — the pre-rebase head itself re-measures
2–4 ms above its own recorded numbers in every regime on this box today. The
load-bearing claims all hold or improve: zero ledgers over 100 ms in every
regime including the stacked worst case (max 59.5, ~1.75x headroom), full
recovery the moment co-runners exit, flat index-build memory, and lower
full-chunk RSS. One regime-specific note: under co-location the extract
phase is the exposed surface — a longer extract (the SDK trade's ~+4 ms) rides
ahead of co-runner stalls, which is where that trade's cost shows beyond its
solo price.

Post-rebase re-measurement (2026-08-05, c6id.8xlarge)

The table above was measured before the rebase onto merged #917 and before the
SDK pin moved off the abandoned views-walk redesign. Re-run on the shipping
branch, same box, same dataset, same knobs — these supersede the c6id column
where they disagree:

cell	c6id claim above	post-rebase	note
hot solo, full 10k chunk p50 / p99	27.2 / 33.2	28.96 / 35.22 and 29.39 / 35.48 (2 runs)	+2.0 / +2.2 — the SDK trade (see item 13); entirely the extract phase
hot solo, ledgers >100ms	0 of 10,000	1 of 20,000 (2 runs)	one ledger at 144.1ms; did not reproduce — see below
full-chunk merge-generation peak RSS	4.8–4.9GB	5.09 / 5.02GB (2 runs)	+~0.2GB — the SDK trade, see below
hot beside full txindex rebuild: p99	48.3	51.51 (n=359)	interference itself improved: +15.1ms over solo then, +13.9ms now
hot + freeze + full rebuild (worst case)	53.3, max 79.5	53.28, max 66.35 (n=358)	unchanged — the extract cost does not compound under co-location

On the +0.2GB RSS. This is the same SDK trade as the extract regression,
measured in memory rather than latency. The controlled 1k A/B isolates it with
one variable changed: steelman SDK 1.422GB vs easy-wins 1.522GB, +0.100GB. At
full-chunk depth the same effect reads +0.2GB (4.8-4.9 -> 5.02/5.09), roughly
double at 10x the ledgers, consistent with an allocation-rate difference
compounding across more merge generations. It is not a regression in this
branch's own memory work — the one-pass bloom/fence construction still delivers
its 5.22 -> 4.55GB win; the baseline it operates on moved up when the SDK pin
changed.

The stacked worst case is the load-bearing number for the deployment note
below, and it reproduces to 0.02ms. Solo p99 rose ~2ms while stacked p99 did
not: under the stacked load the bottleneck is co-runner contention, not
extract, so the SDK's extra ~3.8ms is absorbed rather than added.

On the one >100ms ledger. A second, instrumented full-chunk run
(GODEBUG=gctrace=1 plus 1Hz sampling of /proc/stat steal and
/proc/vmstat compaction counters) produced zero breaches in 10,000
ledgers. Across both runs the figure is 1 in 20,000.

The two runs' extract distributions are near-identical — p50 12.81/13.40,
p99 19.13/19.09, p99.9 24.72/24.93, p99.99 26.00/27.15 — and the clean run's
max is 27.58ms, i.e. simply the top of its own distribution. The outlier sat
5x above p99.99 with nothing between, which is the shape of a one-off
external event rather than a code path.

Two candidate mechanisms are excluded by the instrumentation: GC pauses
(max stop-the-world across 257 collections: 0.413ms) and hypervisor
steal (0.60s over 6,000s wall, 0.01%). The mechanism is therefore not
identified, only bounded — a residual candidate visible in the data is THP
direct compaction (848 stalls during the clean run, none of which produced a
breach). A systematic allocation/GC cost is present and measurable, but it
is the 20-26ms tail — several of its members adjacent ledger pairs, i.e. one
GC cycle spanning two ledgers — and it tops out well under 30ms. A
seal-boundary correlation was hypothesised and refuted: mean
distance-to-seal for the top-100 spikes is 62.4 vs 63.9 for all ledgers
(uniform ~64), and ledgers within 8 of a seal are indistinguishable (p99
35.18 vs 35.22).

The stated gate (p99 <= 100ms) holds with ~3x margin in both runs.

Cold artifact format: what changes on disk

artifact	on-disk format	bytes vs a pre-campaign build
events index.pack	CHANGED — see below	differ
ledgers .pack	unchanged (still one standard zstd frame per ledger)	differ — see below
events events.pack	unchanged	identical (gated)
txhash .bin	unchanged (8B header + 20B/entry)	identical
txhash .idx	unchanged	identical

index.pack gained a codec byte per record. Each record is one MPHF slot:

before   [0..4) fingerprint | [4..N) roaring bitmap (Bitmap.MarshalBinary)
after    [0..4) fingerprint | [4..5) codec | [5..N) postings in that codec

0x00 is roaring (RunOptimize'd, as before); 0x01 is a delta-varint list —
uvarint count, first event ID absolute, the rest strictly-positive deltas.
Terms at or below deltaPostingMaxCardinality (1024) use the delta codec.
That is the −48% index.pack reduction in the table above (1.498GB → 0.776GB
at stress density). The threshold is a proxy: cardinality correlates with
run structure on pubnet event postings (fat terms are the clustered ones), so
it is a property of the data, not of the format — re-measure before retuning.

The reader dispatches on the codec byte and rejects unknown values
(unknown codec 0x%02x), so it is forward-compatible. Backward: a reader
built before this PR, handed a new index.pack, parses the codec byte as the
head of a roaring bitmap; roaring validates its serial cookie, so this fails
cleanly rather than returning wrong postings — but it surfaces as
invalid bitmap at slot N, i.e. shaped like corruption rather than like a
version skew. The packfile container version is deliberately left at 1: the
change is to the item body, not the trailer or index that the version's
"bump on any breaking change" note governs. Flagging it explicitly so the
choice is reviewed rather than inferred.

Not isolated in a single commit: it lives inside commit (6) alongside the
one-pass bloom/fence work and a hardening pass. The format change proper is
5 non-test files, +317/−92 (10 files, +657/−154 with tests) of that commit's
39 files, +2,286/−701.

Ledger frames differ but the format does not. DefaultZstdEncodeWorkers
is now 2 (libzstd internal multithreading), which changes the frame byte
stream — single-threaded and multithreaded encodes of identical input split
jobs differently. The output remains ONE standard zstd frame with the header
shape FrameHeaderValid gates, so every decoder reads it unchanged and no
reader needs to know. The reason it is called out at all is the
byte-identity contract: the freeze copies hot frames into the cold pack
verbatim while walk/backfill re-encodes independently, so both materializers
must use the same worker count. They cannot diverge — daemon.go resolves
zstd_encode_workers once and feeds both the hot tuning and the walk
materializer from that single value. The only residue is that a chunk frozen
by a pre-campaign binary is not byte-identical to one frozen by this build;
that matters solely to a consumer comparing artifacts across builds.

Deployment note

The recommended production class is c6id.8xlarge-equivalent (or better), which resolves the topology question: on that class, single-disk co-location of live ingest with freezes and full index rebuilds — including all three stacked simultaneously (p99 53.3ms, zero violations) — meets the ≤100ms p99 target with wide headroom and no pacing anywhere — the device write queue never saturates (freeze +≈3ms) and the memory-bandwidth knee is never approached (rebuild +≈11ms). The interference mechanisms and their fingerprints are documented in the handoff doc for anyone deploying on smaller classes, where the mitigations remain valid: hot/cold disk separation (universal guarantee for the write channel) and the BuildColdIndex consumption pacer (small-box parity for the bandwidth channel; deliberately not implemented in this PR since the recommended class doesn't need it). The freeze runs unthrottled by design; the write-smoothing residue that remains is beneficial on any topology.

Testing

• Unit + -race across all touched packages, green at every commit individually, not just the tip.
• Byte-identity gates: streaming index build vs the in-memory oracle; freeze vs walk for all three artifact kinds.
• Regression tests that provably fail without their fix: the bitmap COW race, sealing-disabled-until-validated, failed-open-leaves-state-untouched, shared scratch naming.
• Before/after benchmarks for every claim in the results table, re-run after the final review round.
• The consolidation series: every commit condition-reviewed at implementation against its design-panel verdict; the integrated series re-reviewed across four dimensions with each finding adversarially verified (zero correctness defects); all six byte-identity gates green at every landing, plus new gates: events freeze-vs-walk identity across crash states, manifest golden bytes, barrier-before-publish ordering in both engines, TXHRUN01 golden header.
• The durability-protocol changes were additionally validated by a 120-iteration SIGKILL crash-recovery soak (both engines, power-loss dirent emulation, freeze byte-parity from crash-inherited states — 0 failures; the harness lands in a follow-up PR).

Deliberately not in this PR

• A rate-limiter for the rolling tx-hash index rebuild (only needed on hardware classes below the recommended one; sized and documented in the handoff doc).
• Captive-core co-location measurements (methodology ready, needs a core binary).
• Query-side contention work (Full-history query benchmark commands — reads across all data types  #856), including the SMT policy question for serving workloads.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C3xaaoD9LhzyKPdnZCJa2N

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	golang/​github.com/​stellar/​go-stellar-sdk@​v0.7.4-0.20260825061852-f479ee9335f3 ⏵ v0.7.4-0.20260825074110-c41b9e042433	+1
	golang/​github.com/​stellar/​streamhash@​v0.0.0-20260713164615-c72a4e6f578d ⏵ v0.1.1-0.20260816071515-6eab56dfbe5f	+1

View full report

[tamirms]

Rebased onto feature/full-history @ 9c195b57 (previous head 24c6e613 → 5beec6b4, still 12 commits). Now MERGEABLE. Resolution log — the non-mechanical decisions, for review:

Mechanical bulk. The base folded events/ into stores/event (#951) while this branch rewrote those files across 7 commits, so most conflicts resolved as this branch's change, translated into the folded package (imports, events.X → package-local). Campaign-added files were auto-relocated by git's directory-rename detection.

Real reconciliations:

1. Import cycle — the fold makes stores/event ↔ runspill circular (impossible in the old two-package layout). The postings codec (AppendTermPostings/AppendPostings/TermPostingsLen/DecodeAscendingIDs) moved into runspill as single owner; stores/event keeps same-named delegates, and runspill's API takes raw [16]byte so it stays parent-free.
2. match.go — this branch rewrote query.go (delta postings, Postings type, Intersect/Union) while the base replaced it with the Matches iterator the rpcv2: the getEvents v2 cross-chunk events pager #934 pager consumes. Resolution: base's iterator machinery, this branch's Postings API inside it (unionForFilters returns Postings; the window AND became ClipRange; streamUnion walks SelectIDs).
3. AppendTerms (the arena indexer) learned the two term kinds base's fullhistory: index event type and topic count for getEvents v1 #904 added — event type and topic count — so hot chunks keep those getEvents filters. Pinned by the existing golden sweep (TestAppendTerms_GoldenAgainstTermsForBytes, byte-equality across every truncation prefix).
4. Ingest signature — base's (seq, view) (int64, error) wins; this branch's new freeze test adapted.
5. Health wiring dropped — Serve the v2 read path: every method except the events reads (#885, #886; subsumes #889's wiring) #908 deleted the standalone health subsystem (read server owns health), so this branch's healthState plumbing (daemon.go/startup.go) does not survive.
6. Test injection hooks — the mirror's index() no longer exists; base-ported tests now inject via hotIdx.overlay with full-per-term id sets (overlay entries shadow window rows, matching this branch's own test comments).
7. SDK pin — tamirms/extract-easy-wins was rebased onto current main (f479ee93, which includes #5988) → c41b9e04; go.mod pins that, and the base's Serve the v2 read path: every method except the events reads (#885, #886; subsumes #889's wiring) #908 adapters gained view-generic clone helpers (~[]byte) in the same commit as the pin. This branch's newer streamhash pin kept.

Verification: full rpcv2 suite green at the final commit; production packages build at every one of the 12 commits; end-state audit (rebased tree vs old head) shows every differing file accounted for by the base's 6 commits, the package fold, or the adaptations above — the only exception being the documented freeze-test signature change. Known warts, deliberately left: two intermediate commits have test-only compile breaks in stores/event (fixed by the following commit; production code unaffected), and a few comments in mid-stack commits say event.pack where the artifact is events.pack (corrected by the final state).

🤖 Generated with Claude Code