spike(storage): evaluate sort and partition reuse for construction (#1506)

[DecisionNerd]

Closes #1506.

Summary

This evaluates DataFusion/Arrow sorting and partitioning against GraphForge's construction sort and partition code, following the #1505 protocol. The full evidence is in docs/development/evidence/shape-sort-partition-spike-1506.md. Raw data, drivers and the independent review are in docs/development/evidence/sort-partition-spike-1506/.

Experiment code (cfg(test) / test-support only; never selected by default; invalid modes fail closed):

• GF_SHAPE_SORT_SPIKE now selects baseline, arrow-fixed (spike(storage): Arrow fixed-partition sort experiment (#1506) #1511's slice, renamed), arrow or datafusion in the real partition loaders:
  • fixed-width partitions;
  • compact details, sorted through a zero-copy LargeBinary view that still moves offsets;
  • Arrow row partitions.
• DataFusion external sort: a SortExec with a FairSpillPool and a caller-owned spill directory, which streams real sealed spill segments.
• DataFusion hash repartition, then per-partition SortExec, then SortPreservingMergeExec.
• examples/sort_partition_spike.rs for kernel measurements, with a counting allocator. It is added to Bazel and to the parity map.

Results (bounded to the tested mechanisms and envelope)

Acceptance criterion	Evidence
Runnable experiment, or reviewed proof, for every candidate	Arrow and DataFusion sorts run in the real loaders. The external sort and hash+merge run as unit tests and in the kernel bench. Six incompatibility proofs (no DataFusion range partitioning, hash breaks concatenation order and is not a recordable format parameter, round-robin, no Arrow routing kernel, lexsort ≡ sort_to_indices, UUIDv7 high-bit) were confirmed by a separate read-only reviewer against the version-matched sources.
Skew/hub and over-budget, safe refusal vs success	A 132 MiB single-hub partition against a 32 MiB budget: the current loader refuses in 15 µs before allocating. DataFusion's external sort succeeds inside the same 32 MiB (pool peak 33.26 of 33.55 MB, 12 spills) with identical output, 1.45× slower than an in-memory sort with the budget raised. A pool too small for one batch returns a structured refusal with no partial output.
Fair repeated measurements; maintenance/correctness tradeoff	S18/S20 complete-ingest pairs: 3 rotated rounds per mode, quiet-gated. Wall ≤ +1.1%, CPU ≤ +1.5%, peak RSS +21–45 MiB for the candidates. Kernel: Arrow/DataFusion 1.9–3.3× slower with 28–78 MiB transient heap per 1 Mi records; the kernel alone is 1.85×. Tradeoff table in the doc.
No #1465, speculation or time-box used as a proof	None used.
Bounded conclusions	Retain in-memory sorting and recorded range splitters. The hybrid external sort is a candidate only for over-budget partitions, pending #1507 (spill authority) and #1509 (a streaming consumer), and it needs a maintainer decision because it changes the refusal contract.

Correctness

• cargo test -p graphforge-storage --lib: 1,296 passed in each of the four modes.
• same_input_twice_produces_identical_digests: identical 5 shaped and 28 encoded digests across all modes, matching the perf(storage): partition loads materialize detail records padded, a 5-6x in-memory multiplier #1441 values.
• All 18 Graph500 ingests reopen, and their recount and query result_sha256 values are identical within each scale.

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace -- -D warnings
• cargo clippy -p graphforge-storage --lib --examples --features test-support -- -D warnings
• make pre-push-fast: passed, including the Cargo/Bazel drift check.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 4b33148d-78f8-4013-b176-9588be919885

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}