Skip to content

perf: qualify range and index roadmap experiments - #19

Merged
josephjohncox merged 2 commits into
mainfrom
perf/roadmap-experiments
Jul 22, 2026
Merged

perf: qualify range and index roadmap experiments#19
josephjohncox merged 2 commits into
mainfrom
perf/roadmap-experiments

Conversation

@josephjohncox

Copy link
Copy Markdown
Owner

Summary

Runs the six-area performance roadmap as bounded, evidence-gated experiments and lands only the changes that passed their fixed, pre-registered gates. Failed runtime candidates are deleted rather than shipped behind loosened thresholds. Adds concrete ExactBatchRangeSet and BoxIndex examples, and hardens every experiment verifier against forged evidence.

The canonical stable API (RangeSet, create_range_set, BackendRegistry), exact-batch geometry, payload algebra, atomicity, and all existing gates are unchanged.

Accepted into the runtime (2)

  • Geometry-only RangeSnapshot cache (treemendous/rangeset.py): reuses one immutable snapshot while geometry is unchanged and no payload policy is set; payload-bearing snapshots still clone/detach exactly as before. 40-block confirmation: unchanged N=10 000 upper-95 0.00057 (≤0.25), cached N=10000/N=1000 median 0.958 (≤1.50), worst write-then-observe upper-95 1.035 (≤1.10). Scalar-mutation regression vs. immutable baseline: median 0.98 (100 samples).
  • Lease projection caches + O(1) fence lookup (treemendous/applications/_shared/leasing.py, leasing/_common.py, Candidate B): direct token lookup with zero public snapshots and zero linear scans; lazy immutable projections invalidated on publication. 30-block: fence upper-95 0.177, repeated-snapshot N=8192 upper-95 0.266 (both ≤0.50).

Rejected and removed (deletion, not gate-weakening)

Area Result Reason
RangeSet.mutate_many transaction (E2) REJECTED B16/N1000 upper-95 >1.07 per payload, 23× with no payload; BackendAdapter has no atomic state-replacement primitive. No method/protocol/export added.
Segmented COW native storage (E3) REJECTED Wins large-N (N=100k/B16 local <0.60×) but tuned one-block fast path still gives N64/B16 upper-95 1.116 > 1.10. Runtime fully reverted to vector storage.
Concrete-engine backend injection (E5) REJECTED cpp_boundary confirmed faster, but current-default py_boundary upper-95 hit 1.125 on pool_group (≤1.10 default-stability gate). No injection retained.
Lease Candidate D (authoritative free-delta) REJECTED N8192/N2048 normalized time 3.7× (≤2×). Reverted.
Adaptive BoxIndex for RadioSpectrum (E6) REJECTED Projection wins narrow queries (0.80–0.91×) but retained memory 1.30–1.37× (≤1.25×) and broad queries regress to 1.24–1.38×. Runtime stays linear BoxIndex(2).

New executable examples

  • ExactBatch: atomic_memory_map_updates, atomic_partition_availability_updates, genomic_mask_batch_updates
  • BoxIndex: warehouse_space_time_reservations (3D), video_region_timeline_overlap (3D), robot_volume_time_conflicts (4D)

All are registered in examples/README.md and executed from an unrelated CWD by tests/docs/test_readme.py.

Diagnostic evidence (informative only)

  • Exact-batch application matrix (180 cells): ExactBatchRangeSet beats scalar cpp_boundary replay from B=1 at N≤1000 across all shapes, but loses at N=10 000 (1.1–3.6×) — the O(N) full-state copy E3 tried and failed to remove.
  • Segmented-storage rejection is durably reproducible from a tracked patch (tests/performance/experiments/fixtures/exact_batch_segmented_tuned.patch) plus an archived triplet under docs/evidence/experiments/, bound by source/binary/patch SHA-256 and verifiable offline (just verify-exact-batch-storage-archive).

Verifier hardening

Every experiment verifier rejects fabricated runtime/build/backend provenance, bool/int and int/float coercion, duplicate keys, non-canonical JSON, and relabeled/duplicated matrix rows. The snapshot benchmark's order-confounded pooled analysis was replaced with pre-registered balanced AB/BA blocks and whole-block bootstrap.

Boundaries preserved

  • No compatibility façade, deprecated alias, silent fallback, or weakened exact result.
  • Experiment code lives only under tests/performance/experiments/; no stable package API, ABI, or gate changed.
  • Explicit backend/index/guard qualification errors are never caught to select a fallback.

Validation

  • just check (layout, scenarios, Ruff, format, mypy, full pytest + coverage, packaging/docs, bytecode) passed locally.
  • Focused experiment suites: 166 passed; snapshot-cache + lease-publication + exact-batch suites green.
  • All 12 tracked examples and both README quickstarts execute from an unrelated CWD.
  • Durable archive verifier, snapshot/lease/application-backend generators, and their strict verifiers round-trip.

Experiment matrices are machine-specific diagnostics; accepted-runtime decisions rest on the fixed gates above, not on any single headline number.

@josephjohncox
josephjohncox force-pushed the perf/roadmap-experiments branch from d8d995a to 7abab6a Compare July 22, 2026 09:14
CI checks out a shallow clone, so `git show <baseline-commit>:...` for the
pinned pre-segmentation commit failed with exit 128. Fetch that commit by SHA
on demand (GitHub permits reachable-SHA fetches) and skip with a clear reason
only when it is genuinely unavailable (offline), instead of hard-failing.
@josephjohncox
josephjohncox merged commit a310dc3 into main Jul 22, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant