Skip to content

Add reproducible benchmark JSON artifacts - #439

Open
helly25 wants to merge 4 commits into
design/string-interningfrom
perf/benchmark-artifacts
Open

helly25 wants to merge 4 commits into
design/string-interningfrom
perf/benchmark-artifacts

Conversation

@helly25

@helly25 helly25 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Add one strict, reusable measurement envelope for every Arena, SegmentedSequence, HAMT, and StringInterner benchmark. This makes raw performance evidence attributable, comparable, and ready for later CI and artifact-store ingestion.

AG;DR

  • Adds a standard-library-only runner that executes Google Benchmark with nine repetitions, random interleaving, numeric-seconds warmup, and minimum-time controls.
  • Records raw benchmark rows plus full Git, host, CPU, compiler, Bazel, C++ baseline, command, load, and timing provenance.
  • Resolves Apple Silicon chip identity through the hardware profiler before accepting generic arm processor text, without storing serial-number or UUID fields.
  • Adds explicit valid/suspect/invalid classification so surprising runs are retained rather than silently discarded.
  • Adds a content hash and strict validation that rejects weak controls, empty results, incomplete provenance, and modified artifacts.
  • Publishes the versioned JSON Schema for later Artifactory ingestion and chart generation.
  • Integrates focused regression tests into pre-commit, including exact numeric warmup formatting and Apple chip detection.

Stack:

Validation at 8a2d2f155:

  • python3 -m unittest tools/benchmark_artifact_test.py (7 tests)
  • pre-commit run --files tools/benchmark_artifact.py tools/benchmark_artifact_test.py tools/benchmark_artifact.schema.json .pre-commit-config.yaml
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant