ci: collect and compare benchmark results

[deepso7]

Closes #138

Adds the Bench workflow with three parallel measurement jobs, strict main-only baseline writes to bench-data, and fork-safe PR comparison artifacts. Failed measurement tiers contribute an empty slice so successful tiers still reach comparison, while any incomplete main run leaves the baseline untouched.

The workflow bootstraps bench-data on its first fully successful main run.

Checks run locally:

• python3 -m unittest discover -s bench -p "test_*.py"
• pnpm test:workflows
• actionlint .github/workflows/bench.yml
• git diff --check

---

Summary by cubic

Adds a Bench CI workflow that closes #138 by running benchmarks on main and PRs, storing baseline results in bench-data, and uploading a comparison artifact for each PR. The Gungraun collector parses tagged integer Ir metrics, the Criterion collector reads benchmark identifiers from benchmark.json, and the baseline job pins its actions to immutable SHAs.

• Three parallel measurement jobs run the Rust micro, Rust wall, and Node FFI benchmark suites.
• Failed measurement jobs still produce empty result slices so successful ones reach comparison.
• Baseline updates only when all three jobs succeed on main; the first successful run bootstraps the branch.
• PR comparisons fetch the relevant baseline and mark it stale if the merge base changed.
• Benchmark command lists were moved into scripts/run-benches.sh, shared by the workflow and the local just recipes.

^{Written for commit d550803. Summary will update on new commits.}

[greptile-apps]

Greptile Summary

The PR adds automated benchmark collection, baseline persistence, and pull-request comparison reporting.

• Runs Rust instruction-count, Rust wall-clock, and Node FFI measurements in parallel.
• Updates bench-data only after every measurement tier succeeds on main.
• Produces fork-safe comparison artifacts for pull requests.
• Centralizes local and CI Rust benchmark commands in scripts/run-benches.sh.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/bench.yml	Adds parallel benchmark jobs, complete-run-gated baseline writes, and pull-request comparison artifact generation; the previously reported write-job action references are now pinned.
scripts/bench_results.py	Updates Criterion identifier extraction and supports Gungraun’s tagged integer metric representation.
scripts/run-benches.sh	Centralizes Rust wall-clock and instruction-count benchmark command sequences.
justfile	Delegates local Rust benchmark recipes to the shared benchmark runner.
bench/test_bench_results.py	Updates Criterion fixtures and adds coverage for tagged Gungraun integer metrics.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Trigger[Push to main or pull request] --> Micro[Rust micro benchmarks]
  Trigger --> Wall[Rust wall benchmarks]
  Trigger --> Node[Node FFI benchmarks]
  Micro --> Gate{Event and result status}
  Wall --> Gate
  Node --> Gate
  Gate -->|Main push and all complete| Baseline[Update bench-data baseline]
  Gate -->|Pull request| Compare[Merge slices and compare with baseline]
  Compare --> Artifact[Upload comparison artifact]

Loading

_{Reviews (2): Last reviewed commit: "refactor: share benchmark command lists" | Re-trigger Greptile}

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[deepso7]

Addressed @cubic-dev-ai’s duplication finding in d550803. The benchmark command lists now live in scripts/run-benches.sh; both the justfile targets and CI delegate to that script.

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/bench_results.py">

<violation number="1" location="scripts/bench_results.py:118">
P1: estimate.with_name('benchmark.json') points into the new/ directory (<id>/new/benchmark.json), but Criterion stores benchmark.json as a sibling of new/ and base/ (<id>/benchmark.json). load_json will fail on every estimate, so collect_criterion always raises BenchError and the rust-wall measurement job always fails. Use estimate.parent.parent / 'benchmark.json'.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: estimate.with_name('benchmark.json') points into the new/ directory (/new/benchmark.json), but Criterion stores benchmark.json as a sibling of new/ and base/ (/benchmark.json). load_json will fail on every estimate, so collect_criterion always raises BenchError and the rust-wall measurement job always fails. Use estimate.parent.parent / 'benchmark.json'.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/bench_results.py, line 118:

<comment>estimate.with_name('benchmark.json') points into the new/ directory (<id>/new/benchmark.json), but Criterion stores benchmark.json as a sibling of new/ and base/ (<id>/benchmark.json). load_json will fail on every estimate, so collect_criterion always raises BenchError and the rust-wall measurement job always fails. Use estimate.parent.parent / 'benchmark.json'.</comment>

<file context>
@@ -115,8 +115,9 @@ def collect_criterion(root: Path, output: Path, git_sha: str, since: Path) -> No
             continue
-        relative = estimate.relative_to(root)
-        name = "/".join(relative.parts[:-2])
+        name = load_json(estimate.with_name("benchmark.json")).get("full_id")
+        if not isinstance(name, str) or not name:
+            raise BenchError(f"missing full_id in {estimate.with_name('benchmark.json')}")
</file context>

Suggested change

	name = load_json(estimate.with_name("benchmark.json")).get("full_id")
	if not isinstance(name, str) or not name:
	raise BenchError(f"missing full_id in {estimate.with_name('benchmark.json')}")
	benchmark = load_json(estimate.parent.parent / "benchmark.json")
	name = benchmark.get("full_id")
	if not isinstance(name, str) or not name:
	raise BenchError(f"missing full_id in {estimate.parent.parent / 'benchmark.json'}")

[deepso7]

@cubic-dev-ai’s latest benchmark.json finding does not match the pinned Criterion output layout. The final live run on d550803 executed this exact code successfully: rust-wall collected and uploaded all 28 expected rows. The downloaded artifact contains those 28 validated rows, so estimate.with_name("benchmark.json") is the correct path for this repository’s Criterion version. Moving it to estimate.parent.parent would break the proven collector.

[deepso7]

@greptileai rereview