shard the full benchmark per fixture (generate-once + matrix + aggregate)

[BryanFRD]

Problem

The full benchmark is a single job that runs every fixture × every tool sequentially (plus the competitor npm installs). Wall-clock is the sum of all fixtures, dominated by mono-large (200 packages × 10k commits). The micro benchmarks are already matrix-sharded with a build-once + aggregate job — the full benchmark never got the same treatment.

Sharding per fixture would make wall-clock ≈ the slowest single fixture, which in turn buys the budget to raise runs (currently 10) for tighter confidence intervals.

Why it isn't just strategy: matrix

The action is a monolith: build → generate fixtures → run → download baseline → compare → upload baseline → comment. Naively matrixing the caller would mean:

• every shard regenerates all fixtures (the expensive part — that alone cancels the win);
• every shard runs compare and uploads a competing hyperfine-baseline artifact.

So the action has to be decomposable first.

Design

Action inputs (same shape as the binary-dir input added in #160/#161 — default ''/false keeps current behaviour):

• fixtures-dir — use pre-generated fixtures and skip the internal Generate benchmark fixtures step.
• shard (bool) — run + write latest.json only; skip baseline download, compare, upload and PR comment.

No fixture-filter is needed: run.sh already discovers fixtures from --fixtures-dir, so a shard just gets a directory containing its one fixture (exactly the filtered-fixtures/ trick FerrFlow's Fixtures jobs already use).

Merge script — scripts/merge-results.sh. latest.json is metadata + a flat map:

{ "timestamp": …, "ferrflow_version": …, "runner_cores": …,
  "benchmarks": { "<fixture>-<tool>-<variant>-<cmd>": { "median_ms": …, "stddev_ms": …, "memory_mb": … } },
  "ferrflow_parallel": { … }, "install_sizes": { … } }

Keys are fixture-scoped, so merging shards is a map union:

jq -s '.[0] + { benchmarks: (map(.benchmarks) | add),
                ferrflow_parallel: (map(.ferrflow_parallel) | add),
                install_sizes: (map(.install_sizes) | add) }'

Covered by a bats test like the other scripts.

Caller (FerrFlow ci.yml):

• bench-fixtures — generate all bench fixtures once → artifact.
• benchmark — strategy.matrix.fixture: [single, mono-small, mono-medium, mono-100-1k, mono-50-5k, mono-large, complex]; each shard downloads ferrflow-binary + the fixtures artifact, copies its own fixture into a filtered dir, calls the action with binary-dir + fixtures-dir + shard: true, uploads bench-partial-<fixture>.
• benchmark-aggregate — downloads all partials, merges, runs compare, uploads hyperfine-baseline + step summary. Mirrors the existing Micro Benchmark (aggregate) job.

Shard per fixture, never per tool

Each fixture's tools must stay on one runner: the perf page compares ferrflow against competitors within a fixture, so splitting tools across machines would compare timings from different hardware. Per-fixture sharding keeps every comparison intra-runner and valid. The trade-off is that absolute numbers between fixtures come from different machines — acceptable, since nothing presents cross-fixture absolutes as comparable. runner_cores / ferrflow_parallel are per-runner stats and should be recorded per shard rather than merged blindly.

Follow-up

Once wall-clock drops, raise runs (10 → 20-30). Note that more runs tightens variance within a run; it doesn't remove systematic runner-to-runner noise, so the relative full-regression-threshold stays the right gate for regressions over time.