Skip to content

[Bench] Fixes for pure Fizz bench - #96771

Merged
gaearon merged 3 commits into
canaryfrom
fix/sandbox-bench-profile-order
Aug 5, 2026
Merged

[Bench] Fixes for pure Fizz bench#96771
gaearon merged 3 commits into
canaryfrom
fix/sandbox-bench-profile-order

Conversation

@gaearon

@gaearon gaearon commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Harness fixes, all surfaced while benching a Fizz-only React PR (react/react#37022):

  • Profile pass now alternates arm order by VM index. The timed runs are ABBA, but the CPU profile pass ran base-then-cand on every VM. The second arm runs warmer (page cache, governor), which deflated every function in the cand profile by a uniform few percent — untouched Flight frames "improved" ~7%. With balanced order the same untouched frames diff to ~1.5% across 14 paired VMs while real movers keep their magnitude. Each VM records its capture order in the profiles tarball (prof-order.txt) so analyses can check order drift directly.
  • The fingerprint now covers every server-side React layer. FP_FILES hashed only the vendored Flight server files, which a Fizz-only React change leaves byte-identical — both arms of a valid A/B reported the same fingerprint. Added the vendored Fizz bundle and the shared react-server runtime (hooks/cache) in both suites. The VM boot loop also warns loudly when arms fingerprint identically, instead of leaving it for the analysis footer.
  • --prepare writes a terminal status phase. It previously exited leaving phase: "measuring", so bench-status prescribed collecting measurement data that never existed. bench-status now reports prepare-only runs as complete rather than "DEAD — relaunch".

Verification

  • Full 16-VM e2e run with the ordering + Fizz-fingerprint changes (run-pr37022-e2e-prof2): arms fingerprint distinctly (18ea275e vs f9805be5), profile order alternates per VM, untouched hot frames diff to ~zero in the cross-VM profile aggregate.
  • bench-status.mjs against a real --prepare run dir reports complete (--prepare: caches only).
  • --dry-run smoke of the modified launcher; node --check on all three scripts.

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: cf8a1ce

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Stats skipped

Commit: cf8a1ce
View workflow run

@gaearon gaearon changed the title sandbox-bench: balance profile-pass arm order, cover Fizz in fingerprint [Bench] Fixes for pure Fizz bench Aug 5, 2026
@gaearon
gaearon marked this pull request as ready for review August 5, 2026 22:46
@gaearon
gaearon requested a review from timneutkens August 5, 2026 22:46
Comment thread .agents/skills/sandbox-bench/scripts/sandbox-e2e.mjs Outdated
@gaearon
gaearon enabled auto-merge (squash) August 5, 2026 23:00
@gaearon
gaearon merged commit adcfa7f into canary Aug 5, 2026
71 checks passed
@gaearon
gaearon deleted the fix/sandbox-bench-profile-order branch August 5, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants