Two CodSpeed Simulation reports compared GitHub-hosted runs from different CPU models and produced blocking regressions.
The test report compares oci_blob_serve_fixed_hash at 211.497 µs on an AMD EPYC 7763 with 243.455 µs on an AMD EPYC 9V74, a 13.13% regression. The commit modifies one file: crates/peryx-upstream/tests/unit/client/exec_tests.rs. We rebuilt base 6fac09c and head c6d76fcf with Rust 1.96.0 under identical build inputs and fixed source and target paths. Both benchmark executables are 644,216,480 bytes and have SHA-256 e1a9a8bd47ff4a6bab1ae30ca18e7ecb384fc73ec49df824d9eb7169cf41a74f; cmp confirms byte identity. Their benchmark and harness symbols and instructions match.
The range-unit report compares the same 211.497 µs EPYC 7763 baseline with 485.516 µs on an EPYC 9V74. That change had a separate reachable code-generation regression, which we found and fixed. The report still compares the same mismatched CPU models and flag sets. This case shows why CodSpeed should retain the report for diagnosis instead of treating every mismatch as harmless.
CodSpeed's benchmark variance documentation says shared CI runners do not guarantee the same physical machine and identifies hardware and linked libraries as causes of Simulation variance. The environment-diff announcement says recent runners collect and compare this metadata. Both reports contain a runner diff for the CPU model and flags.
Please mark a comparison incompatible or N/A when the collected metadata proves that the base and head CPU model or flags differ. Retain the report and environment diff for diagnosis while same-environment regressions remain blocking. CodSpeed v5 applies this pattern to cross-version comparisons by marking them N/A instead of showing a misleading diff, as described in the v5 announcement.
Two CodSpeed Simulation reports compared GitHub-hosted runs from different CPU models and produced blocking regressions.
The test report compares
oci_blob_serve_fixed_hashat 211.497 µs on an AMD EPYC 7763 with 243.455 µs on an AMD EPYC 9V74, a 13.13% regression. The commit modifies one file:crates/peryx-upstream/tests/unit/client/exec_tests.rs. We rebuilt base6fac09cand headc6d76fcfwith Rust 1.96.0 under identical build inputs and fixed source and target paths. Both benchmark executables are 644,216,480 bytes and have SHA-256e1a9a8bd47ff4a6bab1ae30ca18e7ecb384fc73ec49df824d9eb7169cf41a74f;cmpconfirms byte identity. Their benchmark and harness symbols and instructions match.The range-unit report compares the same 211.497 µs EPYC 7763 baseline with 485.516 µs on an EPYC 9V74. That change had a separate reachable code-generation regression, which we found and fixed. The report still compares the same mismatched CPU models and flag sets. This case shows why CodSpeed should retain the report for diagnosis instead of treating every mismatch as harmless.
CodSpeed's benchmark variance documentation says shared CI runners do not guarantee the same physical machine and identifies hardware and linked libraries as causes of Simulation variance. The environment-diff announcement says recent runners collect and compare this metadata. Both reports contain a runner diff for the CPU model and flags.
Please mark a comparison incompatible or N/A when the collected metadata proves that the base and head CPU model or flags differ. Retain the report and environment diff for diagnosis while same-environment regressions remain blocking. CodSpeed v5 applies this pattern to cross-version comparisons by marking them N/A instead of showing a misleading diff, as described in the v5 announcement.