feat(e2e): add perf test - #382
Open
sgopinath1 wants to merge 7 commits into
Open
Conversation
Add perf_tests/run_perf.sh (held submit for metrics 1–2, release + drain for metric 3) and run_all.sh for multi-tier shell sweeps. Add native-host pytest harness with JSON export and PR-vs-nightly compare, optional perf.yml workflow, and exclude @pytest.mark.perf from default E2E CI.
Fail run_perf.sh on partial submits and drain timeout; quiet control release output and fall back when column is missing. Parse RELEASE_WALL_S and PERF_JOB_NAME in the harness, upload run_perf.sh once per suite, and redeploy the ephemeral cluster between multi-tier runs. Add compare tier mismatch warnings, compare unit tests, perf.yml threshold input, and warn when sinfo is unavailable in the perf pytest.
sgopinath1
requested review from
powderluv,
sajmera-pensando,
shiv-tyagi and
yansun1996
as code owners
July 2, 2026 08:34
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an optional end-to-end performance benchmarking suite for Spur’s scheduler/ingestion path, including a shell-based tier runner, a pytest harness that redeploys an ephemeral native-host cluster between tiers, a JSON compare tool for regression detection, and CI/workflow wiring to keep perf out of default E2E runs while enabling manual PR-vs-nightly comparisons.
Changes:
- Introduces
perf_tests/shell scripts to run single-tier and multi-tier perf benchmarks and emit a structuredKEY=VALUEmetrics block. - Adds a pytest-based native-host perf harness (
perf_harness/) to run tiers over SSH, parse metrics into typed results/JSON, and generate markdown summaries. - Adds a manual GitHub Actions workflow for PR vs nightly perf comparisons and updates CI/E2E to exclude
@pytest.mark.perfby default.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/README.md | Documents the new perf harness location under native-host E2E tests. |
| tests/pytest.ini | Registers the perf marker for optional perf tests. |
| tests/native_host/e2e/test_scheduler_perf.py | Adds the optional @pytest.mark.perf test entrypoint that runs the perf suite and prints a summary/JSON. |
| tests/native_host/e2e/test_perf_harness_parse.py | Unit test coverage for parsing the run_perf.sh metrics block. |
| tests/native_host/e2e/test_perf_harness_compare.py | Unit test coverage for suite comparison logic and CLI exit behavior. |
| tests/native_host/e2e/perf_harness/README.md | Documents harness behavior, env vars, local usage, and CI workflow usage. |
| tests/native_host/e2e/perf_harness/harness.py | Implements SSH execution, metrics parsing, typed results, JSON I/O, and summary formatting. |
| tests/native_host/e2e/perf_harness/compare.py | Implements JSON suite comparison and a CLI suitable for CI regression gating. |
| tests/native_host/e2e/perf_harness/init.py | Exposes the harness/compare API surface for imports. |
| perf_tests/run_perf.sh | Adds the core shell benchmark for held submit, release, drain, and latency sampling. |
| perf_tests/run_all.sh | Adds a multi-tier driver that aggregates per-tier metrics into a combined table. |
| perf_tests/README.md | Documents manual usage and environment variables for the perf scripts. |
| .github/workflows/perf.yml | Adds a manual workflow to benchmark PR binaries vs latest nightly on the bare-metal cluster. |
| .github/workflows/e2e.yml | Excludes perf-marked tests from the default native-host E2E workflow run. |
| .github/workflows/ci.yml | Includes perf_tests/ in uploaded E2E assets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add SPDX headers to perf shell scripts. Filter drain polling by tier JOB_NAME instead of all user jobs. Reject non-positive SPUR_PERF_TIERS values at parse time with a unit test.
…perf Move shell benchmarks, pytest harness, and compare/report tooling into tests/native_host/perf/ as a sibling of e2e/. Lift cluster.py and conftest.py to tests/native_host/ for shared fixtures. run_perf.sh emits PERF_METRICS_JSON; pytest uses --perf-json; PR-vs-nightly comparison runs via run_perf_compare.sh and python -m perf.report. Update workflows and building.rst docs; remove perf_tests/ and e2e/perf_harness/.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #382 +/- ##
==========================================
- Coverage 68.40% 68.38% -0.02%
==========================================
Files 134 134
Lines 36281 36281
==========================================
- Hits 24818 24810 -8
- Misses 11463 11471 +8 🚀 New features to boost your workflow:
|
shiv-tyagi
requested changes
Jul 3, 2026
Download PR binaries and test assets from the successful CI run in perf.yml instead of checking out and building on the self-hosted runner. Add perf-harness-unit job for test_perf_parse/report, harden run_perf_compare.sh cleanup (sshpass/BatchMode, remote bin dir removal), and drop broken ingestion-benchmark doc links.
shiv-tyagi
previously approved these changes
Jul 3, 2026
Resolve e2e.yml by adopting main's driver-VM E2E matrix and excluding perf-marked tests from native-host pytest via -m "not perf".
Relocate e2e job payloads to tests/native_host/fixtures/ and expose FIXTURES_DIR from cluster.py so ship_fixture() resolves the correct path after cluster.py was lifted out of e2e/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Spur scheduling/ingestion perf test with held-job submit for metrics 1–2, release + drain for metric 3 (queue wait from controller timestamps, not client RTT).
perf_tests/):run_perf.shfor one tier,run_all.shfor multi-tier sweeps on an existing cluster. Metrics are printed as a stdout KEY=VALUE block; the script fails on partial submits and drain timeout.tests/native_host/e2e/perf_harness/): deploys an ephemeral E2E cluster, runsrun_perf.shover SSH, parses results into JSON, and prints a markdown summary. Multi-tier runs redeploy the cluster between tiers so later tiers are not skewed by accumulated job state.perf_harness.compare): diff two suite JSON files (e.g. PR vs nightly) with configurable regression threshold and tier-mismatch warnings.@pytest.mark.perftest excluded from default E2E (-m "not perf"). New manualperf.ymlworkflow builds PR binaries vs latestnightlyrelease on the bare-metal cluster and prints a comparison report to the workflow log.Design notes
spur submit -HsdiagPRE/POST delta aftersdiag --resetStartTime − SubmitTimeon the same held-then-released jobsShell-only sweeps (
run_all.sh) intentionally reuse one controller (warm state after tier 1). The pytest harness redeploys between tiers for comparable numbers.