run: accept --running_platform so platforms get separate series - #548
Draft
paulorsousa wants to merge 1 commit into
Draft
run: accept --running_platform so platforms get separate series#548paulorsousa wants to merge 1 commit into
paulorsousa wants to merge 1 commit into
Conversation
running_platform was already part of the time-series key in get_ts_metric_name, already a label in get_project_ts_tags, already threaded through common_exporter_logic and timeseries_test_sucess_flow, and already filterable in compare via --running_platform. Only the run entry points never supplied it: run_remote.py and run_local.py both called timeseries_test_sucess_flow without it, and no run argument existed to set it. The consequence is that runs on different hardware share one series. Tagging cannot substitute, because check_rts_labels applies TS.ALTER whenever incoming labels differ from the key's, so labels describe the series and the latest push wins. Observed on RediSearch benchmarks: an m7i.8xlarge VM baseline and an m7i.metal-24xl run against the same commit read back as a single 8-sample series, and its interval converged around a mean describing neither platform. The flag lives in common_run_args beside --architecture, which is its closest analogue: both describe the machine, and both belong in the key. Leaving it unset keeps keys byte-identical to before, so no existing series move. export_redis_metrics is deliberately untouched. It builds its own key prefix for server-side metrics, which are not used for gating, and changing that prefix would relocate a separate metric family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #548 +/- ##
==========================================
+ Coverage 47.63% 47.67% +0.03%
==========================================
Files 74 74
Lines 8917 8918 +1
==========================================
+ Hits 4248 4252 +4
+ Misses 4669 4666 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
running_platformis already:get_ts_metric_name(utils/utils.py)get_project_ts_tags(utils/remote.py)common_exporter_logicandtimeseries_test_sucess_flowcompare, via--running_platform(compare/args.py)Only the run entry points never supply it.
run_remote.pyandrun_local.pybothcall
timeseries_test_sucess_flowwithout it, and no run argument exists to setit, so in practice it is always
None.The consequence is that benchmark runs on different hardware share a single
series. Tagging cannot substitute for this:
check_rts_labelsappliesTS.ALTERwhenever incoming labels differ from the key's, so labels describe the series
and the most recent push wins. Provenance cannot be recovered per sample.
Observed while building regression coverage for RediSearch: an
m7i.8xlargeVMbaseline and an
m7i.metal-24xlrun against the same commit read back as one8-sample series. Its confidence interval converged happily, around a mean that
described neither platform. Nothing on the surface looked wrong.
Change
Adds
--running_platformtocommon_run_args, beside--architecture— itsclosest analogue, since both describe the machine the benchmark ran on and both
belong in the key. Both entry points forward it. 13 lines, no new plumbing.
Compatibility
Leaving the flag unset produces byte-identical keys, so no existing series move
and no historical data is invalidated. Setting it starts a new series per
platform, which is the intent.
Verification
Checked directly against the key-building helpers:
m7andm7i-metalproduce different keys--running_platformparses in bothrun-remoteandrun-localCaveat for reviewers: this was verified by exercising the pure key-building
helpers with third-party imports stubbed, not by running the project's own test
suite, which was not available in the environment used.
Deliberately out of scope
export_redis_metricsis untouched. It builds its own key prefix for server-sidemetrics, which are not used for regression gating, and changing that prefix would
relocate a separate metric family.
Note for consumers
Callers pinning
redisbench_admin>=0.12.29from PyPI will not have this flaguntil it is released; passing it against a released version fails with
unrecognized arguments.
🤖 Generated with Claude Code