Skip to content

run: accept --running_platform so platforms get separate series - #548

Draft
paulorsousa wants to merge 1 commit into
masterfrom
feat/running-platform-run-args
Draft

run: accept --running_platform so platforms get separate series#548
paulorsousa wants to merge 1 commit into
masterfrom
feat/running-platform-run-args

Conversation

@paulorsousa

Copy link
Copy Markdown
Contributor

Problem

running_platform is already:

  • part of the time-series key, in get_ts_metric_name (utils/utils.py)
  • a label, in get_project_ts_tags (utils/remote.py)
  • threaded through common_exporter_logic and timeseries_test_sucess_flow
  • filterable in compare, via --running_platform (compare/args.py)

Only the run entry points never supply it. run_remote.py and run_local.py both
call timeseries_test_sucess_flow without it, and no run argument exists to set
it, so in practice it is always None.

The consequence is that benchmark runs on different hardware share a single
series. Tagging cannot substitute for this: check_rts_labels applies TS.ALTER
whenever incoming labels differ from the key's, so labels describe the series
and the most recent push wins. Provenance cannot be recovered per sample.

Observed while building regression coverage for RediSearch: an m7i.8xlarge VM
baseline and an m7i.metal-24xl run against the same commit read back as one
8-sample series. Its confidence interval converged happily, around a mean that
described neither platform. Nothing on the surface looked wrong.

Change

Adds --running_platform to common_run_args, beside --architecture — its
closest analogue, since both describe the machine the benchmark ran on and both
belong in the key. Both entry points forward it. 13 lines, no new plumbing.

Compatibility

Leaving the flag unset produces byte-identical keys, so no existing series move
and no historical data is invalidated. Setting it starts a new series per
platform, which is the intent.

unset : ci.benchmarks.redislabs/by.hash/circleci/O/R/t/oss-standalone/<sha>/Ops/sec
m7    : ci.benchmarks.redislabs/by.hash/circleci/O/R/t/redisearch-m7/oss-standalone/<sha>/Ops/sec
metal : ci.benchmarks.redislabs/by.hash/circleci/O/R/t/redisearch-m7i-metal/oss-standalone/<sha>/Ops/sec

Verification

Checked directly against the key-building helpers:

  • m7 and m7i-metal produce different keys
  • unset produces a key byte-identical to before the change
  • the label is present when set and absent when unset
  • --running_platform parses in both run-remote and run-local

Caveat for reviewers: this was verified by exercising the pure key-building
helpers with third-party imports stubbed, not by running the project's own test
suite, which was not available in the environment used.

Deliberately out of scope

export_redis_metrics is untouched. It builds its own key prefix for server-side
metrics, which are not used for regression gating, and changing that prefix would
relocate a separate metric family.

Note for consumers

Callers pinning redisbench_admin>=0.12.29 from PyPI will not have this flag
until it is released; passing it against a released version fails with
unrecognized arguments.

🤖 Generated with Claude Code

running_platform was already part of the time-series key in get_ts_metric_name,
already a label in get_project_ts_tags, already threaded through
common_exporter_logic and timeseries_test_sucess_flow, and already filterable in
compare via --running_platform. Only the run entry points never supplied it:
run_remote.py and run_local.py both called timeseries_test_sucess_flow without
it, and no run argument existed to set it.

The consequence is that runs on different hardware share one series. Tagging
cannot substitute, because check_rts_labels applies TS.ALTER whenever incoming
labels differ from the key's, so labels describe the series and the latest push
wins. Observed on RediSearch benchmarks: an m7i.8xlarge VM baseline and an
m7i.metal-24xl run against the same commit read back as a single 8-sample
series, and its interval converged around a mean describing neither platform.

The flag lives in common_run_args beside --architecture, which is its closest
analogue: both describe the machine, and both belong in the key. Leaving it
unset keeps keys byte-identical to before, so no existing series move.

export_redis_metrics is deliberately untouched. It builds its own key prefix for
server-side metrics, which are not used for gating, and changing that prefix
would relocate a separate metric family.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.67%. Comparing base (aca22be) to head (449e375).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #548      +/-   ##
==========================================
+ Coverage   47.63%   47.67%   +0.03%     
==========================================
  Files          74       74              
  Lines        8917     8918       +1     
==========================================
+ Hits         4248     4252       +4     
+ Misses       4669     4666       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant