Add --workload-path to aggregate, for parity with run - #60
Add --workload-path to aggregate, for parity with run#60serhiy-bzhezytskyy wants to merge 3 commits into
Conversation
add_workload_source() offers a workload either as a repository or as a local path, and run, list and info all use it. aggregate defined only --workload-repository, so a workload developed locally and benchmarked with run --workload-path could not be aggregated afterwards: solr-orbit: error: unrecognized arguments: --workload-path=... aggregate now uses the same helper, and goes through configure_workload_params so that a path is normalised and validated the way it is everywhere else -- that is also what rejects --workload-revision together with --workload-path. Since the path sets workload.path, which is what makes the loader choose SimpleWorkloadRepository, aggregate() no longer names a repository when one was given. The fixture supplied no workload_path, so the aggregator saw a Mock attribute that happens to be truthy; it is now explicit, and the two new tests cover each mode.
There was a problem hiding this comment.
Thanks for this fix. I suppose most testing has been done referencing the default workload github repository. But I agree it makes sense to support a path to a local custom workload repo as well for aggregate. I have not dived deep into the aggregate command though.
There is a case for going further — a test run already records the workload it used, so aggregate could resolve the spec from the stored run and need no workload-source argument at all.
It would of course be elegant if we auto resolved the workload location from the stored run. Perhaps as a separate improvement in addition to this more explicit one?
I notice a slight differece between the docs in https://docs.opensearch.org/latest/benchmark/reference/commands/aggregate/#additional-options and https://apache.github.io/solr-orbit/reference/commands/aggregate.html#options. Solr Orbit does not document the --workload-repository option at all. I wonder if it is a documentation bug?
The aggregate reference page listed only --test-runs, --test-runs-id and --results-file, while command-flags.md already documented --workload-repository for aggregate. The two pages disagreed, and neither mentioned --workload-revision, which aggregate has accepted all along via add_workload_source(). Adds --workload-path (new in this PR), --workload-repository and --workload-revision to both pages, using the wording already used for run.
|
Thanks Jan. On auto-resolving the workload from the stored test run — agreed, a separate improvement. This PR keeps to the parity fix ( On the docs: you're right, and it's slightly worse than it looks. The two pages disagreed with each other — Pushed a commit documenting all three on both pages, using the wording already used for Worth noting this doc gap is probably why the parity gap went unnoticed: with |
|
Filed the auto-resolve follow-up as #61 so it isn't lost. One thing worth flagging that came out of writing it up: the stored test run doesn't currently record where the workload came from. |
Description
add_workload_source()offers a workload either as a repository or as a local path, andrun,listandinfoall use it.aggregatedefined only its own--workload-repository, so a workload developedlocally and benchmarked with
run --workload-pathcould not be aggregated afterwards:aggregatenow uses the same helper, so it accepts either source, keeps the two mutually exclusive, andgains
--workload-revisionfor free. It also goes throughconfigure_workload_params, so a path isnormalised and validated the way it is everywhere else — that is also what rejects
--workload-revisiontogether with
--workload-path.command_requires_workload=Falsebecauseaggregatetakes the workloadname from the stored test run, not from the command line.
One behavioural detail: a path sets
workload.path, which is what makes the loader chooseSimpleWorkloadRepository. So when a path was given,aggregate()no longer names a repository — doingboth would send the loader looking for the workload inside that repository instead. That guard is
load-bearing, since argparse's default for
--workload-repositoryis the string"default", notNone.Issues Resolved
Resolves #57
Testing
New functionality includes testing
Two new tests:
test_aggregate_names_the_workload_repository(unchanged behaviour when a repository isused) and
test_aggregate_leaves_a_workload_path_alone(norepository.nameis configured when a pathis given).
The shared
mock_argsfixture previously supplied noworkload_path, so the aggregator saw aMockattribute that happens to be truthy. It is now explicit — that fixture change is what makes the first
test meaningful rather than accidentally passing.
CLI verified by hand on this branch:
aggregate --workload-path=...is accepted where it previouslyerrored, and combining it with
--workload-revisionis correctly rejected.Full suite:
1100 passed, 5 skipped, plus one unrelated pre-existing flake described below.ruff checkclean.On the flake, so it isn't mistaken for fallout from this PR:
tests/workload/workload_test.py::DocumentCorpusTests::test_union_document_corpora_is_symmetricfails onroughly one interpreter start in eight.
DocumentCorpus.unionbuilds its document list withlist(set(...).union(...))whileDocumentCorpus.__eq__compares that list in order, so symmetry dependson hash iteration order. It reproduces deterministically under
PYTHONHASHSEED=7onmain's untouchedworkload code and on the other two fix branches, and this branch does not touch
solrorbit/workload/atall. I'll report it separately rather than fold an unrelated fix in here.
Notes
Found while running a multi-configuration geonames campaign comparing Solr on Lucene 10.4 against
Lucene 11. Third of three
aggregategaps, and the only one that is a parity gap rather than a bug.There is a case for going further — a test run already records the workload it used, so
aggregatecouldresolve the spec from the stored run and need no workload-source argument at all. I've gone with the
parity fix because it is the smaller change and matches the existing pattern, but I'm open to the other
shape if you'd rather have it.