Skip to content

Add --workload-path to aggregate, for parity with run - #60

Open
serhiy-bzhezytskyy wants to merge 3 commits into
apache:mainfrom
serhiy-bzhezytskyy:fix/aggregate-workload-path
Open

Add --workload-path to aggregate, for parity with run#60
serhiy-bzhezytskyy wants to merge 3 commits into
apache:mainfrom
serhiy-bzhezytskyy:fix/aggregate-workload-path

Conversation

@serhiy-bzhezytskyy

Copy link
Copy Markdown
Contributor

Description

add_workload_source() offers a workload either as a repository or as a local path, and run, list and
info all use it. aggregate defined only its own --workload-repository, so a workload developed
locally and benchmarked with run --workload-path could not be aggregated afterwards:

solr-orbit: error: unrecognized arguments: --workload-path=/path/to/my-workload

aggregate now uses the same helper, so it accepts either source, keeps the two mutually exclusive, and
gains --workload-revision for free. It also goes through configure_workload_params, so a path is
normalised and validated the way it is everywhere else — that is also what rejects --workload-revision
together with --workload-path. command_requires_workload=False because aggregate takes the workload
name from the stored test run, not from the command line.

One behavioural detail: a path sets workload.path, which is what makes the loader choose
SimpleWorkloadRepository. So when a path was given, aggregate() no longer names a repository — doing
both would send the loader looking for the workload inside that repository instead. That guard is
load-bearing, since argparse's default for --workload-repository is the string "default", not None.

Issues Resolved

Resolves #57

Testing

  • New functionality includes testing

  • Two new tests: test_aggregate_names_the_workload_repository (unchanged behaviour when a repository is
    used) and test_aggregate_leaves_a_workload_path_alone (no repository.name is configured when a path
    is given).

  • The shared mock_args fixture previously supplied no workload_path, so the aggregator saw a Mock
    attribute that happens to be truthy. It is now explicit — that fixture change is what makes the first
    test meaningful rather than accidentally passing.

  • CLI verified by hand on this branch: aggregate --workload-path=... is accepted where it previously
    errored, and combining it with --workload-revision is correctly rejected.

  • Full suite: 1100 passed, 5 skipped, plus one unrelated pre-existing flake described below.
    ruff check clean.

On the flake, so it isn't mistaken for fallout from this PR:
tests/workload/workload_test.py::DocumentCorpusTests::test_union_document_corpora_is_symmetric fails on
roughly one interpreter start in eight. DocumentCorpus.union builds its document list with
list(set(...).union(...)) while DocumentCorpus.__eq__ compares that list in order, so symmetry depends
on hash iteration order. It reproduces deterministically under PYTHONHASHSEED=7 on main's untouched
workload code and on the other two fix branches, and this branch does not touch solrorbit/workload/ at
all. I'll report it separately rather than fold an unrelated fix in here.

Notes

Found while running a multi-configuration geonames campaign comparing Solr on Lucene 10.4 against
Lucene 11. Third of three aggregate gaps, and the only one that is a parity gap rather than a bug.

There is a case for going further — a test run already records the workload it used, so aggregate could
resolve the spec from the stored run and need no workload-source argument at all. I've gone with the
parity fix because it is the smaller change and matches the existing pattern, but I'm open to the other
shape if you'd rather have it.

add_workload_source() offers a workload either as a repository or as a local
path, and run, list and info all use it. aggregate defined only
--workload-repository, so a workload developed locally and benchmarked with
run --workload-path could not be aggregated afterwards:

  solr-orbit: error: unrecognized arguments: --workload-path=...

aggregate now uses the same helper, and goes through configure_workload_params
so that a path is normalised and validated the way it is everywhere else --
that is also what rejects --workload-revision together with --workload-path.
Since the path sets workload.path, which is what makes the loader choose
SimpleWorkloadRepository, aggregate() no longer names a repository when one
was given.

The fixture supplied no workload_path, so the aggregator saw a Mock attribute
that happens to be truthy; it is now explicit, and the two new tests cover
each mode.

@janhoy janhoy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix. I suppose most testing has been done referencing the default workload github repository. But I agree it makes sense to support a path to a local custom workload repo as well for aggregate. I have not dived deep into the aggregate command though.

There is a case for going further — a test run already records the workload it used, so aggregate could resolve the spec from the stored run and need no workload-source argument at all.

It would of course be elegant if we auto resolved the workload location from the stored run. Perhaps as a separate improvement in addition to this more explicit one?

I notice a slight differece between the docs in https://docs.opensearch.org/latest/benchmark/reference/commands/aggregate/#additional-options and https://apache.github.io/solr-orbit/reference/commands/aggregate.html#options. Solr Orbit does not document the --workload-repository option at all. I wonder if it is a documentation bug?

The aggregate reference page listed only --test-runs, --test-runs-id and
--results-file, while command-flags.md already documented
--workload-repository for aggregate. The two pages disagreed, and neither
mentioned --workload-revision, which aggregate has accepted all along via
add_workload_source().

Adds --workload-path (new in this PR), --workload-repository and
--workload-revision to both pages, using the wording already used for run.
@serhiy-bzhezytskyy

Copy link
Copy Markdown
Contributor Author

Thanks Jan.

On auto-resolving the workload from the stored test run — agreed, a separate improvement. This PR keeps to the parity fix (run accepts a workload source, aggregate should too); I can open a follow-up issue for the auto-resolve idea so it isn't lost.

On the docs: you're right, and it's slightly worse than it looks. The two pages disagreed with each other — command-flags.md already listed --workload-repository under "aggregate flags", while reference/commands/aggregate.md documented only --test-runs, --test-runs-id and --results-file. Neither page mentioned --workload-revision, which aggregate has accepted all along, since it calls the same add_workload_source() helper as run.

Pushed a commit documenting all three on both pages, using the wording already used for run. --workload-path and --workload-repository are mutually exclusive (argparse group), so they read the same way there as elsewhere.

Worth noting this doc gap is probably why the parity gap went unnoticed: with --workload-repository undocumented on that page, the absence of --workload-path beside it wasn't visible to a reader.

@serhiy-bzhezytskyy

Copy link
Copy Markdown
Contributor Author

Filed the auto-resolve follow-up as #61 so it isn't lost.

One thing worth flagging that came out of writing it up: the stored test run doesn't currently record where the workload came from. TestRun.as_dict() persists the workload name, workload-params and workload-revision, and from_dict() restores those three — but nothing says whether the run used the default repository, another repository, or a local path. So auto-resolving would need that field persisted first, plus a fallback for test-run files written before the change (there's precedent for that grace period in metrics.py). The issue spells that out, along with the two decisions that seem worth settling before any code: whether the field is optional-with-fallback indefinitely or on a deprecation clock, and whether it's worth coordinating with OpenSearch Benchmark, since aggregator.py there is the same file apart from the import.

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aggregate has no --workload-path, so runs made with --workload-path can't be aggregated

3 participants