docs: restructure benchmarking design docs to match actual scope

[michael-johnston]

docs: restructure benchmarking design docs to match actual scope

What is the problem with the docs

The three benchmarking design documents sound like they form a hierarchy, but they do not. A reader who starts at Benchmarking System expects an umbrella design. What they get is the execution and operations design, written first, with later work parked in two other files whose names sound like "more metadata."

That mismatch shows up in three ways:

1. Names do not match scope. "System" claims everything. "Integration" and "Metadata convention" both sound like how metadata is attached to packages, so it is hard to tell them apart.
2. The two-tier story is incomplete. benchmark_system.md §1.1 says the architecture is two-tiered (ado experiments vs Nexus integration). A third concern — logical benchmarks, instances, and result aggregation — has no place in that picture, and the detailed "Tier 2" lives in another file that does not say it is Tier 2.
3. Cross-links and leftovers are stale. benchmark_system.md still marks REQ 2.3 and REQ 7.1 as PENDING. Those decisions were later made in the integration doc. Nothing at the top of the design index tells you which file answers which requirement group.

The two later docs are also easy to confuse because both are about registering things. They register different things:

• Run-side: benchmark experiments and benchmark submissions (runs of those experiments on a problem)
• Problem-side: logical benchmarks and benchmark instances

Current file	Title a reader expects	What the file actually designs	Specific issues
docs/design/benchmark_system.md	The whole benchmarking design	How experiments are executed and operated: ado + Ray, GitHub triggers, admin cluster, versioning conventions (REQ-1, REQ-4, REQ-5.1, REQ-6, REQ-7)	Name is too broad. Mixes a high-level architecture sketch with ops detail. Still lists REQ 2.3 / REQ 7.1 as open. The two-tier table hides the later aggregation work.
docs/design/benchmark_integration_design.md	How the system plugs into Nexus	How to add benchmark experiments and benchmark submissions (runs of those experiments on a problem): nexus.yaml, benchmark_packages/, benchmark_submissions/space.yaml (REQ-2, REQ-3)	"Integration" does not say experiments and submissions. Easy to confuse with the metadata convention, which is also about attaching metadata.
docs/design/benchmark_metadata_convention.md	A naming convention for extra fields	How to add logical benchmarks and benchmark instances, plus (for now) how an experiment binds to a logical benchmark so results can be aggregated (REQ-5.2 / 5.3)	"Convention" undersells the schemas. Relation to the integration doc is only "builds on," not "problem-side vs run-side." Binding could later move to the experiments/submissions doc.

Proposed Solution

Treat benchmarking as one short overview plus three focused designs, named for the artifacts they cover and the requirement groups they close. Do not keep benchmark_system or benchmark_registration as filenames — the first will still be read as the whole design, and the second does not distinguish run-side registration from problem-side registration.

Proposed layout

docs/design/benchmarking.md                         overview: concepts, layers, REQ → doc map
  ├── benchmark_execution.md                        REQ-1, 4, 6, 7   (today’s system doc, trimmed)
  ├── benchmark_experiments_and_submissions.md      REQ-2, 3         (today’s integration doc)
  └── logical_benchmarks_and_instances.md           REQ-5.2 / 5.3    (today’s metadata convention)

New file	Title	Content
benchmarking.md	Benchmarking Architecture	Entry point only: terminology, three layers, and a table mapping REQ-1…7 to the child docs. No schemas.
benchmark_execution.md	Benchmark Execution and Operations	ado experiment packaging, Ray, result store, GitHub/admin triggers, sweep governance. Point at experiments/submissions for REQ 2.3 / 7.1 instead of leaving them PENDING.
benchmark_experiments_and_submissions.md	Benchmark Experiments and Submissions	Adding experiments (benchmark_packages, nexus.yaml) and submissions (benchmark_submissions/space.yaml), discovery CLI.
logical_benchmarks_and_instances.md	Logical Benchmarks and Instances	Adding logical benchmark definitions and instances. Experiment binding stays here for now. Metric mapping and aggregation.

The overview should state three layers so each child doc has a clear parent:

1. Execution — ado experiment packages, Ray, result store
2. Experiments and submissions — which experiments a package exposes, which submissions run them
3. Logical benchmarks and instances — shared problem definition, concrete instances, comparable results

Implementation notes

• git mv the three existing files, then add benchmarking.md as the new index.
• Update docs/design/index.md to list the overview first, then the three children with one-line scopes.
• Update cross-links in the design docs and docs/contributing/add_benchmark.md.
• Do not rewrite the technical content except to fix stale PENDING items and the two-tier wording.
• Leave experiment binding in logical_benchmarks_and_instances.md for now. It may later move to benchmark_experiments_and_submissions.md.

[michael-johnston]

Actually I would probably change following

benchmark_registration.md -> registring benchmark experiments and submissions
logical_benchmarks.md -> registering/adding new logical benchmarks and benchmark instances

Edit: Above changes made in description

[christian-pinto]

Makes sense to me, I just wanted point out that with this new schema we can completely decouple nexus packages (only for software delivery) from benchmarking.

I would order the three layers as:

1. Experiment registration
2. Experiment mapping to logical benchmarks and instances
3. Experiment execution

Why:

• In a PR arriving soon the benchmark binding becomes part of the experiment itself.
• Experiments will use artifacts from the logical benchmark instances.

[michael-johnston]

Yes the binding is the glue between the problem definition (logical benchmark + instance) and the runtime side (experiments+susbmission) it can live on either side (currently in benchmark, we move to experiment), similarly the docs on it could also lie in either side.

As a first easiest restricting I suggest leaving the bindings in with logical benchmarks and then the flow is

• Logical Benchmark and Instacnce definition (temporarily hold the bindings)
• Experiment registration and benchmark submissions (you need the instances that are defined previously, bindings could be here)
• Experiment Execution