Skip to content

feat(benchmarks): add model matrix run script - #521

Merged
DhavalRepo18 merged 1 commit into
mainfrom
feat/benchmark-run-script
Aug 20, 2026
Merged

feat(benchmarks): add model matrix run script#521
DhavalRepo18 merged 1 commit into
mainfrom
feat/benchmark-run-script

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Description

Adds an executable benchmarks/run.sh launcher for running the lite scenario suite across the configured eight-model Stirrup benchmark matrix. The script accepts scenario and leaderboard roots via positional arguments or environment variables, validates required inputs, skips existing trajectories, and continues across individual scenario errors.

Type of Change

  • New Benchmark Scenario (Industry/Asset type)
  • Evaluation Metric / Scorer
  • Agentic Orchestration Logic (ReAct, Plan-Execute, etc.)
  • Infrastructure / Tooling Improvement

Industry Relevance

Provides a repeatable model-comparison entry point and consistent output layout for benchmark trajectories, reports, and Stirrup workspaces.

Related Issues

  • None

Testing & Validation

  • Unit Tests: 43 passed in src/benchmark/tests/test_scenario_suite_runner.py.
  • Scenario Validation: Mocked uv and verified all eight correctly quoted runner invocations.
  • Shell Validation: bash -n benchmarks/run.sh and missing-argument behavior passed.
  • Data Integrity: No benchmark data, PII, credentials, or sensitive industrial data added.

Checklist

  • The script follows repository shell conventions and passes syntax validation.
  • I have performed a self-review.
  • The script includes inline usage guidance for required inputs.
  • I have signed off the commit (DCO).

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@DhavalRepo18 DhavalRepo18 self-assigned this Aug 20, 2026
@DhavalRepo18
DhavalRepo18 self-requested a review August 20, 2026 20:43
@DhavalRepo18
DhavalRepo18 merged commit 3a7bd23 into main Aug 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants