Skip to content

Expand evaluation cost reporting - #518

Merged
DhavalRepo18 merged 1 commit into
mainfrom
feat/evaluation-cost-reporting
Aug 18, 2026
Merged

Expand evaluation cost reporting#518
DhavalRepo18 merged 1 commit into
mainfrom
feat/evaluation-cost-reporting

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Description

  • add pricing for new Claude, GPT, Gemini, and MiniMax models
  • split aggregate estimated cost into input and output totals
  • add per-runner/model pass rates, operational metrics, cost per scenario, and cost per pass to score summaries
  • render per-runner/model pass counts in the text summary
  • add regression coverage for cost components and report fields

Type of Change

  • New Benchmark Scenario (Industry/Asset type)
  • Evaluation Metric / Scorer
  • Agentic Orchestration Logic (ReAct, Plan-Execute, etc.)
  • Infrastructure / Tooling Improvement

Industry Relevance

Makes benchmark quality and estimated operating cost directly comparable across runner/model combinations.

Related Issues

None.

Testing & Validation

  • Unit Tests: uv run pytest src/evaluation/tests/test_metrics.py src/evaluation/tests/test_report.py (18 passed)
  • Full evaluation suite: uv run pytest src/evaluation/tests (90 passed, 7 unrelated failures: one reflects the existing local lite.yaml edit; six exercise static-JSON CAR metadata not implemented on the base branch)
  • Scenario Validation: Not run; this change does not alter agent trajectory execution.
  • Data Integrity: No PII or sensitive industrial data is included.

Checklist

  • My code follows the project formatting conventions and passes git diff --check.
  • I have performed a self-review of my code.
  • I have updated the documentation (README or /docs) accordingly.
  • I have signed off my commits (DCO).

Signed-off-by: root <root@bench.vpc.cloud9.ibm.com>
@ShuxinLin
ShuxinLin requested a review from DhavalRepo18 August 18, 2026 20:41
@DhavalRepo18
DhavalRepo18 merged commit 9143a63 into main Aug 18, 2026
5 checks passed
@ShuxinLin
ShuxinLin deleted the feat/evaluation-cost-reporting branch August 18, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants