a320-bench score CLI + aggregation (#86) - #93
Merged
Conversation
score PATH... [--json] [--detail] [--include-errors] turns recorded trajectories (files or run directories, recursive) into ScoreCards: a human table (per-run rows + per scenario x model aggregates with mean/std, pass/dangerous/error/invalid rates) or JSON for plots. Kept in cli_score.py, binding-free: a reviewer scores with pip install -e bench/ alone. Exit 2 on an unreadable/corrupt trajectory or no files found; a badly-scored run is a result, not an error. Tests generate the four acceptance fixtures in-test with the ScriptedAdapter over the real apu_gen_fault scenario and assert the strict score ordering end to end through score_file, plus --json/--detail/exit codes. Verified on the real Gemini smoke trajectory: scored 0.90 (coverage 0.75 for the omitted isolation step, order 1.00, end_state 1.00) -- the metric assigns the predicted value to the actual real-model run. 94 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
… paths (#86) cli.py imported a320_bench.episode (which imports the compiled a320_sim) and a320_bench.scenario at module top, so `a320-bench score` — the entry point a320_bench.cli:main — failed to even start on a reviewer's binding-free machine (pip install -e bench/ alone), defeating the slice's whole point. The narrow `import a320_bench.cli_score` check passed and hid it. Both imports are now deferred into the run/serve paths that need them, mirroring the existing lazy-litellm pattern; the score dispatch touches neither. _collect now collapses overlapping arguments (a file passed twice, or a directory plus a file inside it) by resolved path, so a run is never double-counted into the aggregate means. Added type hints to the CLI's public helpers. Tests: a binding-free score-dispatch regression (a320_sim/litellm blocked in sys.modules), path dedupe, good-file + empty-dir -> exit 0, and a truncated/incomplete run serializing as JSON null. 98 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
… not the leaf module) The #86 review caught a320-bench score pulling a320_sim via a module-top import in cli.py while the test only checked the clean leaf module. Rule: test the no-binding property through cli:main with the dependency blocked in sys.modules, and keep binding/optional imports lazy inside their branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #86 (slice G of epic #20). Builds on the scoring module (#91, merged).
What
a320-bench score PATH... [--json] [--detail] [--include-errors](cli_score.py, binding-free): recorded trajectories — files or run directories (recursive**/*.jsonl, matchingrun's layout) — become ScoreCards. Default output is a human table (per-run rows + per(scenario, model)aggregates: mean/std of score and components, pass/dangerous/provider-error/invalid rates);--jsonemits ScoreCards + aggregates for the paper's plots;--detailadds the per-command classification. Exit 2 on an unreadable/corrupt trajectory or no files found; a badly-scored run is a result, not an error.cli.pyand importing only the pure scoring layer, so a reviewer scores results withpip install -e bench/alone (noa320_sim, no litellm).cli.pydispatchesscorebefore any scenario load.Tests
94 passed (
pytest bench/tests mcp/tests). The four acceptance fixtures are generated in-test with the ScriptedAdapter over the realapu_gen_faultscenario and the strict score ordering is asserted end to end throughscore_file(full 1.00 > omitted 0.90 > ext-pwr-only 0.55 > nothing 0.00); plus--jsonover a directory parses with a single aggregate group,--detailcarries the command classification, the human table renders, and both exit-2 paths.Real-trajectory check: run against the actual Gemini smoke trajectory, the scorer reports 0.90 (coverage 0.75 for the omitted isolation step, order 1.00, end_state 1.00) — the metric assigns the predicted value to the real run that omitted the step.
🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm