Phase 5 (epic #19): scenario suite infrastructure — schema, runner, trajectories, providers - #81
Merged
Conversation
#70) run_episode owns the Sim (setup, world controls, injection schedule, validity gate, success predicates, active_failures as ground truth) and hands the agent a benchmark-profile MCP session over the SDK's memory transport. Every turn and tool call is recorded to a self-contained JSONL trajectory (embedded scenario, tool surface, vendor pin, versions, sim clock around each call) so the #20 scorer never re-simulates. Providers are a minimal sync protocol; ScriptedAdapter plays fixed tool calls so CI drives the whole pipeline with no LLM: the procedure script must pass, a procedure-ignoring script must fail, clear_failure must be a recorded error that repairs nothing, budgets and the one-nudge ending are exercised, and a scenario whose ECAM never manifests aborts as invalid_scenario without consulting the agent. The loader now also cross-checks instructions_profile against INSTRUCTIONS_PROFILES (the reviewer's pending item from #75, possible now that #73 is merged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…03 path tests (#70) Quality-review fixes: document EpisodeResult's valid/all_passed semantics, type _tool_result_text against mcp.types.CallToolResult, and add three tests that drive the runner down paths the author's suite never took (L-003): a malformed report_done as the episode's first call must be a recorded error that ends nothing, an adapter raising in start() must still close the trajectory with a final record, and the sim-time budget must end the episode on the call that burst it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…ate truncated final line Review follow-ups on #78: the scorer must see every message the agent saw (nudge record), a harness crash leaves evidence instead of an empty orphan JSONL, and read_trajectory tolerates a crash-truncated last line while still raising on mid-file corruption. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
In-process episode runner + JSONL trajectory recording (#70)
LiteLLMAdapter drives any provider through litellm's OpenAI-format completion API (verified against the pinned 1.93.0: tools/tool_choice params, tool_calls[].function.name/.arguments). MCP tool schemas map 1:1 to function-calling tools; tool results and the nudge go back into the message history; malformed argument JSON becomes empty args so the tool schema rejects it as the agent's recorded error. litellm is pinned exactly as the [providers] extra (the translation layer is part of a run's identity, version recorded in every trajectory meta) and is never imported without it. a320-bench run --scenario ... --model ... --runs N records one JSONL per episode; invalid runs (harness/scenario problems) exit non-zero, a failed procedure is a result, not an error. Assistant records now carry the provider's finish reason and token usage (provider_raw). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…ons, Python CI job (#72) D-022..D-025 in docs/decisiones.md: benchmark determinism as checkpoints-with-tolerance without patching the vendor RNG (reinterpreting #19's deterministic replay; seed field reserved in every trajectory), benchmark tool profile (no inject/clear, report_done), in-process runner over the memory transport with the self-contained JSONL trajectory, and litellm pinned exactly as the provider layer. Second scenario: elec.gen.1 on engines-running — the network never sags (bus tie feeds AC 1 from GEN 2), so the exam is restraint: reset attempt, isolate, and keep both engines running; shutting an engine down over a generator fault is the classic forbidden action. Empirically verified on pin 13bce4b, scripted-procedure test included. scenarios/README.md documents the scenario anatomy and the citation policy (procedure identity + public secondary source + fidelity notes with pin and date; FCOM wording never reproduced). CI gets the Python layer for the first time: maturin build of bindings/ (cargo cache on its workspace) + pytest over mcp/tests and bench/tests, wired into ci-success. No [providers] extra: CI runs no LLM and no network. a320_bench imports go lazy (PEP 562) so the #20 scorer can read trajectories without the compiled binding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…LI errors (#71) Quality review of the litellm slice, checked against the installed litellm 1.93.0 (L-005): - assistant history: omit the tool_calls key on no-call turns instead of sending an explicit null — the OpenAI passthrough puts the message dict verbatim in the request body (llms/openai/chat/gpt_transformation.py:451) and OpenAI rejects null; the Anthropic path reads it with .get() (prompt_templates/factory.py:2539) so nothing changes there. - malformed (or non-object) tool arguments still become {}, but the original payload now survives in Turn.raw[malformed_tool_arguments] so the trajectory keeps the evidence for the scorer. - CLI: --sampling is validated by argparse (JSON object or exit 2), --runs must be >= 1, and a missing [providers] extra or a ScenarioError print a one-line message instead of a traceback. - tests: parallel tool calls (order + history), content alongside tool_calls, sampling forwarded to completion, tool_calls key omitted, usage into raw, malformed payload preserved, CLI argument rejection (L-003: paths the original suite never took). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
Review follow-up on #79: a paid batch whose every run died on a bad key or a network error must not end green. INVALID and ERROR count as infrastructure failures; an agent failing the procedure stays a result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
Phase 5 decisions, ENG 1 GEN scenario, citation policy, Python CI job (#72)
litellm provider adapter + a320-bench CLI (#71)
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration PR for the #19 epic's five slices, all merged into dev and reviewed:
report_done, noinject_failure/clear_failure)a320-benchCLI52 tests green on integrated dev (
pytest bench/tests mcp/tests), plus the full Rust suite. The Python layer is now covered by CI for the first time (thepythonjob ran green on #80).Epic #19 itself stays open: coverage beyond ELEC (HYD scenarios) and the real-model smoke run (
a320-bench run --scenario scenarios/elec/apu_gen_fault.yaml --model anthropic/claude-opus-4-8, needs an API key) are still pending, and #20 (scoring) is untouched.Closes #68
Closes #69
Closes #70
Closes #71
Closes #72
🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm