In-process episode runner + JSONL trajectory recording (#70) - #78
Merged
Conversation
#70) run_episode owns the Sim (setup, world controls, injection schedule, validity gate, success predicates, active_failures as ground truth) and hands the agent a benchmark-profile MCP session over the SDK's memory transport. Every turn and tool call is recorded to a self-contained JSONL trajectory (embedded scenario, tool surface, vendor pin, versions, sim clock around each call) so the #20 scorer never re-simulates. Providers are a minimal sync protocol; ScriptedAdapter plays fixed tool calls so CI drives the whole pipeline with no LLM: the procedure script must pass, a procedure-ignoring script must fail, clear_failure must be a recorded error that repairs nothing, budgets and the one-nudge ending are exercised, and a scenario whose ECAM never manifests aborts as invalid_scenario without consulting the agent. The loader now also cross-checks instructions_profile against INSTRUCTIONS_PROFILES (the reviewer's pending item from #75, possible now that #73 is merged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…03 path tests (#70) Quality-review fixes: document EpisodeResult's valid/all_passed semantics, type _tool_result_text against mcp.types.CallToolResult, and add three tests that drive the runner down paths the author's suite never took (L-003): a malformed report_done as the episode's first call must be a recorded error that ends nothing, an adapter raising in start() must still close the trajectory with a final record, and the sim-time budget must end the episode on the call that burst it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…ate truncated final line Review follow-ups on #78: the scorer must see every message the agent saw (nudge record), a harness crash leaves evidence instead of an empty orphan JSONL, and read_trajectory tolerates a crash-truncated last line while still raising on mid-file corruption. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #70 (slice C of the Phase 5 epic #19). Builds on #73 (benchmark tool profile) and #75 (scenario loader), both merged.
Architecture
The runner is in-process and privileged: it owns the
a320_sim.Simand does the harness work with direct calls — start state,domain=worldcontrols, the injection schedule (after_setup_sor a bounded when-predicate wait), the validity gate, success predicates, andactive_failures()as ground truth. The agent gets a real MCP session over the SDK's memory transport (create_connected_server_and_client_session, verified against mcp 1.28.1: accepts theFastMCPinstance and initializes the session itself,shared/memory.py:50-100) speaking to acreate_server(sim, profile="benchmark")server. One thread, one event loop, freshSim+ server per episode (unsendable, D-010).Validity gate: after injection + settle,
expected_ecam.must_appearmust be on the ECAM or the run ends asinvalid_scenariowithout consulting the agent — scoring a run whose failure never manifested would grade the agent on a scenario that did not happen.Trajectory (JSONL, one file per run):
meta(embedded scenario, adapter info, instructions profile, recorded tool surface, vendor pin, package versions,seed: nullreserved),setup(gate result, active failures, full snapshot),assistant/tool_call(args, result,is_error, sim clock before/after, wall ms),final(reason,report_donepayload, final ECAM/failures/snapshot, per-predicatesuccess_eval). Self-contained: #20 scores without re-simulating.Episode end:
agent_done(report_done) ·end_turn_without_done(two empty turns, one nudge in between) ·budget_tool_calls/budget_sim_time·provider_error(partial trajectory kept) ·invalid_scenario.Also: the scenario loader now cross-checks
instructions_profileagainstINSTRUCTIONS_PROFILES— the reviewer's pending item from #75, unblocked by #73's merge.Tests
python -m pytest bench/tests mcp/tests -q: 33 passed, no LLM, no network. The scripted procedure resolves the scenario (all_passed: true— an executable smoke test of the ground truth against the vendor pin); a procedure-ignoring script fails;clear_failureis a recorded error that repairs nothing (the failure survives to the final record); budgets end the episode at exactly the cap; script exhaustion ends after exactly one nudge; a never-manifesting ECAM aborts asinvalid_scenariowith zero agent tool calls; the sim clock is monotonic across recorded calls.🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm