litellm provider adapter + a320-bench CLI (#71) - #79
Merged
Conversation
LiteLLMAdapter drives any provider through litellm's OpenAI-format completion API (verified against the pinned 1.93.0: tools/tool_choice params, tool_calls[].function.name/.arguments). MCP tool schemas map 1:1 to function-calling tools; tool results and the nudge go back into the message history; malformed argument JSON becomes empty args so the tool schema rejects it as the agent's recorded error. litellm is pinned exactly as the [providers] extra (the translation layer is part of a run's identity, version recorded in every trajectory meta) and is never imported without it. a320-bench run --scenario ... --model ... --runs N records one JSONL per episode; invalid runs (harness/scenario problems) exit non-zero, a failed procedure is a result, not an error. Assistant records now carry the provider's finish reason and token usage (provider_raw). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…LI errors (#71) Quality review of the litellm slice, checked against the installed litellm 1.93.0 (L-005): - assistant history: omit the tool_calls key on no-call turns instead of sending an explicit null — the OpenAI passthrough puts the message dict verbatim in the request body (llms/openai/chat/gpt_transformation.py:451) and OpenAI rejects null; the Anthropic path reads it with .get() (prompt_templates/factory.py:2539) so nothing changes there. - malformed (or non-object) tool arguments still become {}, but the original payload now survives in Turn.raw[malformed_tool_arguments] so the trajectory keeps the evidence for the scorer. - CLI: --sampling is validated by argparse (JSON object or exit 2), --runs must be >= 1, and a missing [providers] extra or a ScenarioError print a one-line message instead of a traceback. - tests: parallel tool calls (order + history), content alongside tool_calls, sampling forwarded to completion, tool_calls key omitted, usage into raw, malformed payload preserved, CLI argument rejection (L-003: paths the original suite never took). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
Review follow-up on #79: a paid batch whose every run died on a bad key or a network error must not end green. INVALID and ERROR count as infrastructure failures; an agent failing the procedure stays a result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This was referenced Jul 24, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #71 (slice D of the Phase 5 epic #19). Builds on #78 (episode runner).
What
providers/litellm_adapter.py: real providers throughlitellm.completion(OpenAI format), verified against the pinned 1.93.0 —tools/tool_choiceparams andtool_calls[].function.{name,arguments}response shape. MCP tool schemas map 1:1 (inputSchema→function.parameters). Tool results (errors prefixedERROR:) and the runner's nudge go back into history verbatim; malformed argument JSON becomes{}so the server's schema rejects it as the agent's recorded error, never a harness crash.inforecords provider, model, sampling and litellm version into every trajectory's meta.litellm==1.93.0pinned exactly as the[providers]extra — CI installs without it (the module is never imported eagerly; the adapter testsimportorskip).cli.py+a320-benchentry point:a320-bench run --scenario <yaml> --model <litellm id> --runs N --out runs/ [--sampling '{...}']. Exit code is red only for invalid runs (harness/scenario problems); an agent failing the procedure is a result, not an error.episode.py: assistant records now carryprovider_raw(finish reason, token usage) for cost/usage reporting in EPIC: scoring metric + baselines/ablations #20.Tests
python -m pytest bench/tests -q: 32 passed — schema/message mapping, history round-trip with tool_call_id, error marking, malformed-JSON handling, adapter info, CLI parser. No network:litellm.completionis monkeypatched.Pending manual step (needs an API key)
a320-bench run --scenario scenarios/elec/apu_gen_fault.yaml --model anthropic/claude-opus-4-8 --out runs/— the end-to-end smoke with a real model; no provider key is available in this environment.🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm