Experimental protocol + matrix runbook (#90) - #94
Merged
Conversation
Slice I of epic #20, docs only (blocked on provider keys for the real matrix): the metric doc gains an Experimental protocol section (>=2 models, N=10/cell power justification, ablation axes as existing levers — instructions profile, sampling, tool surface, QRH access — and what the vector reports), and bench/README.md gains the exact run+score runbook the matrix executes. No new code: run records, score measures, the matrix is a loop over the two. Keys stay in the environment, never in a trajectory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #90 (slice I of epic #20). Docs only — blocked on provider API keys for the real matrix, but the gap is fully specified so it runs letter-by-letter the day keys exist.
What
docs/fase5-metrica.mdgains an Experimental protocol section: the ≥2 baselines (Claude + Gemini, both validated end-to-end), N=10 runs/cell with a power justification, the ablation axes expressed as existing harness levers (instructions profile per D-023/D-017,--sampling, tool-surface profile, QRH access) — zero new code — and what the aggregate vector reports (not just the scalar).bench/README.mdgains the runbook: the exacta320-bench run ... && a320-bench score runs/ --jsoncommands the matrix executes, including Gemini's Vertex-expressapi_basepassthrough via--sampling. Keys stay in the environment, never in a file or trajectory.No CI/network dependency;
scoreruns binding-free on a reviewer's machine.🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm