feat(cli): add paired evidence comparison - #290
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Eval authors often need to inspect what changed between two conditions before they know which
assertions are worth measuring. The bounded evidence already exists after
ingest, but it is splitacross condition and run directories. That makes open-ended exploration awkward and encourages
assertions to be written before seeing realistic behavior.
Closes #263.
Strategy
Add
eval-magic compareas a post-ingest, pre-judging operation. It selects an iteration and evalID, then pairs the persisted
judge-evidence.mdbundles for both conditions. The report preservesthe same bounded prompt, final message, diff, changed-file list, conversation, tool-use, and
provenance evidence used by judges.
The generated report lives at
iteration-N/compare/<eval-id>.md, and the command prints that pathso a driving agent can open it directly. Multi-run cells are paired and labelled by numeric run
index.
Generated runbooks include one exact compare command per selected eval after
ingest. The judgingguide describes how to turn exploratory observations into concrete assertions and repeated or
sampled eval runs.
Example
Without a paired report, an operator has to locate and correlate each condition's evidence bundle
manually. The paired workflow is:
The report places both conditions together for each matching run.
Safety and compatibility
ingestevidence,but not judge dispatch or finalization.
for every arm. A validation failure does not replace a prior complete report.
wrapper preserves the evidence body byte-for-byte and chooses a fence longer than any backtick
sequence in the bundle.
validity context. Missing validity artifacts are not presented as a clean verdict.
new-skillandrevisioncondition names are preserved. Mode B source and resolved-revisionprovenance reaches the report through the persisted evidence bundles.
behavior, promotion behavior, or finalized benchmarks.
Verification
cargo fmt --checkcargo buildcargo test --quiet(922 unit tests, 201 CLI tests, and 187 run tests)cargo clippy --all-targets -- -D warningsgit diff --check