Skip to content

feat(cli): add paired evidence comparison - #290

Merged
slowdini merged 1 commit into
devfrom
feat/compare-evidence
Aug 24, 2026
Merged

feat(cli): add paired evidence comparison#290
slowdini merged 1 commit into
devfrom
feat/compare-evidence

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Why

Eval authors often need to inspect what changed between two conditions before they know which
assertions are worth measuring. The bounded evidence already exists after ingest, but it is split
across condition and run directories. That makes open-ended exploration awkward and encourages
assertions to be written before seeing realistic behavior.

Closes #263.

Strategy

Add eval-magic compare as a post-ingest, pre-judging operation. It selects an iteration and eval
ID, then pairs the persisted judge-evidence.md bundles for both conditions. The report preserves
the same bounded prompt, final message, diff, changed-file list, conversation, tool-use, and
provenance evidence used by judges.

The generated report lives at iteration-N/compare/<eval-id>.md, and the command prints that path
so a driving agent can open it directly. Multi-run cells are paired and labelled by numeric run
index.

Generated runbooks include one exact compare command per selected eval after ingest. The judging
guide describes how to turn exploratory observations into concrete assertions and repeated or
sampled eval runs.

Example

Without a paired report, an operator has to locate and correlate each condition's evidence bundle
manually. The paired workflow is:

$ eval-magic compare --iteration 1 --eval implement-feature
Wrote /path/to/iteration-1/compare/implement-feature.md

The report places both conditions together for each matching run.

Safety and compatibility

  • The command works when an eval declares no assertions. It requires completed ingest evidence,
    but not judge dispatch or finalization.
  • Validation requires exactly two conditions, matching run slots, and a non-empty evidence bundle
    for every arm. A validation failure does not replace a prior complete report.
  • Embedded task, transcript, tool, and patch content is fenced as untrusted read-only evidence. The
    wrapper preserves the evidence body byte-for-byte and chooses a fence longer than any backtick
    sequence in the bundle.
  • Available permission-denial, guard-denial, stray-write, and skill-shadow artifacts are linked as
    validity context. Missing validity artifacts are not presented as a clean verdict.
  • Both new-skill and revision condition names are preserved. Mode B source and resolved-revision
    provenance reaches the report through the persisted evidence bundles.
  • The report is explicitly exploratory: it is neither a grade nor a statistically reliable result.
  • The command consumes existing iteration artifacts and does not change artifact schemas, snapshot
    behavior, promotion behavior, or finalized benchmarks.

Verification

  • cargo fmt --check
  • cargo build
  • cargo test --quiet (922 unit tests, 201 CLI tests, and 187 run tests)
  • cargo clippy --all-targets -- -D warnings
  • git diff --check

@slowdini
slowdini merged commit 5f2047b into dev Aug 24, 2026
7 checks passed
@slowdini
slowdini deleted the feat/compare-evidence branch August 24, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compare: paired evidence for interactive judging

1 participant