feat(judging): add multi-sample pass^k grading - #289
Merged
Conversation
Reuse each run's bounded evidence across independent judge verdicts and retain vote proportions plus pass^k without changing the one-sample artifact contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Repeated LLM verdicts over one execution expose judge consistency without paying to rerun the agent. This adds assertion-level and campaign-wide judge sampling, then carries the individual verdicts, vote proportion, and pass^k endpoint through grading and benchmark aggregation.
The implementation builds on the bounded judge evidence bundle from #288: every sample for a run receives the exact same persisted
judge-evidence.md, so sampling varies the judgment rather than the evidence under review.Closes #262.
Part of #244.
Strategy
llm_judge.samples, thenrun --judge-samples, then the compatibility default of one.__sample-Njudge tasks while reusing one bounded evidence bundle. The framework-owned__skill_invokedmeta-check remains single-shot.passed,failed,total, vote proportionp, andpass_power_k = p^N. Mixed gradings weight each authored assertion equally: binary assertions contribute zero or one, and sampled assertions contribute their derived endpoint.An eval can opt into sampling directly:
{ "id": "clear-review", "type": "llm_judge", "rubric": "The review identifies the most important defect and explains its impact.", "samples": 10 }Operators can instead set a campaign default:
The assertion value takes precedence over the command-line default. With six passing verdicts out of 10, the retained vote proportion is
0.6and pass^k is0.6^10, approximately0.006047.Compatibility and safety boundaries
passed/nrollup.quality__sample-1colliding with a generated sample forquality.Verification
cargo fmt --checkcargo buildcargo test— 1,300 tests passedcargo clippy --all-targets -- -D warningsgit diff --checkReview focus
The main review surface is the compatibility split between one-sample binary artifacts and multi-sample artifacts, plus the equal-assertion weighting used when a run mixes binary and sampled assertions.