Add an eval suite for the handoff skill - #4
Draft
joshwand wants to merge 2 commits into
Draft
Conversation
Not ready to merge. The linter is verified and the fixtures are written, but the rubric has never been run end-to-end against freshly generated handoffs, which is the part that would tell us whether it measures anything. Opening it separately so it can be finished without holding up the skill. The rewrite in the skill PR was justified by comparing two real handoffs, one useful and one not. That comparison was the whole method, and it was done by hand and thrown away. This keeps it. Prompts regress in a way code doesn't: an edit can leave a skill still producing something fluent and well organised that has quietly stopped being actionable. Nothing fails. You find out two sessions later when an agent starts on the wrong thing. Two layers, because there are two kinds of failure. Format regressions are mechanical, so lint_handoff.py decides them in stdlib Python — the seven sections present and ordered, prose rule held, constraints not empty, something paste-able actually named. Substance regressions need judgment, so rubric.md carries 21 assertions for a model to grade, weighted so that four load-bearing ones cap the score if failed. The calibration pair is the part worth keeping honest. good.md and bad.md are both written from case 01, and the grader scores them before it scores anything real: good must clear 18, bad must stay under 10. A grader that can't separate them is rewarding fluent prose, which is precisely the failure this exists to catch — bad.md is fluent, well organised, and useless, because it is what the skill produced before. Three cases: the hard one with real state and constraints, a planning-only session that tests whether required sections get padded when there's nothing to put in them, and one blocked on a human where two live options have to survive unchosen. Fixtures are synthetic. evals/local/ is gitignored for grading real handoffs without publishing project details. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The baseline was about to be lost. The old skill never lived in this repo — its only copy was ~/.claude/skills/handoff/SKILL.md, which has since been overwritten with the rewritten version. Without a control arm a rubric score on the new skill alone can't distinguish "this is good" from "this judge is generous", so it is recovered here from the session transcript, typos intact, with its provenance stated: reconstructed, not git-attested. handoff.md briefs the next agent on wiring Inspect AI to the three existing cases. Written with the handoff skill and linted with lint_handoff.py, which passes it 0 failed 0 warned — the first end-to-end run of either against something not written as a fixture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — stacked on #2, and not ready to merge. See "What's missing" below.
Retargets to
mainautomatically once #2 lands.Why
The rewrite in #2 was justified by comparing two real handoffs, one useful and one not. That comparison was the entire method, it was done by hand, and it was thrown away. This keeps it.
Prompts regress in a way code doesn't. An edit can leave a skill still producing something fluent and well organised that has quietly stopped being actionable — nothing fails, no test goes red, and you find out two sessions later when an agent starts on the wrong thing.
Two layers, no dependencies
Format regressions are mechanical, so a script decides them. Substance regressions need judgment, so a rubric and a model decide those.
lint_handoff.py— stdlib Python. Seven sections present and in order, prose rule held, no preamble, "Standing constraints" not empty, and something paste-able actually named — including inside "Read first" specifically, since that's the section most likely to rot into "read the design doc". FAIL exits non-zero; WARN flags recap phrasing and missing second-person address.rubric.md— 21 substance assertions, model-graded, with four load-bearing ones (first action executable, finished separated from half-finished, says where to stop, prohibitions carry their enforcement mechanism) that cap the score if failed.The calibration pair
calibration/good.mdandcalibration/bad.mdare both written from case 01, and get graded before anything real: good must clear 18, bad must stay under 10.A grader that can't separate them is rewarding fluent prose, which is precisely the failure being hunted —
bad.mdis fluent, well organised, and useless, because it's what the skill produced before #2. It's not a strawman: beyond being vague it describes a deliberatexfailas "currently failing", which invites the next agent to destroy the signal by fixing it.Cases
01-midstream-implementation02-planning-only03-blocked-on-humanEach has a
session.md(input) and anexpected.md(facts that must survive, and how the case is usually failed). Fixtures are synthetic;evals/local/is gitignored for grading real handoffs without publishing project details.What's verified
The linter, and only the linter. It passes
good.mdclean and produces 9 failures and 3 warnings onbad.md; markdown detection was smoke-tested against a real markdown file; usage and missing-file paths exit 2 and 1.Next step is briefed
handoff.mdon this branch is the briefing for wiring this suite up to Inspect AI (UK AISI) against the three cases. Inspect was chosen over promptfoo (OpenAI agreed to acquire it in 2026), DeepEval (RAG-shaped metrics, hosted pull), the hosted platforms (account dependency), and a hand-rolled runner drivingclaude -p(Claude Code's own system prompt is an uncontrolled variable above every generation). It's Python, needs no account, and has bootstrap CIs and pass/fail gates natively rather than hand-rolled.evals/baseline/SKILL.mdis the pre-rewrite skill, preserved as the control arm for a future A/B. It was recovered from a session transcript rather than version control — its only copy had been overwritten — soevals/baseline/README.mdstates that provenance plainly.What's missing — why this is a draft
The rubric has never been run end-to-end. No handoff has been generated from a case in a fresh chat and graded. Until that happens the 21 assertions are untested, the 18/10 calibration thresholds are guesses, and the whole substance layer is unproven. That needs a fresh chat per case, and grading a handoff in the session that wrote it tests nothing — the grader can see everything the handoff left out.
Open questions that only running it will answer: whether the thresholds are set anywhere near right, whether any assertion is unjudgeable in practice, and whether three cases is enough to distinguish a real gap from sampling noise.
🤖 Generated with Claude Code