From ba7da9bed0a24bd8a19831ecca2fdac01e496831 Mon Sep 17 00:00:00 2001 From: joshwand Date: Sun, 2 Aug 2026 18:36:57 -0700 Subject: [PATCH 1/2] Add an eval suite for the handoff skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Not ready to merge. The linter is verified and the fixtures are written, but the rubric has never been run end-to-end against freshly generated handoffs, which is the part that would tell us whether it measures anything. Opening it separately so it can be finished without holding up the skill. The rewrite in the skill PR was justified by comparing two real handoffs, one useful and one not. That comparison was the whole method, and it was done by hand and thrown away. This keeps it. Prompts regress in a way code doesn't: an edit can leave a skill still producing something fluent and well organised that has quietly stopped being actionable. Nothing fails. You find out two sessions later when an agent starts on the wrong thing. Two layers, because there are two kinds of failure. Format regressions are mechanical, so lint_handoff.py decides them in stdlib Python — the seven sections present and ordered, prose rule held, constraints not empty, something paste-able actually named. Substance regressions need judgment, so rubric.md carries 21 assertions for a model to grade, weighted so that four load-bearing ones cap the score if failed. The calibration pair is the part worth keeping honest. good.md and bad.md are both written from case 01, and the grader scores them before it scores anything real: good must clear 18, bad must stay under 10. A grader that can't separate them is rewarding fluent prose, which is precisely the failure this exists to catch — bad.md is fluent, well organised, and useless, because it is what the skill produced before. Three cases: the hard one with real state and constraints, a planning-only session that tests whether required sections get padded when there's nothing to put in them, and one blocked on a human where two live options have to survive unchosen. Fixtures are synthetic. evals/local/ is gitignored for grading real handoffs without publishing project details. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 5 +- README.md | 6 + evals/README.md | 84 +++++++++ evals/calibration/README.md | 38 ++++ evals/calibration/bad.md | 17 ++ evals/calibration/good.md | 30 +++ .../01-midstream-implementation/expected.md | 55 ++++++ .../01-midstream-implementation/session.md | 62 ++++++ evals/cases/02-planning-only/expected.md | 47 +++++ evals/cases/02-planning-only/session.md | 40 ++++ evals/cases/03-blocked-on-human/expected.md | 49 +++++ evals/cases/03-blocked-on-human/session.md | 53 ++++++ evals/lint_handoff.py | 176 ++++++++++++++++++ evals/rubric.md | 100 ++++++++++ 14 files changed, 761 insertions(+), 1 deletion(-) create mode 100644 evals/README.md create mode 100644 evals/calibration/README.md create mode 100644 evals/calibration/bad.md create mode 100644 evals/calibration/good.md create mode 100644 evals/cases/01-midstream-implementation/expected.md create mode 100644 evals/cases/01-midstream-implementation/session.md create mode 100644 evals/cases/02-planning-only/expected.md create mode 100644 evals/cases/02-planning-only/session.md create mode 100644 evals/cases/03-blocked-on-human/expected.md create mode 100644 evals/cases/03-blocked-on-human/session.md create mode 100644 evals/lint_handoff.py create mode 100644 evals/rubric.md diff --git a/.gitignore b/.gitignore index 4d91d01..0e5d36c 100644 --- a/.gitignore +++ b/.gitignore @@ -6,4 +6,7 @@ repomix-output.md repomix-output.xml .cursorindexingignore -.venv \ No newline at end of file +.venv +# real handoffs to grade against the rubric, kept out of the public repo +evals/local/ +__pycache__/ \ No newline at end of file diff --git a/README.md b/README.md index 98738b4..f2da2f8 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,12 @@ The blocks that are large but only situationally needed, kept out of the always- - **`interactive-planning`** — the one-question-at-a-time elicitation prompt behind `.ip`. - **`handoff`** — the briefing behind `.cn`, for moving work into a fresh chat. Its output is written to the agent picking the work up rather than as a recap for you, so the sections are operative: what to read and in what order, the state inherited, the scope and where to stop, the standing constraints, and the definition of done. +### `evals/` + +A small suite for the `handoff` skill, because prompts regress quietly — a skill can be edited into something that still produces a plausible document while dropping the parts that made it useful. + +`lint_handoff.py` is stdlib-only and decides the mechanical rules: sections present and in order, plain prose held, something paste-able actually named. `rubric.md` holds 21 substance assertions for a model to grade, over three synthetic sessions in `cases/`. `calibration/` is a known-good and known-bad pair used to check the grader before trusting it on anything real — a grader that can't separate those two is rewarding fluent prose, which is the failure being hunted. + ## Structured memory Credit where due: the memory bank is an adaptation of the [Cline Memory Bank](https://docs.cline.bot/improving-your-prompting-skills/cline-memory-bank). diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..b18099c --- /dev/null +++ b/evals/README.md @@ -0,0 +1,84 @@ +# Evals + +A small suite for the `handoff` skill. Prompts regress silently — a skill can get +edited into something that still produces a plausible document while quietly +dropping the parts that made it useful. This catches that. + +Two layers, because the failures come in two kinds. Format regressions are cheap +and mechanical, so a script decides them. Substance regressions need judgment, so +a rubric and a model decide those. No dependencies either way: stdlib Python and +markdown. + +``` +lint_handoff.py deterministic structural checks +rubric.md 21 substance assertions, model-graded +cases/ synthetic sessions to generate handoffs from +calibration/ a known-good and known-bad pair, to test the grader +local/ gitignored; your real handoffs +``` + +## The linter + +``` +python3 evals/lint_handoff.py handoff.md +``` + +Checks the seven sections are present and in order, the plain-prose rule holds, +nothing precedes the first section, "Standing constraints" is not empty, and +something paste-able (a path, a command, a SHA) actually appears — including +inside "Read first" specifically. + +FAIL exits non-zero. WARN never fails the run; warnings flag recap phrasing, +missing second-person address, and documents thin enough to be worth re-reading. + +Verify it works on the pair: + +``` +python3 evals/lint_handoff.py evals/calibration/good.md # PASS, 0 failed, 0 warned +python3 evals/lint_handoff.py evals/calibration/bad.md # FAIL, 9 failed, 3 warned +``` + +## The full eval + +The linter cannot tell a specific handoff from a vague one, which is the failure +that matters. For that: + +1. **Calibrate the grader.** Score `calibration/good.md` and `calibration/bad.md` + against `rubric.md`. Good must land at 18+, bad at 10 or below. If they land + close together, stop — see `calibration/README.md`. +2. **Generate.** In a fresh chat with the skill available, paste a case's + `session.md` as context and ask for a handoff (`.cn`). A fresh chat matters: + grading a handoff in the session that wrote it tests nothing, because the + grader can see everything the handoff left out. +3. **Lint** the result. +4. **Grade** it against `rubric.md` plus the case's `expected.md`. Report failed + assertion numbers, not just a total. + +Three or four handoffs per case is more informative than one — the interesting +question is which assertions fail *repeatedly*, since a single miss is sampling +noise and a pattern is a gap in the skill. + +## The cases + +| Case | Shape | Stresses | +|---|---|---| +| `01-midstream-implementation` | Real state, a blocked carryover, protected paths, a human-only step, an open decision | Everything; this is the hard one | +| `02-planning-only` | Nothing built, no constraints, no blockers | Graceful degradation — does the format pad or stay honest | +| `03-blocked-on-human` | Stopped on a human action, two live options | Whether open decisions survive unchosen | + +Each case has a `session.md` (the input) and an `expected.md` (the facts that must +survive, and how the case is usually failed). + +## Adding a case + +New cases should come from a session that produced a *bad* handoff. That is the +one reliable source of eval material: a fixture invented to be tricky tests what +you imagined, while one derived from a real failure tests what actually goes +wrong. Synthesize the shape, drop the project specifics, and write the +`expected.md` from what the real handoff lost. + +## local/ + +`evals/local/` is gitignored. Drop real handoffs there and grade them against the +same rubric without publishing project details. Nothing in the suite depends on +it being populated. diff --git a/evals/calibration/README.md b/evals/calibration/README.md new file mode 100644 index 0000000..ba80336 --- /dev/null +++ b/evals/calibration/README.md @@ -0,0 +1,38 @@ +# Calibration pair + +Two handoffs written from `../cases/01-midstream-implementation/`. They exist to +test the grader, not the skill. + +Score these before scoring anything real. A grader that cannot separate them is +not measuring anything, and its verdict on a fresh handoff is noise. + +| File | Expected rubric score | Expected linter result | +|---|---|---| +| `good.md` | 18 or above, all four load-bearing assertions passed | PASS, no warnings | +| `bad.md` | 10 or below, load-bearing assertions failed | FAIL — 9 failures (seven missing sections, a preamble, nothing concrete anywhere) and 3 warnings | + +If `good.md` scores below 18, the grader is too harsh, or the rubric has an +assertion the skill never promised. If `bad.md` scores above 10, the grader is +rewarding fluent prose — which is the exact failure this whole eval exists to +catch, since `bad.md` is fluent, well organised, and nearly useless. + +## What bad.md gets wrong + +It is not a strawman. It is the shape the skill produced before the rewrite, and +it reads fine until you try to act on it. + +Every specific has been sanded off: no SHA, no test count, no file paths, no test +node id, `rapidfuzz` never named, the design doc referred to as "the design doc" +rather than by path. The half-built tie breaker — the thing that blocks all the +work — appears as "some unfinished work in the candidate selection code", so the +next agent starts on the engine and discovers the stub the hard way. + +Two errors are worse than vagueness. The deliberate xfail is described as "one of +the fuzzy name tests is currently failing", which invites the next agent to fix +it and thereby destroy the signal. And "pick up the next step of the plan" omits +that step 4 and only step 4 was authorized, so step 5 is a live risk. + +Everything in the constraints category is simply gone: the protected file and its +hook, the container build failure behind the dropped dependency, the human-only +credential rotation, the instruction not to commit. None of it is contradicted — +it is just absent, which reads exactly like there being none. diff --git a/evals/calibration/bad.md b/evals/calibration/bad.md new file mode 100644 index 0000000..722a7a5 --- /dev/null +++ b/evals/calibration/bad.md @@ -0,0 +1,17 @@ +Who this is for: +This is for Josh, who is working on tessera, a Python CLI for reconciling invoices against a ledger. This session continued a multi-step rewrite of the reconciler. + +What we covered: +We worked through the early steps of the reconciler rewrite plan and got them committed. Along the way we dealt with a dependency that turned out not to work for us and had to be swapped for something in the standard library. We also looked at the matching code and at how fuzzy name comparison should handle accented characters, which is a question that is still open. The test suite is in good shape throughout. + +What was confirmed: +The first several steps of the rewrite are done and committed, and the tests pass. The matcher will use the standard library rather than the third-party option that was tried. Josh approved moving on to the next step of the plan. + +Still in progress: +The matching engine has not been built yet. There is some unfinished work in the candidate selection code. One of the fuzzy name tests is currently failing. The normalization question remains unresolved. + +Next steps: +Pick up the next step of the plan and build the matching engine. Read the design document first for the details of how it should work. It would also be worth resolving the normalization question at some point, since it affects the failing test. + +References to load: +Read the design doc in the memory knowledge base, and the current task state file. diff --git a/evals/calibration/good.md b/evals/calibration/good.md new file mode 100644 index 0000000..592f25b --- /dev/null +++ b/evals/calibration/good.md @@ -0,0 +1,30 @@ +What you are picking up: +You are continuing a six step rewrite of the reconciler in tessera, a Python CLI that reconciles invoices against a ledger. Steps 1 through 3 are done and committed at 4c1f9ab. Josh has authorized step 4, and only step 4. + +Read first: +Read _memory/knowledgeBase/designs/ReconcilerRewrite.md before anything else. It is the plan, step 4 is your scope within it, and it is authoritative over this briefing wherever the two disagree. Then read _memory/currentState/currentTaskState.md for where things stood at the end of the last session. Then skim tessera/match/candidates.py and tessera/match/engine.py, in that order, because the first one blocks the second. + +State you inherit: +Steps 1 through 3 are complete and committed together at 4c1f9ab. The suite is 214 passing, 0 skipped, run with .venv/bin/pytest -q. + +tessera/match/candidates.py is half built, and it blocks everything else. The scoring function is written and tested; the tie breaker below it is a TODO stub that returns the first candidate. Step 4 consumes that function, so finishing the tie breaker comes before any work on the engine. + +tests/test_fuzzy.py::test_unicode_names is xfail deliberately, pending the normalization decision below. It is not a regression, and it must not be made to pass by changing the assertion. + +Your scope: +Finish the tie breaker in tessera/match/candidates.py first. Then build the matching engine at tessera/match/engine.py as step 4 of the plan describes, with tests for both. + +Do not start step 5, and do not commit; Josh reviews before anything lands. Keep _memory/currentState/currentTaskState.md current as you go. If step 4 turns out to need a change in a protected file, stop and propose it rather than working around it, as described below. + +Standing constraints: +tessera/config/rates.py is owner edited. Write is permission denied on it, and a pre-commit hook rejects any commit that touches it. When step 2 needed a change there, the change went into tessera/config/rates_proposed.py alongside it and Josh applied it by hand. Do the same rather than trying to edit the original. + +Do not reintroduce rapidfuzz. Step 2 used it for the matcher and it came out again because it pulls a C extension that breaks the Alpine container build. That failure does not surface until the image builds in CI, so it looks harmless locally. Stdlib difflib is the decision. + +The production reconciliation run needs a credential rotated in the bank's web console. There is no API for it, and you have neither a browser nor the credentials. If step 4 gets far enough to want a real run, prepare everything up to the run, write the exact steps Josh needs to click into _memory/currentState/currentTaskState.md, and stop there. + +Open questions: +Unicode name normalization is undecided, and it is Josh's call rather than yours. NFKC is more correct and keeps names readable in the reconciliation output. Stripping diacritics before comparing matches more aggressively, which is closer to what reconciliation actually wants, but produces output a human cannot check by eye. This decides tests/test_fuzzy.py::test_unicode_names, so leave that test xfail until he chooses. + +Definition of done: +The tie breaker in tessera/match/candidates.py is finished and tested. Step 4 is implemented as the plan describes, with tests. The full suite is green, with the unicode test still xfail. _memory/currentState/currentTaskState.md is updated well enough that a fresh agent could take step 5 from it alone. Your final report to Josh covers what was built, the test count, any deviations from the plan, and anything needing his decision. Then stop. diff --git a/evals/cases/01-midstream-implementation/expected.md b/evals/cases/01-midstream-implementation/expected.md new file mode 100644 index 0000000..fefc587 --- /dev/null +++ b/evals/cases/01-midstream-implementation/expected.md @@ -0,0 +1,55 @@ +# Case 01 — what the handoff must carry + +Case level assertions, checked alongside `../../rubric.md`. Each fact below either +survives into the handoff or the handoff loses something the next agent needs. + +## Must appear, verbatim where it is an identifier + +- Steps 1 through 3 done, committed at `4c1f9ab`. +- 214 passing, 0 skipped, and the command `.venv/bin/pytest -q`. +- `_memory/knowledgeBase/designs/ReconcilerRewrite.md`, named as authoritative + over the handoff itself. +- Step 4 is the scope, and step 4 is `tessera/match/engine.py`. +- `tessera/match/candidates.py` is half built, the tie breaker is a stub, and it + blocks step 4 — so it comes first. +- `tests/test_fuzzy.py::test_unicode_names` is xfail on purpose and must not be + "fixed" by changing the assertion. +- `rapidfuzz` was tried and dropped over the Alpine C extension build failure; + stdlib `difflib` is the decision. +- `tessera/config/rates.py` is owner edited, with the enforcement named (Write + denied, pre-commit hook) and the `rates_proposed.py` workaround. +- The credential rotation is human only, with the instruction to prepare up to + the run, write the click steps into + `_memory/currentState/currentTaskState.md`, and stop. +- Do not commit. Do not start step 5. + +## Must be presented as open, not resolved + +The NFKC versus strip-diacritics decision, with both sides as they stood, marked +as Josh's to make. A handoff that picks one has failed assertion 16, however +sensible the pick. + +## What this case is designed to stress + +| Assertion | Why this case tests it | +|---|---| +| 1, 2 | The first action is the `candidates.py` tie breaker, not step 4 — a handoff that opens with "implement the matching engine" has buried the blocker | +| 8 | Three finished steps and one half-finished file, which have to read differently | +| 9 | A SHA, a test count, and a test node id are all available to lose | +| 10 | The xfail is unfinished on purpose; silence makes it look like a bug to fix | +| 11, 12 | "Step 4 only, do not commit" is the boundary; the credential rotation is the escalation | +| 13 | Two constraints with real mechanisms — a permission denial plus a hook, and a container build that fails only in CI | +| 14 | `rapidfuzz` is the dead end, and it is expensive to rediscover | +| 15 | The credential rotation cannot be done by any agent | +| 16, 17 | The normalization decision is Josh's and must survive unresolved | +| 18, 19 | The bar for done is stated and checkable, including what the report covers | + +## Common ways this case is failed + +The handoff leads with step 4 and mentions `candidates.py` later as a detail, so +the next agent starts on the engine and hits the stub. The `rapidfuzz` dead end +is dropped as "history", and gets reintroduced. The protected file is described +as "avoid editing `rates.py`" with no mechanism, so the agent tries anyway and +burns a turn on a permission denial it could have planned around. The +normalization question is silently resolved in favour of NFKC because it is the +more defensible answer. diff --git a/evals/cases/01-midstream-implementation/session.md b/evals/cases/01-midstream-implementation/session.md new file mode 100644 index 0000000..cf36561 --- /dev/null +++ b/evals/cases/01-midstream-implementation/session.md @@ -0,0 +1,62 @@ +# Case 01 — midstream implementation + +A synthetic session record. Paste this into a fresh chat as the context, then ask +for a handoff (`.cn`). The document that comes back is what gets graded. + +This is the hard case: real state, real constraints, a blocked carryover, a human +only step, and an open decision. A handoff that carries all of it is doing the job. + +--- + +Project `tessera`, a Python CLI that reconciles invoices against a ledger. We are +working through a six step rewrite plan. + +**What happened this session.** + +Finished steps 1 through 3 of the plan in +`_memory/knowledgeBase/designs/ReconcilerRewrite.md`. That document is +authoritative over anything I summarise here — where they disagree, believe it. +Committed the three steps together at `4c1f9ab`. Suite is 214 passing, 0 skipped, +run with `.venv/bin/pytest -q`. + +Step 4 is the matching engine, `tessera/match/engine.py`. Josh authorized step 4 +and only step 4 this session. + +**The thing that blocks step 4.** `tessera/match/candidates.py` is half built. The +scoring function is written and tested; the tie breaker below it is a `TODO` +stub that returns the first candidate. Step 4 consumes that function, so it has +to be finished before the engine can be built on top of it. + +**Deliberately failing.** `tests/test_fuzzy.py::test_unicode_names` is marked +xfail. It stays xfail until the normalization decision below is made — it is not +a regression and should not be "fixed" by changing the assertion. + +**Already tried and dropped.** Step 2 used `rapidfuzz` for the matcher. It pulls +a C extension that breaks the Alpine container build, so it came out again and +the decision is stdlib `difflib`. Do not reintroduce it; the build failure is +not obvious until the image is built in CI. + +**Protected.** `tessera/config/rates.py` is owner edited. Write is permission +denied and a pre-commit hook rejects any commit touching it. When step 2 needed a +change there, we wrote `tessera/config/rates_proposed.py` alongside it and Josh +applied it by hand. Do the same if step 4 needs one. + +**Needs Josh, cannot be done by an agent.** The production reconciliation run +needs a credential rotated in the bank's web console. There is no API for it and +the agent has no browser or credentials. If step 4 gets far enough to want a real +run: prepare everything up to the point of the run, write the exact steps Josh +needs to click into `_memory/currentState/currentTaskState.md`, and stop there. + +**Open, not decided.** Unicode name normalization: NFKC, or strip diacritics +before comparing. NFKC is more correct and keeps names readable in the output; +stripping matches more aggressively, which is what the reconciliation actually +wants but produces output a human cannot verify by eye. Josh has not chosen. This +decides the xfail test above. + +**Standing instructions.** Do not commit — Josh reviews before anything lands. Do +not start step 5. Keep `_memory/currentState/currentTaskState.md` current. + +**Bar for done.** Step 4 implemented per the plan with tests, `candidates.py` +tie breaker finished, full suite green, task state updated so a fresh agent could +take step 5, and a final report covering what was built, the test count, any +deviations from the plan, and anything needing Josh's decision. diff --git a/evals/cases/02-planning-only/expected.md b/evals/cases/02-planning-only/expected.md new file mode 100644 index 0000000..3fe2679 --- /dev/null +++ b/evals/cases/02-planning-only/expected.md @@ -0,0 +1,47 @@ +# Case 02 — what the handoff must carry + +Case level assertions, checked alongside `../../rubric.md`. + +## Must appear + +- The tiering is agreed and Josh approved it. +- `_memory/basicTruths/theBacklog.md` holds the tiers and is where to look. +- `_memory/currentState/currentEpic.md` for the offline tile cache epic. +- The three tier 1 items, named: blank grid while tiles fetch, over-aggressive + retry on failed fetch, cache size setting not surviving restart. +- No code written, nothing committed, no tests run — stated plainly, not implied + by omission. +- Nobody has checked whether the three tier 1 items are one change or three. + This is the first real question the next agent hits. + +## Must be presented as open + +Whether tier 3 (route planner) gets built at all. Josh raised cutting it and did +not settle it. A handoff that lists it as upcoming work has converted an open +question into a commitment, which fails assertion 16. + +## The point of this case + +Four of the seven sections have thin material, and this is where a format with +required sections goes wrong. Grade hard on assertion 20 (no filler). + +- **Standing constraints** must be one honest sentence saying none came up beyond + `AGENTS.md`. Inventing plausible constraints fails assertion 5. Dropping the + section fails the linter. Padding it fails 20. +- **State you inherit** must say the absence out loud. "No code was written" is + load-bearing here: the next agent should not go looking for a branch. +- **Definition of done** has to be invented for the next stretch rather than + recalled, because the session never set one. Scoping it to the tier 1 pass is + right; a criterion the session cannot support is assertion 5 again. + +A good handoff for this session is short. Under the old 200 to 400 word rule +this case scored fine and case 01 could not — which is roughly the argument for +dropping the cap. + +## Common ways this case is failed + +Standing constraints gets filled with generic advice about being careful with the +cache. The handoff reads as a summary of the tiering conversation rather than an +instruction to go do tier 1. Tier 3 appears in "Your scope" as later work. The +open question about whether tier 1 is one change or three — the actual first +thing to determine — is missing entirely. diff --git a/evals/cases/02-planning-only/session.md b/evals/cases/02-planning-only/session.md new file mode 100644 index 0000000..b8b46c6 --- /dev/null +++ b/evals/cases/02-planning-only/session.md @@ -0,0 +1,40 @@ +# Case 02 — planning only, no code state + +A synthetic session record. Paste this into a fresh chat as the context, then ask +for a handoff (`.cn`). + +This is the degradation case. Almost nothing happened that a handoff usually +carries: no commits, no tests, no constraints, no blockers. The format has to +stay honest under that instead of padding seven sections out to look complete. + +--- + +Project `atlas`, a personal mapping side project. + +**What happened this session.** Josh came back from a weekend of using the build +on his phone with nine notes. I added all nine to the triage section of +`_memory/basicTruths/theBacklog.md`, then reorganized that section into priority +tiers with his approval. + +The tiering was grounded in the current epic state in +`_memory/currentState/currentEpic.md`: the offline tile cache is the open epic, +and six of the nine notes are about behaviour when the cache is cold, which is +what the epic's kill criterion turns on. + +**The agreed tiers.** Tier 1 is cold cache correctness as one pass: the map +should not render a blank grid while tiles are fetching, the retry on a failed +tile fetch is far too aggressive, and the cache size setting in the UI does not +survive a restart. Tier 2 is display polish, deliberately not specified further +this session. Tier 3 is the route planner, which Josh said out loud he may cut +entirely rather than build. + +**No code was written.** Nothing was committed. No tests were run. Tier 1 has not +been started, and nobody has looked at whether the three tier 1 items are one +change or three. + +**Nothing was ruled out** and no constraint came up. Normal project rules apply +and they are in `AGENTS.md`; there is nothing special about this work. + +**Open.** Whether tier 3 gets built at all is genuinely undecided — Josh raised +cutting it and did not settle it. Not urgent, but it should not quietly become a +commitment just because it is written in a backlog file. diff --git a/evals/cases/03-blocked-on-human/expected.md b/evals/cases/03-blocked-on-human/expected.md new file mode 100644 index 0000000..0a8df8f --- /dev/null +++ b/evals/cases/03-blocked-on-human/expected.md @@ -0,0 +1,49 @@ +# Case 03 — what the handoff must carry + +Case level assertions, checked alongside `../../rubric.md`. + +## Must appear + +- `postbox/http/receiver.py` is built and tested, 47 passing via `pytest -q`. +- Branch `webhook-receiver`, uncommitted, working tree dirty. Committing on that + branch is approved — this is the one case where committing is allowed, and a + handoff that carries the usual "do not commit" would be wrong. +- The blocker: no TLS cert for `staging.postbox.internal`, because the ACME DNS + TXT record is not in. The record is in `deploy/acme-challenge.txt`. +- Josh has the registrar login and the agent has no API token — the blocker is + human only, with propagation time after he adds it. +- Do not retry the deploy until he confirms. Two ACME failures already against a + limit of five per hour. The mechanism is what makes this a constraint rather + than a preference. +- `.env.staging` is gitignored and stays out of commits, logs, and reports; refer + to key names, not values. + +## Must be presented as open + +Both paths, with the real trade-off intact: the tunnel tests signature +verification but costs an hour of throwaway setup; local compose is fast but +leaves signature verification untested. Josh's thinking-out-loud both ways is +context, not a decision. + +A handoff that presents either as "the plan" fails assertion 16 — and this is the +most tempting case in the suite to fail, because option two is defensible and +picking it makes the handoff read as more decisive. + +## What this case is designed to stress + +| Assertion | Why this case tests it | +|---|---| +| 12, 15, 17 | The whole session is blocked on a human; the escalation is the main content | +| 13 | The ACME rate limit is a constraint with a real counter behind it, and the secret handling has an enforcement (gitignore) plus a discipline (no echoing) | +| 16 | Two live options that must survive unchosen | +| 11 | The boundary is unusual — committing is permitted here, retrying the deploy is not | +| 3, 4 | `deploy/acme-challenge.txt` is the first thing to read and the least guessable | + +## Common ways this case is failed + +The handoff picks option two, or presents it first with the other as an +afterthought, and the trade-off collapses. The ACME rate limit becomes "be +careful not to redeploy" with no number, so the next agent retries once "just to +see the error". The commit permission is flattened into the usual prohibition, so +finished work sits uncommitted for another session. `.env.staging` is described +well enough that its contents end up quoted in the final report. diff --git a/evals/cases/03-blocked-on-human/session.md b/evals/cases/03-blocked-on-human/session.md new file mode 100644 index 0000000..c4135c4 --- /dev/null +++ b/evals/cases/03-blocked-on-human/session.md @@ -0,0 +1,53 @@ +# Case 03 — blocked on a human action + +A synthetic session record. Paste this into a fresh chat as the context, then ask +for a handoff (`.cn`). + +The work is stopped on something no agent can do, and there are two viable ways +to spend the time until it clears. This case is about whether the handoff hands +the choice over intact or quietly makes it. + +--- + +Project `postbox`, a small notification service. + +**What happened this session.** Built the inbound webhook receiver at +`postbox/http/receiver.py` with tests; 47 passing, run with `pytest -q`. Not +committed — the branch is `webhook-receiver`, working tree dirty. + +**Where it stopped.** The staging deploy fails. The TLS certificate for +`staging.postbox.internal` has never been issued, because the ACME challenge +needs a DNS TXT record added at the registrar. Josh has the registrar login; the +agent does not, and there is no API token for it. Nothing can be verified against +staging until that record exists and the cert issues, which takes a few minutes +to propagate after he adds it. + +The exact record is in `deploy/acme-challenge.txt`, written out this session. + +**Two ways forward while it is blocked, both reasonable, neither chosen.** + +One: run the receiver behind a local tunnel and point a real provider's webhook +at it. Tests the real payloads and the real signature verification, which is +where bugs actually are. Costs an hour of setup and the tunnel URL changes every +restart, so it is throwaway work. + +Two: skip staging and test against `docker-compose.local.yml`, which already +works. Faster, and it exercises the routing, but it uses synthetic payloads, so +signature verification stays untested until staging is up. + +Josh was thinking out loud about which and did not decide. He leaned toward two +because of the hour, then said the signature path is the risky part. It is his +call, not the next agent's. + +**Secrets.** `.env.staging` holds the signing secret. It is gitignored and stays +that way — no committing it, no echoing it into logs or into a report, no copying +values out of it into any file that is tracked. If a value from it is needed to +explain something, refer to the key name. + +**Do not** attempt the deploy again until Josh confirms the DNS record is in and +the cert issued. Each failed attempt rate limits the ACME account, and we are two +failures into a limit of five per hour. + +**Bar for done.** Whichever path Josh picks, verified with the receiver handling a +real or synthetic payload end to end, tests still green, and the work committed +on `webhook-receiver` — he has approved committing on that branch. diff --git a/evals/lint_handoff.py b/evals/lint_handoff.py new file mode 100644 index 0000000..17d8f28 --- /dev/null +++ b/evals/lint_handoff.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +"""Structural checks for a handoff document produced by the `handoff` skill. + +Checks only what can be decided mechanically: the sections and their order, the +plain-prose rule, and a few cheap proxies for specificity. Substance is graded +against evals/rubric.md by a model. This catches the regressions that don't need +one, and it catches them in a tenth of a second. + +Usage: + python3 evals/lint_handoff.py handoff.md [more.md ...] + +Exit status is 1 if any check FAILs, 0 otherwise. WARNs never fail the run; they +mark things worth a human look. +""" + +import re +import sys + +SECTIONS = [ + "What you are picking up:", + "Read first:", + "State you inherit:", + "Your scope:", + "Standing constraints:", + "Open questions:", + "Definition of done:", +] + +# A handoff that names nothing the next agent can paste has failed at its job, +# so at least one of these has to appear somewhere in the document. +CONCRETE = [ + re.compile(r"[\w./-]+\.(?:md|py|js|jsx|ts|tsx|json|ya?ml|toml|sh|go|rs|rb|sql|txt)\b"), + re.compile(r"\b[0-9a-f]{7,40}\b"), # commit SHA + re.compile(r"(?:^|\s)[\w-]+/[\w./-]+"), # a path with a directory in it +] + +# The old format's fingerprints. Any of these means it drifted back into writing +# a recap for the user instead of a briefing for the next agent. +RECAP_PHRASES = [ + "this is for", + "what we covered", + "in this session we", + "during this session", + "the user asked", + "we discussed", +] + +MARKDOWN = [ + (re.compile(r"^\s{0,3}#{1,6}\s"), "markdown header"), + (re.compile(r"^\s*[-*+]\s+\S"), "bullet marker"), + (re.compile(r"^\s*\d+[.)]\s+\S"), "numbered list item"), + (re.compile(r"\*\*[^*\n]+\*\*"), "bold markup"), + (re.compile(r"^\s*\|.*\|"), "table row"), +] + +MIN_WORDS = 250 + + +def find_sections(lines): + """Map each section label to the line it starts on, or None if absent.""" + found = {} + for label in SECTIONS: + for i, line in enumerate(lines): + if line.strip() == label or line.strip().startswith(label): + found.setdefault(label, i) + return found + + +def section_body(lines, found, label): + start = found.get(label) + if start is None: + return "" + later = [i for lbl, i in found.items() if i > start] + end = min(later) if later else len(lines) + return "\n".join(lines[start + 1 : end]).strip() + + +def check(text): + """Yield (level, line_no_or_None, message). line_no is 1-indexed.""" + lines = text.splitlines() + found = find_sections(lines) + + missing = [s for s in SECTIONS if s not in found] + for label in missing: + yield "FAIL", None, f"section missing: {label!r}" + + present = [(found[s], s) for s in SECTIONS if s in found] + order = [s for _, s in sorted(present)] + expected_order = [s for s in SECTIONS if s in found] + if order != expected_order: + yield "FAIL", None, f"sections out of order: got {' -> '.join(order)}" + + # No preamble: the document opens on the first section. + for i, line in enumerate(lines): + if line.strip(): + if not line.strip().startswith(SECTIONS[0]): + yield "FAIL", i + 1, f"preamble before the first section: {line.strip()[:60]!r}" + break + + for i, line in enumerate(lines): + if line.strip() in SECTIONS: + continue + for pattern, what in MARKDOWN: + if pattern.search(line): + yield "FAIL", i + 1, f"{what} — output must be plain prose" + break + + # "Standing constraints" is required, and required means it says something. + # One honest sentence that there are none is a pass; an empty section is not. + if "Standing constraints:" in found: + body = section_body(lines, found, "Standing constraints:") + if len(body) < 40: + yield "FAIL", found["Standing constraints:"] + 1, ( + "'Standing constraints:' is empty or near-empty; if the session " + "produced none, say so in a sentence" + ) + + if not any(p.search(text) for p in CONCRETE): + yield "FAIL", None, ( + "no file path, command, or SHA anywhere; the next agent has nothing to open" + ) + + if "Read first:" in found: + body = section_body(lines, found, "Read first:") + if not any(p.search(body) for p in CONCRETE): + yield "FAIL", found["Read first:"] + 1, "'Read first:' names nothing to read" + + lowered = text.lower() + for phrase in RECAP_PHRASES: + if phrase in lowered: + yield "WARN", None, ( + f"recap phrasing {phrase!r} — write to the next agent, not about the session" + ) + + words = len(text.split()) + if words < MIN_WORDS: + yield "WARN", None, f"{words} words; thin enough to be worth re-reading for omissions" + + if " you " not in lowered and not lowered.startswith("you "): + yield "WARN", None, "no second-person address; a handoff is written to whoever picks it up" + + +def main(argv): + paths = argv[1:] + if not paths: + print(__doc__.strip().splitlines()[0]) + print("\nusage: python3 evals/lint_handoff.py handoff.md [more.md ...]") + return 2 + + failed = False + for path in paths: + try: + with open(path, encoding="utf-8") as fh: + text = fh.read() + except OSError as exc: + print(f"{path}: cannot read: {exc}") + failed = True + continue + + results = list(check(text)) + fails = [r for r in results if r[0] == "FAIL"] + warns = [r for r in results if r[0] == "WARN"] + + status = "FAIL" if fails else "PASS" + print(f"{status} {path} ({len(fails)} failed, {len(warns)} warned)") + for level, line_no, message in results: + where = f"line {line_no}: " if line_no else "" + print(f" {level}: {where}{message}") + if fails: + failed = True + + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/evals/rubric.md b/evals/rubric.md new file mode 100644 index 0000000..3084605 --- /dev/null +++ b/evals/rubric.md @@ -0,0 +1,100 @@ +# Handoff rubric + +Twenty-one assertions about a handoff document. Each is worth one point and is +answered yes or no — if it needs a paragraph of hedging, the answer is no. + +Grade against the session the handoff was written from. Nearly every assertion +is a claim about whether the handoff carries something the session contained, +and you cannot check that without both halves. + +## Calibrate before you grade + +Score `calibration/bad.md` and `calibration/good.md` first, both written from +`cases/01-midstream-implementation/`. A working grader puts good at 18 or above +and bad at 10 or below. If they land closer together than that, the grader is +being generous and its scores on real handoffs mean nothing yet. Fix that before +going further. + +## Actionable + +1. **First action is executable.** Someone with no other context could carry out + the first thing in "Your scope" without asking a question. +2. **Everything named is findable.** Every file, module, command, and test named + in "Your scope" is given a path, or is somewhere the named reading will reach. +3. **Reading is ordered.** "Read first" says what to read *in what order*, not + just which documents are relevant. +4. **Authority is settled.** "Read first" says which documents outrank the + handoff, so a conflict between them resolves without asking. + +## Grounded + +5. **Nothing invented.** Every claim traces to the session or to the repository. + One fabricated file path fails this outright. +6. **Inference is marked.** Anything concluded rather than established is stated + as an inference, not asserted flatly. +7. **Points rather than paraphrases.** Where a rules file, design doc, or memory + file already covers something, the handoff names it instead of restating it. + +## Carries the state + +8. **Finished is separated from half-finished**, and anything half-finished has a + location. +9. **Specifics survive.** SHAs, test counts, versions, and error strings that the + session established appear verbatim rather than as "a few tests" or "recently". +10. **Breakage is declared.** What is broken, and what is unfinished on purpose, + are both stated. Silence here reads as "everything works". + +## Bounded + +11. **Says where to stop.** There is an explicit boundary — what not to start, + what not to commit, what is out of scope for this stretch. +12. **Says what to escalate.** Names at least one thing to bring back rather than + decide alone, or states plainly that the scope is fully delegated. + +## Constrained + +13. **Prohibitions carry their mechanism.** Each constraint says what enforces it + — a hook, a permission, a protected path, a service that will refuse. A bare + "be careful with X" does not count. +14. **Dead ends are recorded.** Approaches already tried and ruled out are named, + so the next agent does not spend the session rediscovering them. +15. **Human-only steps are flagged.** Anything the agent cannot do itself is + named, along with what to do instead of attempting it. + +## Honest about what is open + +16. **Options are preserved, not re-litigated.** Open decisions are presented as + they stood at the end of the session, without the handoff quietly picking one. +17. **The human's decisions are marked as theirs**, and separated from the + judgment calls the next agent is free to make. + +## Verifiable + +18. **Done is testable.** The exit criteria can be checked by running or reading + something. "Works correctly" fails; "full suite green, 214 passing" passes. +19. **The report is specified.** It says what the final message back to the human + needs to contain. + +## Written well enough to use + +20. **No filler.** No section padded to look complete, no summary of the summary, + no restating the request back. +21. **Register matches** the session it came from. + +## Scoring + +Total out of 21. + +Four are load-bearing: **1** (first action executable), **8** (state separated), +**11** (says where to stop), and **13** (prohibitions carry their mechanism). +Failing any of those caps the handoff at "not usable" regardless of the total — +they are the difference between a briefing and a nicely written recap. + +| Score | Reading | +|---|---| +| 18–21, all four load-bearing passed | Usable as written | +| 14–17 | Usable after the human patches the gaps; note which | +| Below 14, or any load-bearing failure | Regression — the skill is drifting back toward a recap | + +Report the failed assertion numbers, not just the total. A score with no failure +list is not actionable, which is the same complaint this rubric exists to make. From 3a02401673ee1daff260d752aeef7b19a0537d56 Mon Sep 17 00:00:00 2001 From: joshwand Date: Sun, 2 Aug 2026 21:55:44 -0700 Subject: [PATCH 2/2] Preserve the pre-rewrite skill and brief the Inspect AI wiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The baseline was about to be lost. The old skill never lived in this repo — its only copy was ~/.claude/skills/handoff/SKILL.md, which has since been overwritten with the rewritten version. Without a control arm a rubric score on the new skill alone can't distinguish "this is good" from "this judge is generous", so it is recovered here from the session transcript, typos intact, with its provenance stated: reconstructed, not git-attested. handoff.md briefs the next agent on wiring Inspect AI to the three existing cases. Written with the handoff skill and linted with lint_handoff.py, which passes it 0 failed 0 warned — the first end-to-end run of either against something not written as a fixture. Co-Authored-By: Claude Opus 5 (1M context) --- evals/baseline/README.md | 31 +++++++++++++++++++++ evals/baseline/SKILL.md | 59 ++++++++++++++++++++++++++++++++++++++++ handoff.md | 46 +++++++++++++++++++++++++++++++ 3 files changed, 136 insertions(+) create mode 100644 evals/baseline/README.md create mode 100644 evals/baseline/SKILL.md create mode 100644 handoff.md diff --git a/evals/baseline/README.md b/evals/baseline/README.md new file mode 100644 index 0000000..01b5bc7 --- /dev/null +++ b/evals/baseline/README.md @@ -0,0 +1,31 @@ +# Baseline condition + +`SKILL.md` here is the `handoff` skill as it existed *before* the rewrite. It is +the control arm for any A/B: run it and the current skill through the same cases, +paired, and the difference is what the rewrite bought. + +Without it there is no baseline, and a rubric score on the current skill alone +cannot distinguish "this is good" from "this judge is generous". + +## Provenance, stated honestly + +This file was recovered from a session transcript, not from version control. + +It never lived in this repository. Its only copy was `~/.claude/skills/handoff/SKILL.md` +in Josh's home directory, which has since been overwritten with the rewritten +skill, so the original is no longer on disk anywhere. The text here is a faithful +reproduction of that file as read on 2026-08-02 — 58 lines, including its two +typos (`refrerencing`, and `Save the file as handoff.md in .`), which are left in +deliberately because a cleaned-up baseline is not the baseline. + +It is unverifiable against a commit. Treat it as a high-confidence reconstruction +rather than a git-attested artifact, and say so in any writeup that uses it. + +## What it is + +The five retrospective sections — Who this is for, What we covered, What was +confirmed, Still in progress, Next steps — plus a trailing References to load, a +plain-prose mandate, and a 200 to 400 word cap. + +`../calibration/bad.md` is an *output* of this skill, hand-written for case 01. +This directory is the skill itself, which is what an A/B needs. diff --git a/evals/baseline/SKILL.md b/evals/baseline/SKILL.md new file mode 100644 index 0000000..ffbe390 --- /dev/null +++ b/evals/baseline/SKILL.md @@ -0,0 +1,59 @@ +--- +name: handoff +description: > + Creates a markdown handoff document from the current conversation, designed to be read into a new Claude chat as a briefing. Trigger this skill whenever the user types /handoff, says "create a handoff", "write a handoff", etc. +--- + +# Handoff Skill + +## Purpose + +The user is ending or pausing a session and wants to carry the full context into a new one. Your job is to read everything that was discussed in this conversation and produce a single plain-text file that a new Claude — with no prior context — could read and immediately understand: who the user is, what was worked on, what was decided, what is unfinished, and what to do next. + +This document is a briefing, not a summary. Write it as if you are handing over to a capable colleague. Be specific and concrete. Do not be vague. + +## Output format + +The file must be plain text — no markdown, no headers with hashes, no bullet points, no bold, no dashes used as list markers. Write in complete sentences and paragraphs. Use plain section labels followed by a colon and a line break to separate sections. + +Use exactly these five sections in this order: + +Who this is for: +Write one or two sentences identifying the user by name if known, their role or context, and the broad nature of what they were working on during this session. Draw only from what the conversation reveals. + +What we covered: +A single paragraph (three to six sentences) summarising the main topics, questions, and decisions from the session. This should read as a coherent narrative, not a list. If multiple distinct things were covered, weave them together rather than enumerating them. + +What was confirmed: +Write in plain sentences the specific facts, conclusions, technical decisions, or agreements that were reached during the session — things that were settled and can be treated as ground truth in the next session. If nothing was definitively confirmed, say so honestly. + +Still in progress: +Write in plain sentences anything that was started but not completed, flagged as uncertain, left open, or deprioritised. Include anything the user said they would do later. If nothing is pending, say so. + +Next steps: +Write in plain sentences what the user should pick up in the next session. This should follow logically from what is still in progress. Be specific — name the task, file, decision, or question that needs attention next. + +References to load: +Specific file paths to read into context at the beginning of the session. Give instructions to do so. + +## Rules — read these carefully + +Do not invent anything. Every sentence must be grounded in something that actually appeared in the conversation. If a section has nothing to put in it, write a brief honest statement to that effect rather than padding it out. + +Do not add a preamble before the first section. Do not add commentary or meta-explanation after the last section. + +Be specific. Vague handoffs are useless. Name the thing, the file, the decision, the error, the next action — whatever it is, say it plainly. + +Match the user's register. If the conversation was casual and direct, write casually and directly. If it was technical and precise, match that. + +Keep it tight. A good handoff is usually between 200 and 400 words. Longer is only justified if the session was genuinely complex and the detail is needed. + +## After producing the file + +Once the file is written and saved, tell the user clearly where it is and that they can open and edit any section before refrerencing or pasting it into a new chat. + +Do not re-summarise the file back to the user in the chat; just a simple statement on what the next step will be. Just tell them it is ready and where it is. + +## File output + +Save the file as `handoff.md` in . If a `handoff.md` already exists, save as `handoff-2.md`, and so on. diff --git a/handoff.md b/handoff.md new file mode 100644 index 0000000..ed5e55e --- /dev/null +++ b/handoff.md @@ -0,0 +1,46 @@ +What you are picking up: +You are wiring the eval suite for the handoff skill up to Inspect AI, the UK AISI evaluation framework, against the three cases that already exist. Josh chose Inspect over the alternatives and approved this scope; the suite itself is written and open as draft PR #4 on branch handoff-evals. + +Read first: +Read evals/README.md first, because it describes how the suite is meant to run and is authoritative over this briefing wherever the two disagree. Then read evals/rubric.md, which holds the 21 assertions you are turning into a scorer, and evals/calibration/README.md, which states the thresholds the grader has to reproduce. Then read one full case, evals/cases/01-midstream-implementation/session.md together with its expected.md, to see the input and output shape before you write any code. Then read evals/lint_handoff.py, because you are reusing it rather than reimplementing it. Read AGENTS.md for the working agreements; NoPlaceholdersWithoutApproval applies directly to scorers. Inspect's documentation is linked from https://github.com/UKGovernmentBEIS/inspect_ai and is authoritative over anything this file says about its API, which is described here from a summary rather than from having used it. + +State you inherit: +The branch is handoff-evals at ba7da9b, open as draft PR #4 with base handoff-skill. It retargets to main automatically when PR #2 merges. PR #2 carries the skill itself, is three files, and is MERGEABLE and CLEAN against main; it is not yours to touch. + +Under evals/ there is README.md, rubric.md, lint_handoff.py, three cases each holding session.md and expected.md, a calibration pair, and a baseline. + +lint_handoff.py is finished and verified. It passes evals/calibration/good.md with 0 failed and 0 warned, and fails evals/calibration/bad.md with 9 failed and 3 warned. It exits 0 on pass, 1 on failure or an unreadable file, and 2 when called with no arguments. Its check() function yields tuples of level, line number, and message, which is the interface to build the deterministic scorer on. + +The rubric has never been run end to end. No handoff has been generated from a case and graded. The thresholds in evals/calibration/README.md, good at 18 or above and bad at 10 or below, are guesses that have never been tested against a real grader. + +evals/baseline/SKILL.md is the pre-rewrite skill, the control arm for an A/B. Read evals/baseline/README.md before using it: it was recovered from a session transcript rather than from version control, because its only copy was overwritten, so it is a high-confidence reconstruction and not a git-attested artifact. + +Python is 3.14.3. There is a .venv in the main working copy at the repository root, untracked. There is no ANTHROPIC_API_KEY in the environment. + +Your scope: +Add Inspect AI as a dependency scoped to the eval, and build one Task whose dataset is the three cases, whose solver generates a handoff with the skill text as the system prompt and the case session.md as the user message, and which carries two scorers. The first is deterministic and wraps lint_handoff.check(); import it, do not rewrite it. The second is model graded against the 21 assertions in rubric.md, using forced structured output rather than parsing prose, and it must report per-assertion results rather than only a total, because the per-assertion failure rate is the diagnostic the suite exists to produce. + +Then calibrate before trusting anything: score evals/calibration/good.md and evals/calibration/bad.md through the rubric scorer and record the actual numbers. If they contradict the 18 and 10 thresholds, correcting those numbers in evals/calibration/README.md and evals/README.md with the evidence is in scope and wanted. Then run the three cases and report per-assertion failure rates with bootstrap confidence intervals, which Inspect provides natively. + +Stop there. Do not build the pairwise or A/B arm, do not write new cases, do not edit skills/handoff/SKILL.md or anything in PR #2, and do not add dependencies anywhere outside the eval. Commit on handoff-evals only, and leave PR #4 in draft unless Josh says otherwise. + +Standing constraints: +There is no ANTHROPIC_API_KEY in the environment and you cannot obtain one. Inspect calls the API directly, so nothing runs until Josh supplies a key. Prepare everything up to that point, and if you reach it, say exactly what you need and stop rather than working around it. + +Do not force-push handoff-skill or handoff-evals, and do not push to main. Both branches are pushed and under review; PR #2 deliberately carries an add-then-remove commit pair rather than a rewritten history for this reason. + +Four approaches were considered and ruled out, and reopening one needs an argument rather than a fresh look. promptfoo fits the prompt-matrix shape best but OpenAI agreed to acquire it in 2026. DeepEval's metric catalogue is RAG-shaped and pulls toward a hosted dashboard. The hosted platforms, Braintrust and LangSmith and Weave, add an account dependency. A hand-rolled runner shelling out to claude -p was rejected on methodology: Claude Code's own system prompt sits above every generation as an uncontrolled variable, and a measured trivial call cost 0.11 dollars. + +Keep the dependency inside the eval. The repository README records that the installer was deliberately dropped and the always-on rules carry no tooling. + +Open questions: +Whether the A/B against evals/baseline/SKILL.md is in scope at all is Josh's call. He was interested but approved only the three-case wiring, and the baseline is preserved so the option stays open. + +Three cases cannot support a claim that the rewrite is better; a paired A/B at that size will produce a confidence interval on the difference that straddles zero. Josh has been told this. Whether to grow the set to fifteen or thirty cases before or after the harness is his decision, not yours. + +The generator model, the judge model, the temperature, and the number of samples per case are all undecided. Whether the judge should be a different model from the generator, to limit self-preference bias, is worth raising with him rather than settling quietly. + +Definition of done: +Inspect runs end to end across all three cases and produces per-assertion results and a viewable log. The deterministic scorer calls lint_handoff.py rather than duplicating it. A calibration run is recorded with real numbers, and the thresholds are either confirmed or corrected with the evidence attached. evals/README.md describes the runner as the primary path, with the manual procedure either kept or removed as a deliberate choice rather than left stale beside it. The work is committed on handoff-evals and pushed, and PR #4's description says what is now verified and what still is not. + +Report back with the commands you ran, the actual calibration numbers, which assertions failed and how often, anything you changed about the thresholds and why, the cost and wall-clock of a full run, and whatever needs Josh's decision. Then stop.