Add an eval suite for the handoff skill

.gitignore

@@ -6,4 +6,7 @@
 repomix-output.md
 repomix-output.xml
 .cursorindexingignore
-.venv
+.venv
+# real handoffs to grade against the rubric, kept out of the public repo
+evals/local/
+__pycache__/

README.md

@@ -63,6 +63,12 @@ The blocks that are large but only situationally needed, kept out of the always-
 - **`interactive-planning`** — the one-question-at-a-time elicitation prompt behind `.ip`.
 - **`handoff`** — the briefing behind `.cn`, for moving work into a fresh chat. Its output is written to the agent picking the work up rather than as a recap for you, so the sections are operative: what to read and in what order, the state inherited, the scope and where to stop, the standing constraints, and the definition of done.
 
+### `evals/`
+
+A small suite for the `handoff` skill, because prompts regress quietly — a skill can be edited into something that still produces a plausible document while dropping the parts that made it useful.
+
+`lint_handoff.py` is stdlib-only and decides the mechanical rules: sections present and in order, plain prose held, something paste-able actually named. `rubric.md` holds 21 substance assertions for a model to grade, over three synthetic sessions in `cases/`. `calibration/` is a known-good and known-bad pair used to check the grader before trusting it on anything real — a grader that can't separate those two is rewarding fluent prose, which is the failure being hunted.
+
 ## Structured memory
 
 Credit where due: the memory bank is an adaptation of the [Cline Memory Bank](https://docs.cline.bot/improving-your-prompting-skills/cline-memory-bank).

evals/README.md

@@ -0,0 +1,84 @@
+# Evals
+
+A small suite for the `handoff` skill. Prompts regress silently — a skill can get
+edited into something that still produces a plausible document while quietly
+dropping the parts that made it useful. This catches that.
+
+Two layers, because the failures come in two kinds. Format regressions are cheap
+and mechanical, so a script decides them. Substance regressions need judgment, so
+a rubric and a model decide those. No dependencies either way: stdlib Python and
+markdown.
+
+```
+lint_handoff.py    deterministic structural checks
+rubric.md          21 substance assertions, model-graded
+cases/             synthetic sessions to generate handoffs from
+calibration/       a known-good and known-bad pair, to test the grader
+local/             gitignored; your real handoffs
+```
+
+## The linter
+
+```
+python3 evals/lint_handoff.py handoff.md
+```
+
+Checks the seven sections are present and in order, the plain-prose rule holds,
+nothing precedes the first section, "Standing constraints" is not empty, and
+something paste-able (a path, a command, a SHA) actually appears — including
+inside "Read first" specifically.
+
+FAIL exits non-zero. WARN never fails the run; warnings flag recap phrasing,
+missing second-person address, and documents thin enough to be worth re-reading.
+
+Verify it works on the pair:
+
+```
+python3 evals/lint_handoff.py evals/calibration/good.md   # PASS, 0 failed, 0 warned
+python3 evals/lint_handoff.py evals/calibration/bad.md    # FAIL, 9 failed, 3 warned
+```
+
+## The full eval
+
+The linter cannot tell a specific handoff from a vague one, which is the failure
+that matters. For that:
+
+1. **Calibrate the grader.** Score `calibration/good.md` and `calibration/bad.md`
+   against `rubric.md`. Good must land at 18+, bad at 10 or below. If they land
+   close together, stop — see `calibration/README.md`.
+2. **Generate.** In a fresh chat with the skill available, paste a case's
+   `session.md` as context and ask for a handoff (`.cn`). A fresh chat matters:
+   grading a handoff in the session that wrote it tests nothing, because the
+   grader can see everything the handoff left out.
+3. **Lint** the result.
+4. **Grade** it against `rubric.md` plus the case's `expected.md`. Report failed
+   assertion numbers, not just a total.
+
+Three or four handoffs per case is more informative than one — the interesting
+question is which assertions fail *repeatedly*, since a single miss is sampling
+noise and a pattern is a gap in the skill.
+
+## The cases
+
+| Case | Shape | Stresses |
+|---|---|---|
+| `01-midstream-implementation` | Real state, a blocked carryover, protected paths, a human-only step, an open decision | Everything; this is the hard one |
+| `02-planning-only` | Nothing built, no constraints, no blockers | Graceful degradation — does the format pad or stay honest |
+| `03-blocked-on-human` | Stopped on a human action, two live options | Whether open decisions survive unchosen |
+
+Each case has a `session.md` (the input) and an `expected.md` (the facts that must
+survive, and how the case is usually failed).
+
+## Adding a case
+
+New cases should come from a session that produced a *bad* handoff. That is the
+one reliable source of eval material: a fixture invented to be tricky tests what
+you imagined, while one derived from a real failure tests what actually goes
+wrong. Synthesize the shape, drop the project specifics, and write the
+`expected.md` from what the real handoff lost.
+
+## local/
+
+`evals/local/` is gitignored. Drop real handoffs there and grade them against the
+same rubric without publishing project details. Nothing in the suite depends on
+it being populated.

evals/baseline/README.md

@@ -0,0 +1,31 @@
+# Baseline condition
+
+`SKILL.md` here is the `handoff` skill as it existed *before* the rewrite. It is
+the control arm for any A/B: run it and the current skill through the same cases,
+paired, and the difference is what the rewrite bought.
+
+Without it there is no baseline, and a rubric score on the current skill alone
+cannot distinguish "this is good" from "this judge is generous".
+
+## Provenance, stated honestly
+
+This file was recovered from a session transcript, not from version control.
+
+It never lived in this repository. Its only copy was `~/.claude/skills/handoff/SKILL.md`
+in Josh's home directory, which has since been overwritten with the rewritten
+skill, so the original is no longer on disk anywhere. The text here is a faithful
+reproduction of that file as read on 2026-08-02 — 58 lines, including its two
+typos (`refrerencing`, and `Save the file as handoff.md in .`), which are left in
+deliberately because a cleaned-up baseline is not the baseline.
+
+It is unverifiable against a commit. Treat it as a high-confidence reconstruction
+rather than a git-attested artifact, and say so in any writeup that uses it.
+
+## What it is
+
+The five retrospective sections — Who this is for, What we covered, What was
+confirmed, Still in progress, Next steps — plus a trailing References to load, a
+plain-prose mandate, and a 200 to 400 word cap.
+
+`../calibration/bad.md` is an *output* of this skill, hand-written for case 01.
+This directory is the skill itself, which is what an A/B needs.

evals/baseline/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: handoff
+description: >
+  Creates a markdown handoff document from the current conversation, designed to be read into a new Claude chat as a briefing. Trigger this skill whenever the user types /handoff, says "create a handoff", "write a handoff", etc. 
+---
+
+# Handoff Skill
+
+## Purpose
+
+The user is ending or pausing a session and wants to carry the full context into a new one. Your job is to read everything that was discussed in this conversation and produce a single plain-text file that a new Claude — with no prior context — could read and immediately understand: who the user is, what was worked on, what was decided, what is unfinished, and what to do next.
+
+This document is a briefing, not a summary. Write it as if you are handing over to a capable colleague. Be specific and concrete. Do not be vague.
+
+## Output format
+
+The file must be plain text — no markdown, no headers with hashes, no bullet points, no bold, no dashes used as list markers. Write in complete sentences and paragraphs. Use plain section labels followed by a colon and a line break to separate sections.
+
+Use exactly these five sections in this order:
+
+Who this is for:
+Write one or two sentences identifying the user by name if known, their role or context, and the broad nature of what they were working on during this session. Draw only from what the conversation reveals.
+
+What we covered:
+A single paragraph (three to six sentences) summarising the main topics, questions, and decisions from the session. This should read as a coherent narrative, not a list. If multiple distinct things were covered, weave them together rather than enumerating them.
+
+What was confirmed:
+Write in plain sentences the specific facts, conclusions, technical decisions, or agreements that were reached during the session — things that were settled and can be treated as ground truth in the next session. If nothing was definitively confirmed, say so honestly.
+
+Still in progress:
+Write in plain sentences anything that was started but not completed, flagged as uncertain, left open, or deprioritised. Include anything the user said they would do later. If nothing is pending, say so.
+
+Next steps:
+Write in plain sentences what the user should pick up in the next session. This should follow logically from what is still in progress. Be specific — name the task, file, decision, or question that needs attention next.
+
+References to load:
+Specific file paths to read into context at the beginning of the session. Give instructions to do so. 
+
+## Rules — read these carefully
+
+Do not invent anything. Every sentence must be grounded in something that actually appeared in the conversation. If a section has nothing to put in it, write a brief honest statement to that effect rather than padding it out.
+
+Do not add a preamble before the first section. Do not add commentary or meta-explanation after the last section.
+
+Be specific. Vague handoffs are useless. Name the thing, the file, the decision, the error, the next action — whatever it is, say it plainly.
+
+Match the user's register. If the conversation was casual and direct, write casually and directly. If it was technical and precise, match that.
+
+Keep it tight. A good handoff is usually between 200 and 400 words. Longer is only justified if the session was genuinely complex and the detail is needed.
+
+## After producing the file
+
+Once the file is written and saved, tell the user clearly where it is and that they can open and edit any section before refrerencing or pasting it into a new chat. 
+
+Do not re-summarise the file back to the user in the chat; just a simple statement on what the next step will be. Just tell them it is ready and where it is.
+
+## File output
+
+Save the file as `handoff.md` in . If a `handoff.md` already exists, save as `handoff-2.md`, and so on. 

evals/calibration/README.md

@@ -0,0 +1,38 @@
+# Calibration pair
+
+Two handoffs written from `../cases/01-midstream-implementation/`. They exist to
+test the grader, not the skill.
+
+Score these before scoring anything real. A grader that cannot separate them is
+not measuring anything, and its verdict on a fresh handoff is noise.
+
+| File | Expected rubric score | Expected linter result |
+|---|---|---|
+| `good.md` | 18 or above, all four load-bearing assertions passed | PASS, no warnings |
+| `bad.md` | 10 or below, load-bearing assertions failed | FAIL — 9 failures (seven missing sections, a preamble, nothing concrete anywhere) and 3 warnings |
+
+If `good.md` scores below 18, the grader is too harsh, or the rubric has an
+assertion the skill never promised. If `bad.md` scores above 10, the grader is
+rewarding fluent prose — which is the exact failure this whole eval exists to
+catch, since `bad.md` is fluent, well organised, and nearly useless.
+
+## What bad.md gets wrong
+
+It is not a strawman. It is the shape the skill produced before the rewrite, and
+it reads fine until you try to act on it.
+
+Every specific has been sanded off: no SHA, no test count, no file paths, no test
+node id, `rapidfuzz` never named, the design doc referred to as "the design doc"
+rather than by path. The half-built tie breaker — the thing that blocks all the
+work — appears as "some unfinished work in the candidate selection code", so the
+next agent starts on the engine and discovers the stub the hard way.
+
+Two errors are worse than vagueness. The deliberate xfail is described as "one of
+the fuzzy name tests is currently failing", which invites the next agent to fix
+it and thereby destroy the signal. And "pick up the next step of the plan" omits
+that step 4 and only step 4 was authorized, so step 5 is a live risk.
+
+Everything in the constraints category is simply gone: the protected file and its
+hook, the container build failure behind the dropped dependency, the human-only
+credential rotation, the instruction not to commit. None of it is contradicted —
+it is just absent, which reads exactly like there being none.

evals/calibration/bad.md

@@ -0,0 +1,17 @@
+Who this is for:
+This is for Josh, who is working on tessera, a Python CLI for reconciling invoices against a ledger. This session continued a multi-step rewrite of the reconciler.
+
+What we covered:
+We worked through the early steps of the reconciler rewrite plan and got them committed. Along the way we dealt with a dependency that turned out not to work for us and had to be swapped for something in the standard library. We also looked at the matching code and at how fuzzy name comparison should handle accented characters, which is a question that is still open. The test suite is in good shape throughout.
+
+What was confirmed:
+The first several steps of the rewrite are done and committed, and the tests pass. The matcher will use the standard library rather than the third-party option that was tried. Josh approved moving on to the next step of the plan.
+
+Still in progress:
+The matching engine has not been built yet. There is some unfinished work in the candidate selection code. One of the fuzzy name tests is currently failing. The normalization question remains unresolved.
+
+Next steps:
+Pick up the next step of the plan and build the matching engine. Read the design document first for the details of how it should work. It would also be worth resolving the normalization question at some point, since it affects the failing test.
+
+References to load:
+Read the design doc in the memory knowledge base, and the current task state file.

evals/calibration/good.md

@@ -0,0 +1,30 @@
+What you are picking up:
+You are continuing a six step rewrite of the reconciler in tessera, a Python CLI that reconciles invoices against a ledger. Steps 1 through 3 are done and committed at 4c1f9ab. Josh has authorized step 4, and only step 4.
+
+Read first:
+Read _memory/knowledgeBase/designs/ReconcilerRewrite.md before anything else. It is the plan, step 4 is your scope within it, and it is authoritative over this briefing wherever the two disagree. Then read _memory/currentState/currentTaskState.md for where things stood at the end of the last session. Then skim tessera/match/candidates.py and tessera/match/engine.py, in that order, because the first one blocks the second.
+
+State you inherit:
+Steps 1 through 3 are complete and committed together at 4c1f9ab. The suite is 214 passing, 0 skipped, run with .venv/bin/pytest -q.
+
+tessera/match/candidates.py is half built, and it blocks everything else. The scoring function is written and tested; the tie breaker below it is a TODO stub that returns the first candidate. Step 4 consumes that function, so finishing the tie breaker comes before any work on the engine.
+
+tests/test_fuzzy.py::test_unicode_names is xfail deliberately, pending the normalization decision below. It is not a regression, and it must not be made to pass by changing the assertion.
+
+Your scope:
+Finish the tie breaker in tessera/match/candidates.py first. Then build the matching engine at tessera/match/engine.py as step 4 of the plan describes, with tests for both.
+
+Do not start step 5, and do not commit; Josh reviews before anything lands. Keep _memory/currentState/currentTaskState.md current as you go. If step 4 turns out to need a change in a protected file, stop and propose it rather than working around it, as described below.
+
+Standing constraints:
+tessera/config/rates.py is owner edited. Write is permission denied on it, and a pre-commit hook rejects any commit that touches it. When step 2 needed a change there, the change went into tessera/config/rates_proposed.py alongside it and Josh applied it by hand. Do the same rather than trying to edit the original.
+
+Do not reintroduce rapidfuzz. Step 2 used it for the matcher and it came out again because it pulls a C extension that breaks the Alpine container build. That failure does not surface until the image builds in CI, so it looks harmless locally. Stdlib difflib is the decision.
+
+The production reconciliation run needs a credential rotated in the bank's web console. There is no API for it, and you have neither a browser nor the credentials. If step 4 gets far enough to want a real run, prepare everything up to the run, write the exact steps Josh needs to click into _memory/currentState/currentTaskState.md, and stop there.
+
+Open questions:
+Unicode name normalization is undecided, and it is Josh's call rather than yours. NFKC is more correct and keeps names readable in the reconciliation output. Stripping diacritics before comparing matches more aggressively, which is closer to what reconciliation actually wants, but produces output a human cannot check by eye. This decides tests/test_fuzzy.py::test_unicode_names, so leave that test xfail until he chooses.
+
+Definition of done:
+The tie breaker in tessera/match/candidates.py is finished and tested. Step 4 is implemented as the plan describes, with tests. The full suite is green, with the unicode test still xfail. _memory/currentState/currentTaskState.md is updated well enough that a fresh agent could take step 5 from it alone. Your final report to Josh covers what was built, the test count, any deviations from the plan, and anything needing his decision. Then stop.