From 0e6194a9b88b3eb8059676e0af4c11e3f62046d3 Mon Sep 17 00:00:00 2001 From: joshwand Date: Sun, 2 Aug 2026 18:07:11 -0700 Subject: [PATCH 1/3] Add a handoff skill and point .cn at it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit .cn described the whole behavior inline in one alias line: "give me a standalone prompt for the next agent to continue this process." That leaves the shape of the document entirely to the model, and what comes back is a session recap — what we covered, what was confirmed, what is next — addressed to nobody in particular. The sections are the fix. A briefing for an agent is operative, not retrospective: what to read and in what order, the state it inherits, its scope and where to stop, the standing constraints, and a definition of done. Constraints are the section that recaps never have and the one that saves the most time, so it is required; if a session produced none, the handoff says so rather than dropping it. Output stays plain prose, but precision is exempted from the prose rule — paths, SHAs, test counts and error strings are reproduced verbatim, because a path the next agent cannot paste is a broken handoff. The old 200-400 word cap is gone; it was forcing exactly the vagueness the skill is meant to prevent. Moves .cn from the command aliases into the prompt aliases, alongside .ip and .um, which is where the skill-backed ones live. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 5 ++-- README.md | 3 ++- skills/handoff/SKILL.md | 60 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 3 deletions(-) create mode 100644 skills/handoff/SKILL.md diff --git a/AGENTS.md b/AGENTS.md index 90ef1e4..c41b554 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,8 +60,6 @@ Shorthand I use in prompts. When a message is one of these tokens, expand it as .v verify that the work you have done is correct and works as expected .pi prove it: show the passing tests, the screenshot, the actual output .nd, .ndy, .ynd not done until the functionality completely works; no partial victories -.cn give me a standalone prompt for the next agent to continue this process. - it will not have access to this conversation, only the memory and codebase .rj repeat ("reinject") my goals, plan, and instructions into the conversation .rrr re-read the rules files (or if specified) .dr, .ds don't start new instances of running processes; anything already running @@ -77,6 +75,9 @@ These expand into a prompt rather than a one-line instruction. They take space-d ``` .ip Interactive Planning — use the `interactive-planning` skill +.cn write a standalone briefing for the next agent to continue this work. + it will not have access to this conversation, only the memory and + codebase — use the `handoff` skill .m load memory, then do : npx repomix --quiet --include _memory/ --ignore _memory/knowledgeBase --style markdown --stdout .mc .m, then .c — for moving a too-long chat into a fresh one diff --git a/README.md b/README.md index 932da56..0f4c443 100644 --- a/README.md +++ b/README.md @@ -53,10 +53,11 @@ The aliases are short tokens so you can steer without a lot of typing: `.c` to c ### `skills/` -The two blocks that are large but only situationally needed, kept out of the always-on rules and loaded on demand: +The blocks that are large but only situationally needed, kept out of the always-on rules and loaded on demand: - **`memory-bank`** — the `_memory/` schema, how to set one up for a project, and how to keep it current. Bundles the `currentTaskState.md` template. - **`interactive-planning`** — the one-question-at-a-time elicitation prompt behind `.ip`. +- **`handoff`** — the briefing behind `.cn`, for moving work into a fresh chat. Its output is written to the agent picking the work up rather than as a recap for you, so the sections are operative: what to read and in what order, the state inherited, the scope and where to stop, the standing constraints, and the definition of done. ## Structured memory diff --git a/skills/handoff/SKILL.md b/skills/handoff/SKILL.md new file mode 100644 index 0000000..77e0546 --- /dev/null +++ b/skills/handoff/SKILL.md @@ -0,0 +1,60 @@ +--- +name: handoff +description: Write a standalone briefing that lets a fresh agent continue this work with no access to the current conversation. Use when invoked as .cn or /handoff, when asked for a handoff, a continuation prompt, or a prompt for the next agent, and when a chat has grown too long and the work is moving to a new one. +--- + +# Handoff + +Produce one file that an agent with no memory of this conversation can read and then act on. It gets the repository and `_memory/`; it does not get the chat. Everything it needs that is not already written down somewhere it will read has to be in this file. + +This is an instruction to that agent, not a report to me about the session. Write it in the second person, addressed to whoever picks the work up: "you are continuing", "in scope for you first". If a sentence describes the session rather than telling the next agent something it must know or do, cut it. + +## Output format + +Plain text. No markdown headers, no bullet points, no bold, no dashes as list markers. Complete sentences and paragraphs, with plain section labels followed by a colon and a line break. + +The prose rule governs style, not precision. File paths, commands, commit SHAs, test counts, and error strings are reproduced exactly as they appear, never paraphrased or tidied up. Ordering that matters is carried in the sentence ("read X first, then Y"), not in a numbered list. + +Use these seven sections, in this order. + +**What you are picking up:** One or two sentences. What is being continued, roughly where it stands, and what I have already approved. + +**Read first:** The exact paths and commands to load, in the order to load them, each with a short reason. Include the memory load (`.m`) if the project has a `_memory/`. Say which documents are authoritative over the handoff itself, so that where they disagree the agent believes them and not this file. + +**State you inherit:** What is true right now, not the story of how it got that way. What is finished and committed, with SHAs. Test counts. What is half-built and at which path. What is known broken, and what is deliberately unfinished. Where a decision's reasoning still matters, attach it to the fact rather than narrating it separately. + +**Your scope:** What to do, in order, and explicitly where to stop. Name the boundary: what not to start, what not to commit, what to bring back to me instead of deciding. This is the section that makes a handoff an instruction rather than a suggestion. + +**Standing constraints:** What the agent must not do, and what will stop it if it tries: protected or owner-edited paths, permission and hook denials, services or credentials that are off limits, actions it cannot perform itself and has to hand back to me. Include anything already tried and ruled out, so it isn't attempted again. If the session genuinely produced no constraints, say so in one sentence rather than dropping the section. + +**Open questions:** What is still undecided, with the options as they stood, and which of them need me specifically rather than a judgment call. + +**Definition of done:** Verifiable exit criteria, and what the final report back to me should contain. + +## Rules + +Ground every sentence in something that actually happened in the conversation or exists in the repository. Do not invent. State established facts flatly and mark anything inferred as an inference. + +Point rather than paraphrase. If `AGENTS.md`, a design document, or a memory file already says it, name that file and call it authoritative. Restating it here creates a second copy that will drift from the first. + +Be specific enough to act on. Name the file, the command, the error, the decision. A path the next agent cannot paste is a broken handoff. + +Match my register. If the session was casual and direct, write that way; if it was precise and technical, match that. + +Length is whatever the state and the constraints require, with no filler. Most useful handoffs run four hundred to nine hundred words. A long constraints section is never the thing to cut. + +No preamble before the first section and no commentary after the last. + +Before saving, read it cold: could an agent with no other context carry out the first action in "Your scope" without asking a single question? Is every file, command, and identifier named? Fix it before writing the file. + +## Relationship to memory + +`.ts` and `_memory/currentState/` are the durable record of where the work stands, and they belong to the project. A handoff is a one-shot briefing aimed at a specific next step, written to be pasted into a new chat. + +They overlap, and the handoff should lean on memory rather than duplicate it: update task state first if it is stale, then have the handoff point at it. What belongs in the handoff and nowhere else is the scope boundary, the standing constraints, and the definition of done for this particular stretch of work. + +## File output + +Save as `handoff.md` in the current working directory. If that exists, use `handoff-2.md`, then `handoff-3.md`, and so on. + +Then say where the file is and that it can be edited before use. Do not summarise it back into the chat. From 0c550b8ff11b224aa1d17e8abe4a3ff0d1c5709f Mon Sep 17 00:00:00 2001 From: joshwand Date: Sun, 2 Aug 2026 18:22:08 -0700 Subject: [PATCH 2/3] Add an eval suite for the handoff skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rewrite in the previous commit was justified by comparing two real handoffs, one useful and one not. That comparison was the whole method, and it was done by hand and thrown away. This keeps it. Prompts regress in a way code doesn't: an edit can leave a skill still producing something fluent and well organised that has quietly stopped being actionable. Nothing fails. You find out two sessions later when an agent starts on the wrong thing. Two layers, because there are two kinds of failure. Format regressions are mechanical, so lint_handoff.py decides them in stdlib Python — the seven sections present and ordered, prose rule held, constraints not empty, something paste-able actually named. Substance regressions need judgment, so rubric.md carries 21 assertions for a model to grade, weighted so that four load-bearing ones cap the score if failed. The calibration pair is the part worth keeping honest. good.md and bad.md are both written from case 01, and the grader scores them before it scores anything real: good must clear 18, bad must stay under 10. A grader that can't separate them is rewarding fluent prose, which is precisely the failure this exists to catch — bad.md is fluent, well organised, and useless, because it is what the skill produced before. Three cases: the hard one with real state and constraints, a planning-only session that tests whether required sections get padded when there's nothing to put in them, and one blocked on a human where two live options have to survive unchosen. Fixtures are synthetic. evals/local/ is gitignored for grading real handoffs without publishing project details. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 5 +- README.md | 6 + evals/README.md | 84 +++++++++ evals/calibration/README.md | 38 ++++ evals/calibration/bad.md | 17 ++ evals/calibration/good.md | 30 +++ .../01-midstream-implementation/expected.md | 55 ++++++ .../01-midstream-implementation/session.md | 62 ++++++ evals/cases/02-planning-only/expected.md | 47 +++++ evals/cases/02-planning-only/session.md | 40 ++++ evals/cases/03-blocked-on-human/expected.md | 49 +++++ evals/cases/03-blocked-on-human/session.md | 53 ++++++ evals/lint_handoff.py | 176 ++++++++++++++++++ evals/rubric.md | 100 ++++++++++ 14 files changed, 761 insertions(+), 1 deletion(-) create mode 100644 evals/README.md create mode 100644 evals/calibration/README.md create mode 100644 evals/calibration/bad.md create mode 100644 evals/calibration/good.md create mode 100644 evals/cases/01-midstream-implementation/expected.md create mode 100644 evals/cases/01-midstream-implementation/session.md create mode 100644 evals/cases/02-planning-only/expected.md create mode 100644 evals/cases/02-planning-only/session.md create mode 100644 evals/cases/03-blocked-on-human/expected.md create mode 100644 evals/cases/03-blocked-on-human/session.md create mode 100644 evals/lint_handoff.py create mode 100644 evals/rubric.md diff --git a/.gitignore b/.gitignore index 4d91d01..0e5d36c 100644 --- a/.gitignore +++ b/.gitignore @@ -6,4 +6,7 @@ repomix-output.md repomix-output.xml .cursorindexingignore -.venv \ No newline at end of file +.venv +# real handoffs to grade against the rubric, kept out of the public repo +evals/local/ +__pycache__/ \ No newline at end of file diff --git a/README.md b/README.md index 0f4c443..6fe88bc 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,12 @@ The blocks that are large but only situationally needed, kept out of the always- - **`interactive-planning`** — the one-question-at-a-time elicitation prompt behind `.ip`. - **`handoff`** — the briefing behind `.cn`, for moving work into a fresh chat. Its output is written to the agent picking the work up rather than as a recap for you, so the sections are operative: what to read and in what order, the state inherited, the scope and where to stop, the standing constraints, and the definition of done. +### `evals/` + +A small suite for the `handoff` skill, because prompts regress quietly — a skill can be edited into something that still produces a plausible document while dropping the parts that made it useful. + +`lint_handoff.py` is stdlib-only and decides the mechanical rules: sections present and in order, plain prose held, something paste-able actually named. `rubric.md` holds 21 substance assertions for a model to grade, over three synthetic sessions in `cases/`. `calibration/` is a known-good and known-bad pair used to check the grader before trusting it on anything real — a grader that can't separate those two is rewarding fluent prose, which is the failure being hunted. + ## Structured memory Credit where due: the memory bank is an adaptation of the [Cline Memory Bank](https://docs.cline.bot/improving-your-prompting-skills/cline-memory-bank). diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..b18099c --- /dev/null +++ b/evals/README.md @@ -0,0 +1,84 @@ +# Evals + +A small suite for the `handoff` skill. Prompts regress silently — a skill can get +edited into something that still produces a plausible document while quietly +dropping the parts that made it useful. This catches that. + +Two layers, because the failures come in two kinds. Format regressions are cheap +and mechanical, so a script decides them. Substance regressions need judgment, so +a rubric and a model decide those. No dependencies either way: stdlib Python and +markdown. + +``` +lint_handoff.py deterministic structural checks +rubric.md 21 substance assertions, model-graded +cases/ synthetic sessions to generate handoffs from +calibration/ a known-good and known-bad pair, to test the grader +local/ gitignored; your real handoffs +``` + +## The linter + +``` +python3 evals/lint_handoff.py handoff.md +``` + +Checks the seven sections are present and in order, the plain-prose rule holds, +nothing precedes the first section, "Standing constraints" is not empty, and +something paste-able (a path, a command, a SHA) actually appears — including +inside "Read first" specifically. + +FAIL exits non-zero. WARN never fails the run; warnings flag recap phrasing, +missing second-person address, and documents thin enough to be worth re-reading. + +Verify it works on the pair: + +``` +python3 evals/lint_handoff.py evals/calibration/good.md # PASS, 0 failed, 0 warned +python3 evals/lint_handoff.py evals/calibration/bad.md # FAIL, 9 failed, 3 warned +``` + +## The full eval + +The linter cannot tell a specific handoff from a vague one, which is the failure +that matters. For that: + +1. **Calibrate the grader.** Score `calibration/good.md` and `calibration/bad.md` + against `rubric.md`. Good must land at 18+, bad at 10 or below. If they land + close together, stop — see `calibration/README.md`. +2. **Generate.** In a fresh chat with the skill available, paste a case's + `session.md` as context and ask for a handoff (`.cn`). A fresh chat matters: + grading a handoff in the session that wrote it tests nothing, because the + grader can see everything the handoff left out. +3. **Lint** the result. +4. **Grade** it against `rubric.md` plus the case's `expected.md`. Report failed + assertion numbers, not just a total. + +Three or four handoffs per case is more informative than one — the interesting +question is which assertions fail *repeatedly*, since a single miss is sampling +noise and a pattern is a gap in the skill. + +## The cases + +| Case | Shape | Stresses | +|---|---|---| +| `01-midstream-implementation` | Real state, a blocked carryover, protected paths, a human-only step, an open decision | Everything; this is the hard one | +| `02-planning-only` | Nothing built, no constraints, no blockers | Graceful degradation — does the format pad or stay honest | +| `03-blocked-on-human` | Stopped on a human action, two live options | Whether open decisions survive unchosen | + +Each case has a `session.md` (the input) and an `expected.md` (the facts that must +survive, and how the case is usually failed). + +## Adding a case + +New cases should come from a session that produced a *bad* handoff. That is the +one reliable source of eval material: a fixture invented to be tricky tests what +you imagined, while one derived from a real failure tests what actually goes +wrong. Synthesize the shape, drop the project specifics, and write the +`expected.md` from what the real handoff lost. + +## local/ + +`evals/local/` is gitignored. Drop real handoffs there and grade them against the +same rubric without publishing project details. Nothing in the suite depends on +it being populated. diff --git a/evals/calibration/README.md b/evals/calibration/README.md new file mode 100644 index 0000000..ba80336 --- /dev/null +++ b/evals/calibration/README.md @@ -0,0 +1,38 @@ +# Calibration pair + +Two handoffs written from `../cases/01-midstream-implementation/`. They exist to +test the grader, not the skill. + +Score these before scoring anything real. A grader that cannot separate them is +not measuring anything, and its verdict on a fresh handoff is noise. + +| File | Expected rubric score | Expected linter result | +|---|---|---| +| `good.md` | 18 or above, all four load-bearing assertions passed | PASS, no warnings | +| `bad.md` | 10 or below, load-bearing assertions failed | FAIL — 9 failures (seven missing sections, a preamble, nothing concrete anywhere) and 3 warnings | + +If `good.md` scores below 18, the grader is too harsh, or the rubric has an +assertion the skill never promised. If `bad.md` scores above 10, the grader is +rewarding fluent prose — which is the exact failure this whole eval exists to +catch, since `bad.md` is fluent, well organised, and nearly useless. + +## What bad.md gets wrong + +It is not a strawman. It is the shape the skill produced before the rewrite, and +it reads fine until you try to act on it. + +Every specific has been sanded off: no SHA, no test count, no file paths, no test +node id, `rapidfuzz` never named, the design doc referred to as "the design doc" +rather than by path. The half-built tie breaker — the thing that blocks all the +work — appears as "some unfinished work in the candidate selection code", so the +next agent starts on the engine and discovers the stub the hard way. + +Two errors are worse than vagueness. The deliberate xfail is described as "one of +the fuzzy name tests is currently failing", which invites the next agent to fix +it and thereby destroy the signal. And "pick up the next step of the plan" omits +that step 4 and only step 4 was authorized, so step 5 is a live risk. + +Everything in the constraints category is simply gone: the protected file and its +hook, the container build failure behind the dropped dependency, the human-only +credential rotation, the instruction not to commit. None of it is contradicted — +it is just absent, which reads exactly like there being none. diff --git a/evals/calibration/bad.md b/evals/calibration/bad.md new file mode 100644 index 0000000..722a7a5 --- /dev/null +++ b/evals/calibration/bad.md @@ -0,0 +1,17 @@ +Who this is for: +This is for Josh, who is working on tessera, a Python CLI for reconciling invoices against a ledger. This session continued a multi-step rewrite of the reconciler. + +What we covered: +We worked through the early steps of the reconciler rewrite plan and got them committed. Along the way we dealt with a dependency that turned out not to work for us and had to be swapped for something in the standard library. We also looked at the matching code and at how fuzzy name comparison should handle accented characters, which is a question that is still open. The test suite is in good shape throughout. + +What was confirmed: +The first several steps of the rewrite are done and committed, and the tests pass. The matcher will use the standard library rather than the third-party option that was tried. Josh approved moving on to the next step of the plan. + +Still in progress: +The matching engine has not been built yet. There is some unfinished work in the candidate selection code. One of the fuzzy name tests is currently failing. The normalization question remains unresolved. + +Next steps: +Pick up the next step of the plan and build the matching engine. Read the design document first for the details of how it should work. It would also be worth resolving the normalization question at some point, since it affects the failing test. + +References to load: +Read the design doc in the memory knowledge base, and the current task state file. diff --git a/evals/calibration/good.md b/evals/calibration/good.md new file mode 100644 index 0000000..592f25b --- /dev/null +++ b/evals/calibration/good.md @@ -0,0 +1,30 @@ +What you are picking up: +You are continuing a six step rewrite of the reconciler in tessera, a Python CLI that reconciles invoices against a ledger. Steps 1 through 3 are done and committed at 4c1f9ab. Josh has authorized step 4, and only step 4. + +Read first: +Read _memory/knowledgeBase/designs/ReconcilerRewrite.md before anything else. It is the plan, step 4 is your scope within it, and it is authoritative over this briefing wherever the two disagree. Then read _memory/currentState/currentTaskState.md for where things stood at the end of the last session. Then skim tessera/match/candidates.py and tessera/match/engine.py, in that order, because the first one blocks the second. + +State you inherit: +Steps 1 through 3 are complete and committed together at 4c1f9ab. The suite is 214 passing, 0 skipped, run with .venv/bin/pytest -q. + +tessera/match/candidates.py is half built, and it blocks everything else. The scoring function is written and tested; the tie breaker below it is a TODO stub that returns the first candidate. Step 4 consumes that function, so finishing the tie breaker comes before any work on the engine. + +tests/test_fuzzy.py::test_unicode_names is xfail deliberately, pending the normalization decision below. It is not a regression, and it must not be made to pass by changing the assertion. + +Your scope: +Finish the tie breaker in tessera/match/candidates.py first. Then build the matching engine at tessera/match/engine.py as step 4 of the plan describes, with tests for both. + +Do not start step 5, and do not commit; Josh reviews before anything lands. Keep _memory/currentState/currentTaskState.md current as you go. If step 4 turns out to need a change in a protected file, stop and propose it rather than working around it, as described below. + +Standing constraints: +tessera/config/rates.py is owner edited. Write is permission denied on it, and a pre-commit hook rejects any commit that touches it. When step 2 needed a change there, the change went into tessera/config/rates_proposed.py alongside it and Josh applied it by hand. Do the same rather than trying to edit the original. + +Do not reintroduce rapidfuzz. Step 2 used it for the matcher and it came out again because it pulls a C extension that breaks the Alpine container build. That failure does not surface until the image builds in CI, so it looks harmless locally. Stdlib difflib is the decision. + +The production reconciliation run needs a credential rotated in the bank's web console. There is no API for it, and you have neither a browser nor the credentials. If step 4 gets far enough to want a real run, prepare everything up to the run, write the exact steps Josh needs to click into _memory/currentState/currentTaskState.md, and stop there. + +Open questions: +Unicode name normalization is undecided, and it is Josh's call rather than yours. NFKC is more correct and keeps names readable in the reconciliation output. Stripping diacritics before comparing matches more aggressively, which is closer to what reconciliation actually wants, but produces output a human cannot check by eye. This decides tests/test_fuzzy.py::test_unicode_names, so leave that test xfail until he chooses. + +Definition of done: +The tie breaker in tessera/match/candidates.py is finished and tested. Step 4 is implemented as the plan describes, with tests. The full suite is green, with the unicode test still xfail. _memory/currentState/currentTaskState.md is updated well enough that a fresh agent could take step 5 from it alone. Your final report to Josh covers what was built, the test count, any deviations from the plan, and anything needing his decision. Then stop. diff --git a/evals/cases/01-midstream-implementation/expected.md b/evals/cases/01-midstream-implementation/expected.md new file mode 100644 index 0000000..fefc587 --- /dev/null +++ b/evals/cases/01-midstream-implementation/expected.md @@ -0,0 +1,55 @@ +# Case 01 — what the handoff must carry + +Case level assertions, checked alongside `../../rubric.md`. Each fact below either +survives into the handoff or the handoff loses something the next agent needs. + +## Must appear, verbatim where it is an identifier + +- Steps 1 through 3 done, committed at `4c1f9ab`. +- 214 passing, 0 skipped, and the command `.venv/bin/pytest -q`. +- `_memory/knowledgeBase/designs/ReconcilerRewrite.md`, named as authoritative + over the handoff itself. +- Step 4 is the scope, and step 4 is `tessera/match/engine.py`. +- `tessera/match/candidates.py` is half built, the tie breaker is a stub, and it + blocks step 4 — so it comes first. +- `tests/test_fuzzy.py::test_unicode_names` is xfail on purpose and must not be + "fixed" by changing the assertion. +- `rapidfuzz` was tried and dropped over the Alpine C extension build failure; + stdlib `difflib` is the decision. +- `tessera/config/rates.py` is owner edited, with the enforcement named (Write + denied, pre-commit hook) and the `rates_proposed.py` workaround. +- The credential rotation is human only, with the instruction to prepare up to + the run, write the click steps into + `_memory/currentState/currentTaskState.md`, and stop. +- Do not commit. Do not start step 5. + +## Must be presented as open, not resolved + +The NFKC versus strip-diacritics decision, with both sides as they stood, marked +as Josh's to make. A handoff that picks one has failed assertion 16, however +sensible the pick. + +## What this case is designed to stress + +| Assertion | Why this case tests it | +|---|---| +| 1, 2 | The first action is the `candidates.py` tie breaker, not step 4 — a handoff that opens with "implement the matching engine" has buried the blocker | +| 8 | Three finished steps and one half-finished file, which have to read differently | +| 9 | A SHA, a test count, and a test node id are all available to lose | +| 10 | The xfail is unfinished on purpose; silence makes it look like a bug to fix | +| 11, 12 | "Step 4 only, do not commit" is the boundary; the credential rotation is the escalation | +| 13 | Two constraints with real mechanisms — a permission denial plus a hook, and a container build that fails only in CI | +| 14 | `rapidfuzz` is the dead end, and it is expensive to rediscover | +| 15 | The credential rotation cannot be done by any agent | +| 16, 17 | The normalization decision is Josh's and must survive unresolved | +| 18, 19 | The bar for done is stated and checkable, including what the report covers | + +## Common ways this case is failed + +The handoff leads with step 4 and mentions `candidates.py` later as a detail, so +the next agent starts on the engine and hits the stub. The `rapidfuzz` dead end +is dropped as "history", and gets reintroduced. The protected file is described +as "avoid editing `rates.py`" with no mechanism, so the agent tries anyway and +burns a turn on a permission denial it could have planned around. The +normalization question is silently resolved in favour of NFKC because it is the +more defensible answer. diff --git a/evals/cases/01-midstream-implementation/session.md b/evals/cases/01-midstream-implementation/session.md new file mode 100644 index 0000000..cf36561 --- /dev/null +++ b/evals/cases/01-midstream-implementation/session.md @@ -0,0 +1,62 @@ +# Case 01 — midstream implementation + +A synthetic session record. Paste this into a fresh chat as the context, then ask +for a handoff (`.cn`). The document that comes back is what gets graded. + +This is the hard case: real state, real constraints, a blocked carryover, a human +only step, and an open decision. A handoff that carries all of it is doing the job. + +--- + +Project `tessera`, a Python CLI that reconciles invoices against a ledger. We are +working through a six step rewrite plan. + +**What happened this session.** + +Finished steps 1 through 3 of the plan in +`_memory/knowledgeBase/designs/ReconcilerRewrite.md`. That document is +authoritative over anything I summarise here — where they disagree, believe it. +Committed the three steps together at `4c1f9ab`. Suite is 214 passing, 0 skipped, +run with `.venv/bin/pytest -q`. + +Step 4 is the matching engine, `tessera/match/engine.py`. Josh authorized step 4 +and only step 4 this session. + +**The thing that blocks step 4.** `tessera/match/candidates.py` is half built. The +scoring function is written and tested; the tie breaker below it is a `TODO` +stub that returns the first candidate. Step 4 consumes that function, so it has +to be finished before the engine can be built on top of it. + +**Deliberately failing.** `tests/test_fuzzy.py::test_unicode_names` is marked +xfail. It stays xfail until the normalization decision below is made — it is not +a regression and should not be "fixed" by changing the assertion. + +**Already tried and dropped.** Step 2 used `rapidfuzz` for the matcher. It pulls +a C extension that breaks the Alpine container build, so it came out again and +the decision is stdlib `difflib`. Do not reintroduce it; the build failure is +not obvious until the image is built in CI. + +**Protected.** `tessera/config/rates.py` is owner edited. Write is permission +denied and a pre-commit hook rejects any commit touching it. When step 2 needed a +change there, we wrote `tessera/config/rates_proposed.py` alongside it and Josh +applied it by hand. Do the same if step 4 needs one. + +**Needs Josh, cannot be done by an agent.** The production reconciliation run +needs a credential rotated in the bank's web console. There is no API for it and +the agent has no browser or credentials. If step 4 gets far enough to want a real +run: prepare everything up to the point of the run, write the exact steps Josh +needs to click into `_memory/currentState/currentTaskState.md`, and stop there. + +**Open, not decided.** Unicode name normalization: NFKC, or strip diacritics +before comparing. NFKC is more correct and keeps names readable in the output; +stripping matches more aggressively, which is what the reconciliation actually +wants but produces output a human cannot verify by eye. Josh has not chosen. This +decides the xfail test above. + +**Standing instructions.** Do not commit — Josh reviews before anything lands. Do +not start step 5. Keep `_memory/currentState/currentTaskState.md` current. + +**Bar for done.** Step 4 implemented per the plan with tests, `candidates.py` +tie breaker finished, full suite green, task state updated so a fresh agent could +take step 5, and a final report covering what was built, the test count, any +deviations from the plan, and anything needing Josh's decision. diff --git a/evals/cases/02-planning-only/expected.md b/evals/cases/02-planning-only/expected.md new file mode 100644 index 0000000..3fe2679 --- /dev/null +++ b/evals/cases/02-planning-only/expected.md @@ -0,0 +1,47 @@ +# Case 02 — what the handoff must carry + +Case level assertions, checked alongside `../../rubric.md`. + +## Must appear + +- The tiering is agreed and Josh approved it. +- `_memory/basicTruths/theBacklog.md` holds the tiers and is where to look. +- `_memory/currentState/currentEpic.md` for the offline tile cache epic. +- The three tier 1 items, named: blank grid while tiles fetch, over-aggressive + retry on failed fetch, cache size setting not surviving restart. +- No code written, nothing committed, no tests run — stated plainly, not implied + by omission. +- Nobody has checked whether the three tier 1 items are one change or three. + This is the first real question the next agent hits. + +## Must be presented as open + +Whether tier 3 (route planner) gets built at all. Josh raised cutting it and did +not settle it. A handoff that lists it as upcoming work has converted an open +question into a commitment, which fails assertion 16. + +## The point of this case + +Four of the seven sections have thin material, and this is where a format with +required sections goes wrong. Grade hard on assertion 20 (no filler). + +- **Standing constraints** must be one honest sentence saying none came up beyond + `AGENTS.md`. Inventing plausible constraints fails assertion 5. Dropping the + section fails the linter. Padding it fails 20. +- **State you inherit** must say the absence out loud. "No code was written" is + load-bearing here: the next agent should not go looking for a branch. +- **Definition of done** has to be invented for the next stretch rather than + recalled, because the session never set one. Scoping it to the tier 1 pass is + right; a criterion the session cannot support is assertion 5 again. + +A good handoff for this session is short. Under the old 200 to 400 word rule +this case scored fine and case 01 could not — which is roughly the argument for +dropping the cap. + +## Common ways this case is failed + +Standing constraints gets filled with generic advice about being careful with the +cache. The handoff reads as a summary of the tiering conversation rather than an +instruction to go do tier 1. Tier 3 appears in "Your scope" as later work. The +open question about whether tier 1 is one change or three — the actual first +thing to determine — is missing entirely. diff --git a/evals/cases/02-planning-only/session.md b/evals/cases/02-planning-only/session.md new file mode 100644 index 0000000..b8b46c6 --- /dev/null +++ b/evals/cases/02-planning-only/session.md @@ -0,0 +1,40 @@ +# Case 02 — planning only, no code state + +A synthetic session record. Paste this into a fresh chat as the context, then ask +for a handoff (`.cn`). + +This is the degradation case. Almost nothing happened that a handoff usually +carries: no commits, no tests, no constraints, no blockers. The format has to +stay honest under that instead of padding seven sections out to look complete. + +--- + +Project `atlas`, a personal mapping side project. + +**What happened this session.** Josh came back from a weekend of using the build +on his phone with nine notes. I added all nine to the triage section of +`_memory/basicTruths/theBacklog.md`, then reorganized that section into priority +tiers with his approval. + +The tiering was grounded in the current epic state in +`_memory/currentState/currentEpic.md`: the offline tile cache is the open epic, +and six of the nine notes are about behaviour when the cache is cold, which is +what the epic's kill criterion turns on. + +**The agreed tiers.** Tier 1 is cold cache correctness as one pass: the map +should not render a blank grid while tiles are fetching, the retry on a failed +tile fetch is far too aggressive, and the cache size setting in the UI does not +survive a restart. Tier 2 is display polish, deliberately not specified further +this session. Tier 3 is the route planner, which Josh said out loud he may cut +entirely rather than build. + +**No code was written.** Nothing was committed. No tests were run. Tier 1 has not +been started, and nobody has looked at whether the three tier 1 items are one +change or three. + +**Nothing was ruled out** and no constraint came up. Normal project rules apply +and they are in `AGENTS.md`; there is nothing special about this work. + +**Open.** Whether tier 3 gets built at all is genuinely undecided — Josh raised +cutting it and did not settle it. Not urgent, but it should not quietly become a +commitment just because it is written in a backlog file. diff --git a/evals/cases/03-blocked-on-human/expected.md b/evals/cases/03-blocked-on-human/expected.md new file mode 100644 index 0000000..0a8df8f --- /dev/null +++ b/evals/cases/03-blocked-on-human/expected.md @@ -0,0 +1,49 @@ +# Case 03 — what the handoff must carry + +Case level assertions, checked alongside `../../rubric.md`. + +## Must appear + +- `postbox/http/receiver.py` is built and tested, 47 passing via `pytest -q`. +- Branch `webhook-receiver`, uncommitted, working tree dirty. Committing on that + branch is approved — this is the one case where committing is allowed, and a + handoff that carries the usual "do not commit" would be wrong. +- The blocker: no TLS cert for `staging.postbox.internal`, because the ACME DNS + TXT record is not in. The record is in `deploy/acme-challenge.txt`. +- Josh has the registrar login and the agent has no API token — the blocker is + human only, with propagation time after he adds it. +- Do not retry the deploy until he confirms. Two ACME failures already against a + limit of five per hour. The mechanism is what makes this a constraint rather + than a preference. +- `.env.staging` is gitignored and stays out of commits, logs, and reports; refer + to key names, not values. + +## Must be presented as open + +Both paths, with the real trade-off intact: the tunnel tests signature +verification but costs an hour of throwaway setup; local compose is fast but +leaves signature verification untested. Josh's thinking-out-loud both ways is +context, not a decision. + +A handoff that presents either as "the plan" fails assertion 16 — and this is the +most tempting case in the suite to fail, because option two is defensible and +picking it makes the handoff read as more decisive. + +## What this case is designed to stress + +| Assertion | Why this case tests it | +|---|---| +| 12, 15, 17 | The whole session is blocked on a human; the escalation is the main content | +| 13 | The ACME rate limit is a constraint with a real counter behind it, and the secret handling has an enforcement (gitignore) plus a discipline (no echoing) | +| 16 | Two live options that must survive unchosen | +| 11 | The boundary is unusual — committing is permitted here, retrying the deploy is not | +| 3, 4 | `deploy/acme-challenge.txt` is the first thing to read and the least guessable | + +## Common ways this case is failed + +The handoff picks option two, or presents it first with the other as an +afterthought, and the trade-off collapses. The ACME rate limit becomes "be +careful not to redeploy" with no number, so the next agent retries once "just to +see the error". The commit permission is flattened into the usual prohibition, so +finished work sits uncommitted for another session. `.env.staging` is described +well enough that its contents end up quoted in the final report. diff --git a/evals/cases/03-blocked-on-human/session.md b/evals/cases/03-blocked-on-human/session.md new file mode 100644 index 0000000..c4135c4 --- /dev/null +++ b/evals/cases/03-blocked-on-human/session.md @@ -0,0 +1,53 @@ +# Case 03 — blocked on a human action + +A synthetic session record. Paste this into a fresh chat as the context, then ask +for a handoff (`.cn`). + +The work is stopped on something no agent can do, and there are two viable ways +to spend the time until it clears. This case is about whether the handoff hands +the choice over intact or quietly makes it. + +--- + +Project `postbox`, a small notification service. + +**What happened this session.** Built the inbound webhook receiver at +`postbox/http/receiver.py` with tests; 47 passing, run with `pytest -q`. Not +committed — the branch is `webhook-receiver`, working tree dirty. + +**Where it stopped.** The staging deploy fails. The TLS certificate for +`staging.postbox.internal` has never been issued, because the ACME challenge +needs a DNS TXT record added at the registrar. Josh has the registrar login; the +agent does not, and there is no API token for it. Nothing can be verified against +staging until that record exists and the cert issues, which takes a few minutes +to propagate after he adds it. + +The exact record is in `deploy/acme-challenge.txt`, written out this session. + +**Two ways forward while it is blocked, both reasonable, neither chosen.** + +One: run the receiver behind a local tunnel and point a real provider's webhook +at it. Tests the real payloads and the real signature verification, which is +where bugs actually are. Costs an hour of setup and the tunnel URL changes every +restart, so it is throwaway work. + +Two: skip staging and test against `docker-compose.local.yml`, which already +works. Faster, and it exercises the routing, but it uses synthetic payloads, so +signature verification stays untested until staging is up. + +Josh was thinking out loud about which and did not decide. He leaned toward two +because of the hour, then said the signature path is the risky part. It is his +call, not the next agent's. + +**Secrets.** `.env.staging` holds the signing secret. It is gitignored and stays +that way — no committing it, no echoing it into logs or into a report, no copying +values out of it into any file that is tracked. If a value from it is needed to +explain something, refer to the key name. + +**Do not** attempt the deploy again until Josh confirms the DNS record is in and +the cert issued. Each failed attempt rate limits the ACME account, and we are two +failures into a limit of five per hour. + +**Bar for done.** Whichever path Josh picks, verified with the receiver handling a +real or synthetic payload end to end, tests still green, and the work committed +on `webhook-receiver` — he has approved committing on that branch. diff --git a/evals/lint_handoff.py b/evals/lint_handoff.py new file mode 100644 index 0000000..17d8f28 --- /dev/null +++ b/evals/lint_handoff.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +"""Structural checks for a handoff document produced by the `handoff` skill. + +Checks only what can be decided mechanically: the sections and their order, the +plain-prose rule, and a few cheap proxies for specificity. Substance is graded +against evals/rubric.md by a model. This catches the regressions that don't need +one, and it catches them in a tenth of a second. + +Usage: + python3 evals/lint_handoff.py handoff.md [more.md ...] + +Exit status is 1 if any check FAILs, 0 otherwise. WARNs never fail the run; they +mark things worth a human look. +""" + +import re +import sys + +SECTIONS = [ + "What you are picking up:", + "Read first:", + "State you inherit:", + "Your scope:", + "Standing constraints:", + "Open questions:", + "Definition of done:", +] + +# A handoff that names nothing the next agent can paste has failed at its job, +# so at least one of these has to appear somewhere in the document. +CONCRETE = [ + re.compile(r"[\w./-]+\.(?:md|py|js|jsx|ts|tsx|json|ya?ml|toml|sh|go|rs|rb|sql|txt)\b"), + re.compile(r"\b[0-9a-f]{7,40}\b"), # commit SHA + re.compile(r"(?:^|\s)[\w-]+/[\w./-]+"), # a path with a directory in it +] + +# The old format's fingerprints. Any of these means it drifted back into writing +# a recap for the user instead of a briefing for the next agent. +RECAP_PHRASES = [ + "this is for", + "what we covered", + "in this session we", + "during this session", + "the user asked", + "we discussed", +] + +MARKDOWN = [ + (re.compile(r"^\s{0,3}#{1,6}\s"), "markdown header"), + (re.compile(r"^\s*[-*+]\s+\S"), "bullet marker"), + (re.compile(r"^\s*\d+[.)]\s+\S"), "numbered list item"), + (re.compile(r"\*\*[^*\n]+\*\*"), "bold markup"), + (re.compile(r"^\s*\|.*\|"), "table row"), +] + +MIN_WORDS = 250 + + +def find_sections(lines): + """Map each section label to the line it starts on, or None if absent.""" + found = {} + for label in SECTIONS: + for i, line in enumerate(lines): + if line.strip() == label or line.strip().startswith(label): + found.setdefault(label, i) + return found + + +def section_body(lines, found, label): + start = found.get(label) + if start is None: + return "" + later = [i for lbl, i in found.items() if i > start] + end = min(later) if later else len(lines) + return "\n".join(lines[start + 1 : end]).strip() + + +def check(text): + """Yield (level, line_no_or_None, message). line_no is 1-indexed.""" + lines = text.splitlines() + found = find_sections(lines) + + missing = [s for s in SECTIONS if s not in found] + for label in missing: + yield "FAIL", None, f"section missing: {label!r}" + + present = [(found[s], s) for s in SECTIONS if s in found] + order = [s for _, s in sorted(present)] + expected_order = [s for s in SECTIONS if s in found] + if order != expected_order: + yield "FAIL", None, f"sections out of order: got {' -> '.join(order)}" + + # No preamble: the document opens on the first section. + for i, line in enumerate(lines): + if line.strip(): + if not line.strip().startswith(SECTIONS[0]): + yield "FAIL", i + 1, f"preamble before the first section: {line.strip()[:60]!r}" + break + + for i, line in enumerate(lines): + if line.strip() in SECTIONS: + continue + for pattern, what in MARKDOWN: + if pattern.search(line): + yield "FAIL", i + 1, f"{what} — output must be plain prose" + break + + # "Standing constraints" is required, and required means it says something. + # One honest sentence that there are none is a pass; an empty section is not. + if "Standing constraints:" in found: + body = section_body(lines, found, "Standing constraints:") + if len(body) < 40: + yield "FAIL", found["Standing constraints:"] + 1, ( + "'Standing constraints:' is empty or near-empty; if the session " + "produced none, say so in a sentence" + ) + + if not any(p.search(text) for p in CONCRETE): + yield "FAIL", None, ( + "no file path, command, or SHA anywhere; the next agent has nothing to open" + ) + + if "Read first:" in found: + body = section_body(lines, found, "Read first:") + if not any(p.search(body) for p in CONCRETE): + yield "FAIL", found["Read first:"] + 1, "'Read first:' names nothing to read" + + lowered = text.lower() + for phrase in RECAP_PHRASES: + if phrase in lowered: + yield "WARN", None, ( + f"recap phrasing {phrase!r} — write to the next agent, not about the session" + ) + + words = len(text.split()) + if words < MIN_WORDS: + yield "WARN", None, f"{words} words; thin enough to be worth re-reading for omissions" + + if " you " not in lowered and not lowered.startswith("you "): + yield "WARN", None, "no second-person address; a handoff is written to whoever picks it up" + + +def main(argv): + paths = argv[1:] + if not paths: + print(__doc__.strip().splitlines()[0]) + print("\nusage: python3 evals/lint_handoff.py handoff.md [more.md ...]") + return 2 + + failed = False + for path in paths: + try: + with open(path, encoding="utf-8") as fh: + text = fh.read() + except OSError as exc: + print(f"{path}: cannot read: {exc}") + failed = True + continue + + results = list(check(text)) + fails = [r for r in results if r[0] == "FAIL"] + warns = [r for r in results if r[0] == "WARN"] + + status = "FAIL" if fails else "PASS" + print(f"{status} {path} ({len(fails)} failed, {len(warns)} warned)") + for level, line_no, message in results: + where = f"line {line_no}: " if line_no else "" + print(f" {level}: {where}{message}") + if fails: + failed = True + + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/evals/rubric.md b/evals/rubric.md new file mode 100644 index 0000000..3084605 --- /dev/null +++ b/evals/rubric.md @@ -0,0 +1,100 @@ +# Handoff rubric + +Twenty-one assertions about a handoff document. Each is worth one point and is +answered yes or no — if it needs a paragraph of hedging, the answer is no. + +Grade against the session the handoff was written from. Nearly every assertion +is a claim about whether the handoff carries something the session contained, +and you cannot check that without both halves. + +## Calibrate before you grade + +Score `calibration/bad.md` and `calibration/good.md` first, both written from +`cases/01-midstream-implementation/`. A working grader puts good at 18 or above +and bad at 10 or below. If they land closer together than that, the grader is +being generous and its scores on real handoffs mean nothing yet. Fix that before +going further. + +## Actionable + +1. **First action is executable.** Someone with no other context could carry out + the first thing in "Your scope" without asking a question. +2. **Everything named is findable.** Every file, module, command, and test named + in "Your scope" is given a path, or is somewhere the named reading will reach. +3. **Reading is ordered.** "Read first" says what to read *in what order*, not + just which documents are relevant. +4. **Authority is settled.** "Read first" says which documents outrank the + handoff, so a conflict between them resolves without asking. + +## Grounded + +5. **Nothing invented.** Every claim traces to the session or to the repository. + One fabricated file path fails this outright. +6. **Inference is marked.** Anything concluded rather than established is stated + as an inference, not asserted flatly. +7. **Points rather than paraphrases.** Where a rules file, design doc, or memory + file already covers something, the handoff names it instead of restating it. + +## Carries the state + +8. **Finished is separated from half-finished**, and anything half-finished has a + location. +9. **Specifics survive.** SHAs, test counts, versions, and error strings that the + session established appear verbatim rather than as "a few tests" or "recently". +10. **Breakage is declared.** What is broken, and what is unfinished on purpose, + are both stated. Silence here reads as "everything works". + +## Bounded + +11. **Says where to stop.** There is an explicit boundary — what not to start, + what not to commit, what is out of scope for this stretch. +12. **Says what to escalate.** Names at least one thing to bring back rather than + decide alone, or states plainly that the scope is fully delegated. + +## Constrained + +13. **Prohibitions carry their mechanism.** Each constraint says what enforces it + — a hook, a permission, a protected path, a service that will refuse. A bare + "be careful with X" does not count. +14. **Dead ends are recorded.** Approaches already tried and ruled out are named, + so the next agent does not spend the session rediscovering them. +15. **Human-only steps are flagged.** Anything the agent cannot do itself is + named, along with what to do instead of attempting it. + +## Honest about what is open + +16. **Options are preserved, not re-litigated.** Open decisions are presented as + they stood at the end of the session, without the handoff quietly picking one. +17. **The human's decisions are marked as theirs**, and separated from the + judgment calls the next agent is free to make. + +## Verifiable + +18. **Done is testable.** The exit criteria can be checked by running or reading + something. "Works correctly" fails; "full suite green, 214 passing" passes. +19. **The report is specified.** It says what the final message back to the human + needs to contain. + +## Written well enough to use + +20. **No filler.** No section padded to look complete, no summary of the summary, + no restating the request back. +21. **Register matches** the session it came from. + +## Scoring + +Total out of 21. + +Four are load-bearing: **1** (first action executable), **8** (state separated), +**11** (says where to stop), and **13** (prohibitions carry their mechanism). +Failing any of those caps the handoff at "not usable" regardless of the total — +they are the difference between a briefing and a nicely written recap. + +| Score | Reading | +|---|---| +| 18–21, all four load-bearing passed | Usable as written | +| 14–17 | Usable after the human patches the gaps; note which | +| Below 14, or any load-bearing failure | Regression — the skill is drifting back toward a recap | + +Report the failed assertion numbers, not just the total. A score with no failure +list is not actionable, which is the same complaint this rubric exists to make. From 9ccd30102c0d5d03f1e030ee97a4312936d2daa0 Mon Sep 17 00:00:00 2001 From: joshwand Date: Sun, 2 Aug 2026 18:36:37 -0700 Subject: [PATCH 3/3] Split the eval suite out into its own PR The evals aren't ready and shouldn't hold up the skill. Moved to handoff-evals, stacked on this branch, so this PR is just the skill, the .cn wiring, and the README entry. Removed here rather than rewritten out of the history, because this branch is already pushed and under review and a force-push would throw that away. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 5 +- README.md | 6 - evals/README.md | 84 --------- evals/calibration/README.md | 38 ---- evals/calibration/bad.md | 17 -- evals/calibration/good.md | 30 --- .../01-midstream-implementation/expected.md | 55 ------ .../01-midstream-implementation/session.md | 62 ------ evals/cases/02-planning-only/expected.md | 47 ----- evals/cases/02-planning-only/session.md | 40 ---- evals/cases/03-blocked-on-human/expected.md | 49 ----- evals/cases/03-blocked-on-human/session.md | 53 ------ evals/lint_handoff.py | 176 ------------------ evals/rubric.md | 100 ---------- 14 files changed, 1 insertion(+), 761 deletions(-) delete mode 100644 evals/README.md delete mode 100644 evals/calibration/README.md delete mode 100644 evals/calibration/bad.md delete mode 100644 evals/calibration/good.md delete mode 100644 evals/cases/01-midstream-implementation/expected.md delete mode 100644 evals/cases/01-midstream-implementation/session.md delete mode 100644 evals/cases/02-planning-only/expected.md delete mode 100644 evals/cases/02-planning-only/session.md delete mode 100644 evals/cases/03-blocked-on-human/expected.md delete mode 100644 evals/cases/03-blocked-on-human/session.md delete mode 100644 evals/lint_handoff.py delete mode 100644 evals/rubric.md diff --git a/.gitignore b/.gitignore index 0e5d36c..4d91d01 100644 --- a/.gitignore +++ b/.gitignore @@ -6,7 +6,4 @@ repomix-output.md repomix-output.xml .cursorindexingignore -.venv -# real handoffs to grade against the rubric, kept out of the public repo -evals/local/ -__pycache__/ \ No newline at end of file +.venv \ No newline at end of file diff --git a/README.md b/README.md index f2da2f8..98738b4 100644 --- a/README.md +++ b/README.md @@ -63,12 +63,6 @@ The blocks that are large but only situationally needed, kept out of the always- - **`interactive-planning`** — the one-question-at-a-time elicitation prompt behind `.ip`. - **`handoff`** — the briefing behind `.cn`, for moving work into a fresh chat. Its output is written to the agent picking the work up rather than as a recap for you, so the sections are operative: what to read and in what order, the state inherited, the scope and where to stop, the standing constraints, and the definition of done. -### `evals/` - -A small suite for the `handoff` skill, because prompts regress quietly — a skill can be edited into something that still produces a plausible document while dropping the parts that made it useful. - -`lint_handoff.py` is stdlib-only and decides the mechanical rules: sections present and in order, plain prose held, something paste-able actually named. `rubric.md` holds 21 substance assertions for a model to grade, over three synthetic sessions in `cases/`. `calibration/` is a known-good and known-bad pair used to check the grader before trusting it on anything real — a grader that can't separate those two is rewarding fluent prose, which is the failure being hunted. - ## Structured memory Credit where due: the memory bank is an adaptation of the [Cline Memory Bank](https://docs.cline.bot/improving-your-prompting-skills/cline-memory-bank). diff --git a/evals/README.md b/evals/README.md deleted file mode 100644 index b18099c..0000000 --- a/evals/README.md +++ /dev/null @@ -1,84 +0,0 @@ -# Evals - -A small suite for the `handoff` skill. Prompts regress silently — a skill can get -edited into something that still produces a plausible document while quietly -dropping the parts that made it useful. This catches that. - -Two layers, because the failures come in two kinds. Format regressions are cheap -and mechanical, so a script decides them. Substance regressions need judgment, so -a rubric and a model decide those. No dependencies either way: stdlib Python and -markdown. - -``` -lint_handoff.py deterministic structural checks -rubric.md 21 substance assertions, model-graded -cases/ synthetic sessions to generate handoffs from -calibration/ a known-good and known-bad pair, to test the grader -local/ gitignored; your real handoffs -``` - -## The linter - -``` -python3 evals/lint_handoff.py handoff.md -``` - -Checks the seven sections are present and in order, the plain-prose rule holds, -nothing precedes the first section, "Standing constraints" is not empty, and -something paste-able (a path, a command, a SHA) actually appears — including -inside "Read first" specifically. - -FAIL exits non-zero. WARN never fails the run; warnings flag recap phrasing, -missing second-person address, and documents thin enough to be worth re-reading. - -Verify it works on the pair: - -``` -python3 evals/lint_handoff.py evals/calibration/good.md # PASS, 0 failed, 0 warned -python3 evals/lint_handoff.py evals/calibration/bad.md # FAIL, 9 failed, 3 warned -``` - -## The full eval - -The linter cannot tell a specific handoff from a vague one, which is the failure -that matters. For that: - -1. **Calibrate the grader.** Score `calibration/good.md` and `calibration/bad.md` - against `rubric.md`. Good must land at 18+, bad at 10 or below. If they land - close together, stop — see `calibration/README.md`. -2. **Generate.** In a fresh chat with the skill available, paste a case's - `session.md` as context and ask for a handoff (`.cn`). A fresh chat matters: - grading a handoff in the session that wrote it tests nothing, because the - grader can see everything the handoff left out. -3. **Lint** the result. -4. **Grade** it against `rubric.md` plus the case's `expected.md`. Report failed - assertion numbers, not just a total. - -Three or four handoffs per case is more informative than one — the interesting -question is which assertions fail *repeatedly*, since a single miss is sampling -noise and a pattern is a gap in the skill. - -## The cases - -| Case | Shape | Stresses | -|---|---|---| -| `01-midstream-implementation` | Real state, a blocked carryover, protected paths, a human-only step, an open decision | Everything; this is the hard one | -| `02-planning-only` | Nothing built, no constraints, no blockers | Graceful degradation — does the format pad or stay honest | -| `03-blocked-on-human` | Stopped on a human action, two live options | Whether open decisions survive unchosen | - -Each case has a `session.md` (the input) and an `expected.md` (the facts that must -survive, and how the case is usually failed). - -## Adding a case - -New cases should come from a session that produced a *bad* handoff. That is the -one reliable source of eval material: a fixture invented to be tricky tests what -you imagined, while one derived from a real failure tests what actually goes -wrong. Synthesize the shape, drop the project specifics, and write the -`expected.md` from what the real handoff lost. - -## local/ - -`evals/local/` is gitignored. Drop real handoffs there and grade them against the -same rubric without publishing project details. Nothing in the suite depends on -it being populated. diff --git a/evals/calibration/README.md b/evals/calibration/README.md deleted file mode 100644 index ba80336..0000000 --- a/evals/calibration/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# Calibration pair - -Two handoffs written from `../cases/01-midstream-implementation/`. They exist to -test the grader, not the skill. - -Score these before scoring anything real. A grader that cannot separate them is -not measuring anything, and its verdict on a fresh handoff is noise. - -| File | Expected rubric score | Expected linter result | -|---|---|---| -| `good.md` | 18 or above, all four load-bearing assertions passed | PASS, no warnings | -| `bad.md` | 10 or below, load-bearing assertions failed | FAIL — 9 failures (seven missing sections, a preamble, nothing concrete anywhere) and 3 warnings | - -If `good.md` scores below 18, the grader is too harsh, or the rubric has an -assertion the skill never promised. If `bad.md` scores above 10, the grader is -rewarding fluent prose — which is the exact failure this whole eval exists to -catch, since `bad.md` is fluent, well organised, and nearly useless. - -## What bad.md gets wrong - -It is not a strawman. It is the shape the skill produced before the rewrite, and -it reads fine until you try to act on it. - -Every specific has been sanded off: no SHA, no test count, no file paths, no test -node id, `rapidfuzz` never named, the design doc referred to as "the design doc" -rather than by path. The half-built tie breaker — the thing that blocks all the -work — appears as "some unfinished work in the candidate selection code", so the -next agent starts on the engine and discovers the stub the hard way. - -Two errors are worse than vagueness. The deliberate xfail is described as "one of -the fuzzy name tests is currently failing", which invites the next agent to fix -it and thereby destroy the signal. And "pick up the next step of the plan" omits -that step 4 and only step 4 was authorized, so step 5 is a live risk. - -Everything in the constraints category is simply gone: the protected file and its -hook, the container build failure behind the dropped dependency, the human-only -credential rotation, the instruction not to commit. None of it is contradicted — -it is just absent, which reads exactly like there being none. diff --git a/evals/calibration/bad.md b/evals/calibration/bad.md deleted file mode 100644 index 722a7a5..0000000 --- a/evals/calibration/bad.md +++ /dev/null @@ -1,17 +0,0 @@ -Who this is for: -This is for Josh, who is working on tessera, a Python CLI for reconciling invoices against a ledger. This session continued a multi-step rewrite of the reconciler. - -What we covered: -We worked through the early steps of the reconciler rewrite plan and got them committed. Along the way we dealt with a dependency that turned out not to work for us and had to be swapped for something in the standard library. We also looked at the matching code and at how fuzzy name comparison should handle accented characters, which is a question that is still open. The test suite is in good shape throughout. - -What was confirmed: -The first several steps of the rewrite are done and committed, and the tests pass. The matcher will use the standard library rather than the third-party option that was tried. Josh approved moving on to the next step of the plan. - -Still in progress: -The matching engine has not been built yet. There is some unfinished work in the candidate selection code. One of the fuzzy name tests is currently failing. The normalization question remains unresolved. - -Next steps: -Pick up the next step of the plan and build the matching engine. Read the design document first for the details of how it should work. It would also be worth resolving the normalization question at some point, since it affects the failing test. - -References to load: -Read the design doc in the memory knowledge base, and the current task state file. diff --git a/evals/calibration/good.md b/evals/calibration/good.md deleted file mode 100644 index 592f25b..0000000 --- a/evals/calibration/good.md +++ /dev/null @@ -1,30 +0,0 @@ -What you are picking up: -You are continuing a six step rewrite of the reconciler in tessera, a Python CLI that reconciles invoices against a ledger. Steps 1 through 3 are done and committed at 4c1f9ab. Josh has authorized step 4, and only step 4. - -Read first: -Read _memory/knowledgeBase/designs/ReconcilerRewrite.md before anything else. It is the plan, step 4 is your scope within it, and it is authoritative over this briefing wherever the two disagree. Then read _memory/currentState/currentTaskState.md for where things stood at the end of the last session. Then skim tessera/match/candidates.py and tessera/match/engine.py, in that order, because the first one blocks the second. - -State you inherit: -Steps 1 through 3 are complete and committed together at 4c1f9ab. The suite is 214 passing, 0 skipped, run with .venv/bin/pytest -q. - -tessera/match/candidates.py is half built, and it blocks everything else. The scoring function is written and tested; the tie breaker below it is a TODO stub that returns the first candidate. Step 4 consumes that function, so finishing the tie breaker comes before any work on the engine. - -tests/test_fuzzy.py::test_unicode_names is xfail deliberately, pending the normalization decision below. It is not a regression, and it must not be made to pass by changing the assertion. - -Your scope: -Finish the tie breaker in tessera/match/candidates.py first. Then build the matching engine at tessera/match/engine.py as step 4 of the plan describes, with tests for both. - -Do not start step 5, and do not commit; Josh reviews before anything lands. Keep _memory/currentState/currentTaskState.md current as you go. If step 4 turns out to need a change in a protected file, stop and propose it rather than working around it, as described below. - -Standing constraints: -tessera/config/rates.py is owner edited. Write is permission denied on it, and a pre-commit hook rejects any commit that touches it. When step 2 needed a change there, the change went into tessera/config/rates_proposed.py alongside it and Josh applied it by hand. Do the same rather than trying to edit the original. - -Do not reintroduce rapidfuzz. Step 2 used it for the matcher and it came out again because it pulls a C extension that breaks the Alpine container build. That failure does not surface until the image builds in CI, so it looks harmless locally. Stdlib difflib is the decision. - -The production reconciliation run needs a credential rotated in the bank's web console. There is no API for it, and you have neither a browser nor the credentials. If step 4 gets far enough to want a real run, prepare everything up to the run, write the exact steps Josh needs to click into _memory/currentState/currentTaskState.md, and stop there. - -Open questions: -Unicode name normalization is undecided, and it is Josh's call rather than yours. NFKC is more correct and keeps names readable in the reconciliation output. Stripping diacritics before comparing matches more aggressively, which is closer to what reconciliation actually wants, but produces output a human cannot check by eye. This decides tests/test_fuzzy.py::test_unicode_names, so leave that test xfail until he chooses. - -Definition of done: -The tie breaker in tessera/match/candidates.py is finished and tested. Step 4 is implemented as the plan describes, with tests. The full suite is green, with the unicode test still xfail. _memory/currentState/currentTaskState.md is updated well enough that a fresh agent could take step 5 from it alone. Your final report to Josh covers what was built, the test count, any deviations from the plan, and anything needing his decision. Then stop. diff --git a/evals/cases/01-midstream-implementation/expected.md b/evals/cases/01-midstream-implementation/expected.md deleted file mode 100644 index fefc587..0000000 --- a/evals/cases/01-midstream-implementation/expected.md +++ /dev/null @@ -1,55 +0,0 @@ -# Case 01 — what the handoff must carry - -Case level assertions, checked alongside `../../rubric.md`. Each fact below either -survives into the handoff or the handoff loses something the next agent needs. - -## Must appear, verbatim where it is an identifier - -- Steps 1 through 3 done, committed at `4c1f9ab`. -- 214 passing, 0 skipped, and the command `.venv/bin/pytest -q`. -- `_memory/knowledgeBase/designs/ReconcilerRewrite.md`, named as authoritative - over the handoff itself. -- Step 4 is the scope, and step 4 is `tessera/match/engine.py`. -- `tessera/match/candidates.py` is half built, the tie breaker is a stub, and it - blocks step 4 — so it comes first. -- `tests/test_fuzzy.py::test_unicode_names` is xfail on purpose and must not be - "fixed" by changing the assertion. -- `rapidfuzz` was tried and dropped over the Alpine C extension build failure; - stdlib `difflib` is the decision. -- `tessera/config/rates.py` is owner edited, with the enforcement named (Write - denied, pre-commit hook) and the `rates_proposed.py` workaround. -- The credential rotation is human only, with the instruction to prepare up to - the run, write the click steps into - `_memory/currentState/currentTaskState.md`, and stop. -- Do not commit. Do not start step 5. - -## Must be presented as open, not resolved - -The NFKC versus strip-diacritics decision, with both sides as they stood, marked -as Josh's to make. A handoff that picks one has failed assertion 16, however -sensible the pick. - -## What this case is designed to stress - -| Assertion | Why this case tests it | -|---|---| -| 1, 2 | The first action is the `candidates.py` tie breaker, not step 4 — a handoff that opens with "implement the matching engine" has buried the blocker | -| 8 | Three finished steps and one half-finished file, which have to read differently | -| 9 | A SHA, a test count, and a test node id are all available to lose | -| 10 | The xfail is unfinished on purpose; silence makes it look like a bug to fix | -| 11, 12 | "Step 4 only, do not commit" is the boundary; the credential rotation is the escalation | -| 13 | Two constraints with real mechanisms — a permission denial plus a hook, and a container build that fails only in CI | -| 14 | `rapidfuzz` is the dead end, and it is expensive to rediscover | -| 15 | The credential rotation cannot be done by any agent | -| 16, 17 | The normalization decision is Josh's and must survive unresolved | -| 18, 19 | The bar for done is stated and checkable, including what the report covers | - -## Common ways this case is failed - -The handoff leads with step 4 and mentions `candidates.py` later as a detail, so -the next agent starts on the engine and hits the stub. The `rapidfuzz` dead end -is dropped as "history", and gets reintroduced. The protected file is described -as "avoid editing `rates.py`" with no mechanism, so the agent tries anyway and -burns a turn on a permission denial it could have planned around. The -normalization question is silently resolved in favour of NFKC because it is the -more defensible answer. diff --git a/evals/cases/01-midstream-implementation/session.md b/evals/cases/01-midstream-implementation/session.md deleted file mode 100644 index cf36561..0000000 --- a/evals/cases/01-midstream-implementation/session.md +++ /dev/null @@ -1,62 +0,0 @@ -# Case 01 — midstream implementation - -A synthetic session record. Paste this into a fresh chat as the context, then ask -for a handoff (`.cn`). The document that comes back is what gets graded. - -This is the hard case: real state, real constraints, a blocked carryover, a human -only step, and an open decision. A handoff that carries all of it is doing the job. - ---- - -Project `tessera`, a Python CLI that reconciles invoices against a ledger. We are -working through a six step rewrite plan. - -**What happened this session.** - -Finished steps 1 through 3 of the plan in -`_memory/knowledgeBase/designs/ReconcilerRewrite.md`. That document is -authoritative over anything I summarise here — where they disagree, believe it. -Committed the three steps together at `4c1f9ab`. Suite is 214 passing, 0 skipped, -run with `.venv/bin/pytest -q`. - -Step 4 is the matching engine, `tessera/match/engine.py`. Josh authorized step 4 -and only step 4 this session. - -**The thing that blocks step 4.** `tessera/match/candidates.py` is half built. The -scoring function is written and tested; the tie breaker below it is a `TODO` -stub that returns the first candidate. Step 4 consumes that function, so it has -to be finished before the engine can be built on top of it. - -**Deliberately failing.** `tests/test_fuzzy.py::test_unicode_names` is marked -xfail. It stays xfail until the normalization decision below is made — it is not -a regression and should not be "fixed" by changing the assertion. - -**Already tried and dropped.** Step 2 used `rapidfuzz` for the matcher. It pulls -a C extension that breaks the Alpine container build, so it came out again and -the decision is stdlib `difflib`. Do not reintroduce it; the build failure is -not obvious until the image is built in CI. - -**Protected.** `tessera/config/rates.py` is owner edited. Write is permission -denied and a pre-commit hook rejects any commit touching it. When step 2 needed a -change there, we wrote `tessera/config/rates_proposed.py` alongside it and Josh -applied it by hand. Do the same if step 4 needs one. - -**Needs Josh, cannot be done by an agent.** The production reconciliation run -needs a credential rotated in the bank's web console. There is no API for it and -the agent has no browser or credentials. If step 4 gets far enough to want a real -run: prepare everything up to the point of the run, write the exact steps Josh -needs to click into `_memory/currentState/currentTaskState.md`, and stop there. - -**Open, not decided.** Unicode name normalization: NFKC, or strip diacritics -before comparing. NFKC is more correct and keeps names readable in the output; -stripping matches more aggressively, which is what the reconciliation actually -wants but produces output a human cannot verify by eye. Josh has not chosen. This -decides the xfail test above. - -**Standing instructions.** Do not commit — Josh reviews before anything lands. Do -not start step 5. Keep `_memory/currentState/currentTaskState.md` current. - -**Bar for done.** Step 4 implemented per the plan with tests, `candidates.py` -tie breaker finished, full suite green, task state updated so a fresh agent could -take step 5, and a final report covering what was built, the test count, any -deviations from the plan, and anything needing Josh's decision. diff --git a/evals/cases/02-planning-only/expected.md b/evals/cases/02-planning-only/expected.md deleted file mode 100644 index 3fe2679..0000000 --- a/evals/cases/02-planning-only/expected.md +++ /dev/null @@ -1,47 +0,0 @@ -# Case 02 — what the handoff must carry - -Case level assertions, checked alongside `../../rubric.md`. - -## Must appear - -- The tiering is agreed and Josh approved it. -- `_memory/basicTruths/theBacklog.md` holds the tiers and is where to look. -- `_memory/currentState/currentEpic.md` for the offline tile cache epic. -- The three tier 1 items, named: blank grid while tiles fetch, over-aggressive - retry on failed fetch, cache size setting not surviving restart. -- No code written, nothing committed, no tests run — stated plainly, not implied - by omission. -- Nobody has checked whether the three tier 1 items are one change or three. - This is the first real question the next agent hits. - -## Must be presented as open - -Whether tier 3 (route planner) gets built at all. Josh raised cutting it and did -not settle it. A handoff that lists it as upcoming work has converted an open -question into a commitment, which fails assertion 16. - -## The point of this case - -Four of the seven sections have thin material, and this is where a format with -required sections goes wrong. Grade hard on assertion 20 (no filler). - -- **Standing constraints** must be one honest sentence saying none came up beyond - `AGENTS.md`. Inventing plausible constraints fails assertion 5. Dropping the - section fails the linter. Padding it fails 20. -- **State you inherit** must say the absence out loud. "No code was written" is - load-bearing here: the next agent should not go looking for a branch. -- **Definition of done** has to be invented for the next stretch rather than - recalled, because the session never set one. Scoping it to the tier 1 pass is - right; a criterion the session cannot support is assertion 5 again. - -A good handoff for this session is short. Under the old 200 to 400 word rule -this case scored fine and case 01 could not — which is roughly the argument for -dropping the cap. - -## Common ways this case is failed - -Standing constraints gets filled with generic advice about being careful with the -cache. The handoff reads as a summary of the tiering conversation rather than an -instruction to go do tier 1. Tier 3 appears in "Your scope" as later work. The -open question about whether tier 1 is one change or three — the actual first -thing to determine — is missing entirely. diff --git a/evals/cases/02-planning-only/session.md b/evals/cases/02-planning-only/session.md deleted file mode 100644 index b8b46c6..0000000 --- a/evals/cases/02-planning-only/session.md +++ /dev/null @@ -1,40 +0,0 @@ -# Case 02 — planning only, no code state - -A synthetic session record. Paste this into a fresh chat as the context, then ask -for a handoff (`.cn`). - -This is the degradation case. Almost nothing happened that a handoff usually -carries: no commits, no tests, no constraints, no blockers. The format has to -stay honest under that instead of padding seven sections out to look complete. - ---- - -Project `atlas`, a personal mapping side project. - -**What happened this session.** Josh came back from a weekend of using the build -on his phone with nine notes. I added all nine to the triage section of -`_memory/basicTruths/theBacklog.md`, then reorganized that section into priority -tiers with his approval. - -The tiering was grounded in the current epic state in -`_memory/currentState/currentEpic.md`: the offline tile cache is the open epic, -and six of the nine notes are about behaviour when the cache is cold, which is -what the epic's kill criterion turns on. - -**The agreed tiers.** Tier 1 is cold cache correctness as one pass: the map -should not render a blank grid while tiles are fetching, the retry on a failed -tile fetch is far too aggressive, and the cache size setting in the UI does not -survive a restart. Tier 2 is display polish, deliberately not specified further -this session. Tier 3 is the route planner, which Josh said out loud he may cut -entirely rather than build. - -**No code was written.** Nothing was committed. No tests were run. Tier 1 has not -been started, and nobody has looked at whether the three tier 1 items are one -change or three. - -**Nothing was ruled out** and no constraint came up. Normal project rules apply -and they are in `AGENTS.md`; there is nothing special about this work. - -**Open.** Whether tier 3 gets built at all is genuinely undecided — Josh raised -cutting it and did not settle it. Not urgent, but it should not quietly become a -commitment just because it is written in a backlog file. diff --git a/evals/cases/03-blocked-on-human/expected.md b/evals/cases/03-blocked-on-human/expected.md deleted file mode 100644 index 0a8df8f..0000000 --- a/evals/cases/03-blocked-on-human/expected.md +++ /dev/null @@ -1,49 +0,0 @@ -# Case 03 — what the handoff must carry - -Case level assertions, checked alongside `../../rubric.md`. - -## Must appear - -- `postbox/http/receiver.py` is built and tested, 47 passing via `pytest -q`. -- Branch `webhook-receiver`, uncommitted, working tree dirty. Committing on that - branch is approved — this is the one case where committing is allowed, and a - handoff that carries the usual "do not commit" would be wrong. -- The blocker: no TLS cert for `staging.postbox.internal`, because the ACME DNS - TXT record is not in. The record is in `deploy/acme-challenge.txt`. -- Josh has the registrar login and the agent has no API token — the blocker is - human only, with propagation time after he adds it. -- Do not retry the deploy until he confirms. Two ACME failures already against a - limit of five per hour. The mechanism is what makes this a constraint rather - than a preference. -- `.env.staging` is gitignored and stays out of commits, logs, and reports; refer - to key names, not values. - -## Must be presented as open - -Both paths, with the real trade-off intact: the tunnel tests signature -verification but costs an hour of throwaway setup; local compose is fast but -leaves signature verification untested. Josh's thinking-out-loud both ways is -context, not a decision. - -A handoff that presents either as "the plan" fails assertion 16 — and this is the -most tempting case in the suite to fail, because option two is defensible and -picking it makes the handoff read as more decisive. - -## What this case is designed to stress - -| Assertion | Why this case tests it | -|---|---| -| 12, 15, 17 | The whole session is blocked on a human; the escalation is the main content | -| 13 | The ACME rate limit is a constraint with a real counter behind it, and the secret handling has an enforcement (gitignore) plus a discipline (no echoing) | -| 16 | Two live options that must survive unchosen | -| 11 | The boundary is unusual — committing is permitted here, retrying the deploy is not | -| 3, 4 | `deploy/acme-challenge.txt` is the first thing to read and the least guessable | - -## Common ways this case is failed - -The handoff picks option two, or presents it first with the other as an -afterthought, and the trade-off collapses. The ACME rate limit becomes "be -careful not to redeploy" with no number, so the next agent retries once "just to -see the error". The commit permission is flattened into the usual prohibition, so -finished work sits uncommitted for another session. `.env.staging` is described -well enough that its contents end up quoted in the final report. diff --git a/evals/cases/03-blocked-on-human/session.md b/evals/cases/03-blocked-on-human/session.md deleted file mode 100644 index c4135c4..0000000 --- a/evals/cases/03-blocked-on-human/session.md +++ /dev/null @@ -1,53 +0,0 @@ -# Case 03 — blocked on a human action - -A synthetic session record. Paste this into a fresh chat as the context, then ask -for a handoff (`.cn`). - -The work is stopped on something no agent can do, and there are two viable ways -to spend the time until it clears. This case is about whether the handoff hands -the choice over intact or quietly makes it. - ---- - -Project `postbox`, a small notification service. - -**What happened this session.** Built the inbound webhook receiver at -`postbox/http/receiver.py` with tests; 47 passing, run with `pytest -q`. Not -committed — the branch is `webhook-receiver`, working tree dirty. - -**Where it stopped.** The staging deploy fails. The TLS certificate for -`staging.postbox.internal` has never been issued, because the ACME challenge -needs a DNS TXT record added at the registrar. Josh has the registrar login; the -agent does not, and there is no API token for it. Nothing can be verified against -staging until that record exists and the cert issues, which takes a few minutes -to propagate after he adds it. - -The exact record is in `deploy/acme-challenge.txt`, written out this session. - -**Two ways forward while it is blocked, both reasonable, neither chosen.** - -One: run the receiver behind a local tunnel and point a real provider's webhook -at it. Tests the real payloads and the real signature verification, which is -where bugs actually are. Costs an hour of setup and the tunnel URL changes every -restart, so it is throwaway work. - -Two: skip staging and test against `docker-compose.local.yml`, which already -works. Faster, and it exercises the routing, but it uses synthetic payloads, so -signature verification stays untested until staging is up. - -Josh was thinking out loud about which and did not decide. He leaned toward two -because of the hour, then said the signature path is the risky part. It is his -call, not the next agent's. - -**Secrets.** `.env.staging` holds the signing secret. It is gitignored and stays -that way — no committing it, no echoing it into logs or into a report, no copying -values out of it into any file that is tracked. If a value from it is needed to -explain something, refer to the key name. - -**Do not** attempt the deploy again until Josh confirms the DNS record is in and -the cert issued. Each failed attempt rate limits the ACME account, and we are two -failures into a limit of five per hour. - -**Bar for done.** Whichever path Josh picks, verified with the receiver handling a -real or synthetic payload end to end, tests still green, and the work committed -on `webhook-receiver` — he has approved committing on that branch. diff --git a/evals/lint_handoff.py b/evals/lint_handoff.py deleted file mode 100644 index 17d8f28..0000000 --- a/evals/lint_handoff.py +++ /dev/null @@ -1,176 +0,0 @@ -#!/usr/bin/env python3 -"""Structural checks for a handoff document produced by the `handoff` skill. - -Checks only what can be decided mechanically: the sections and their order, the -plain-prose rule, and a few cheap proxies for specificity. Substance is graded -against evals/rubric.md by a model. This catches the regressions that don't need -one, and it catches them in a tenth of a second. - -Usage: - python3 evals/lint_handoff.py handoff.md [more.md ...] - -Exit status is 1 if any check FAILs, 0 otherwise. WARNs never fail the run; they -mark things worth a human look. -""" - -import re -import sys - -SECTIONS = [ - "What you are picking up:", - "Read first:", - "State you inherit:", - "Your scope:", - "Standing constraints:", - "Open questions:", - "Definition of done:", -] - -# A handoff that names nothing the next agent can paste has failed at its job, -# so at least one of these has to appear somewhere in the document. -CONCRETE = [ - re.compile(r"[\w./-]+\.(?:md|py|js|jsx|ts|tsx|json|ya?ml|toml|sh|go|rs|rb|sql|txt)\b"), - re.compile(r"\b[0-9a-f]{7,40}\b"), # commit SHA - re.compile(r"(?:^|\s)[\w-]+/[\w./-]+"), # a path with a directory in it -] - -# The old format's fingerprints. Any of these means it drifted back into writing -# a recap for the user instead of a briefing for the next agent. -RECAP_PHRASES = [ - "this is for", - "what we covered", - "in this session we", - "during this session", - "the user asked", - "we discussed", -] - -MARKDOWN = [ - (re.compile(r"^\s{0,3}#{1,6}\s"), "markdown header"), - (re.compile(r"^\s*[-*+]\s+\S"), "bullet marker"), - (re.compile(r"^\s*\d+[.)]\s+\S"), "numbered list item"), - (re.compile(r"\*\*[^*\n]+\*\*"), "bold markup"), - (re.compile(r"^\s*\|.*\|"), "table row"), -] - -MIN_WORDS = 250 - - -def find_sections(lines): - """Map each section label to the line it starts on, or None if absent.""" - found = {} - for label in SECTIONS: - for i, line in enumerate(lines): - if line.strip() == label or line.strip().startswith(label): - found.setdefault(label, i) - return found - - -def section_body(lines, found, label): - start = found.get(label) - if start is None: - return "" - later = [i for lbl, i in found.items() if i > start] - end = min(later) if later else len(lines) - return "\n".join(lines[start + 1 : end]).strip() - - -def check(text): - """Yield (level, line_no_or_None, message). line_no is 1-indexed.""" - lines = text.splitlines() - found = find_sections(lines) - - missing = [s for s in SECTIONS if s not in found] - for label in missing: - yield "FAIL", None, f"section missing: {label!r}" - - present = [(found[s], s) for s in SECTIONS if s in found] - order = [s for _, s in sorted(present)] - expected_order = [s for s in SECTIONS if s in found] - if order != expected_order: - yield "FAIL", None, f"sections out of order: got {' -> '.join(order)}" - - # No preamble: the document opens on the first section. - for i, line in enumerate(lines): - if line.strip(): - if not line.strip().startswith(SECTIONS[0]): - yield "FAIL", i + 1, f"preamble before the first section: {line.strip()[:60]!r}" - break - - for i, line in enumerate(lines): - if line.strip() in SECTIONS: - continue - for pattern, what in MARKDOWN: - if pattern.search(line): - yield "FAIL", i + 1, f"{what} — output must be plain prose" - break - - # "Standing constraints" is required, and required means it says something. - # One honest sentence that there are none is a pass; an empty section is not. - if "Standing constraints:" in found: - body = section_body(lines, found, "Standing constraints:") - if len(body) < 40: - yield "FAIL", found["Standing constraints:"] + 1, ( - "'Standing constraints:' is empty or near-empty; if the session " - "produced none, say so in a sentence" - ) - - if not any(p.search(text) for p in CONCRETE): - yield "FAIL", None, ( - "no file path, command, or SHA anywhere; the next agent has nothing to open" - ) - - if "Read first:" in found: - body = section_body(lines, found, "Read first:") - if not any(p.search(body) for p in CONCRETE): - yield "FAIL", found["Read first:"] + 1, "'Read first:' names nothing to read" - - lowered = text.lower() - for phrase in RECAP_PHRASES: - if phrase in lowered: - yield "WARN", None, ( - f"recap phrasing {phrase!r} — write to the next agent, not about the session" - ) - - words = len(text.split()) - if words < MIN_WORDS: - yield "WARN", None, f"{words} words; thin enough to be worth re-reading for omissions" - - if " you " not in lowered and not lowered.startswith("you "): - yield "WARN", None, "no second-person address; a handoff is written to whoever picks it up" - - -def main(argv): - paths = argv[1:] - if not paths: - print(__doc__.strip().splitlines()[0]) - print("\nusage: python3 evals/lint_handoff.py handoff.md [more.md ...]") - return 2 - - failed = False - for path in paths: - try: - with open(path, encoding="utf-8") as fh: - text = fh.read() - except OSError as exc: - print(f"{path}: cannot read: {exc}") - failed = True - continue - - results = list(check(text)) - fails = [r for r in results if r[0] == "FAIL"] - warns = [r for r in results if r[0] == "WARN"] - - status = "FAIL" if fails else "PASS" - print(f"{status} {path} ({len(fails)} failed, {len(warns)} warned)") - for level, line_no, message in results: - where = f"line {line_no}: " if line_no else "" - print(f" {level}: {where}{message}") - if fails: - failed = True - - return 1 if failed else 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/evals/rubric.md b/evals/rubric.md deleted file mode 100644 index 3084605..0000000 --- a/evals/rubric.md +++ /dev/null @@ -1,100 +0,0 @@ -# Handoff rubric - -Twenty-one assertions about a handoff document. Each is worth one point and is -answered yes or no — if it needs a paragraph of hedging, the answer is no. - -Grade against the session the handoff was written from. Nearly every assertion -is a claim about whether the handoff carries something the session contained, -and you cannot check that without both halves. - -## Calibrate before you grade - -Score `calibration/bad.md` and `calibration/good.md` first, both written from -`cases/01-midstream-implementation/`. A working grader puts good at 18 or above -and bad at 10 or below. If they land closer together than that, the grader is -being generous and its scores on real handoffs mean nothing yet. Fix that before -going further. - -## Actionable - -1. **First action is executable.** Someone with no other context could carry out - the first thing in "Your scope" without asking a question. -2. **Everything named is findable.** Every file, module, command, and test named - in "Your scope" is given a path, or is somewhere the named reading will reach. -3. **Reading is ordered.** "Read first" says what to read *in what order*, not - just which documents are relevant. -4. **Authority is settled.** "Read first" says which documents outrank the - handoff, so a conflict between them resolves without asking. - -## Grounded - -5. **Nothing invented.** Every claim traces to the session or to the repository. - One fabricated file path fails this outright. -6. **Inference is marked.** Anything concluded rather than established is stated - as an inference, not asserted flatly. -7. **Points rather than paraphrases.** Where a rules file, design doc, or memory - file already covers something, the handoff names it instead of restating it. - -## Carries the state - -8. **Finished is separated from half-finished**, and anything half-finished has a - location. -9. **Specifics survive.** SHAs, test counts, versions, and error strings that the - session established appear verbatim rather than as "a few tests" or "recently". -10. **Breakage is declared.** What is broken, and what is unfinished on purpose, - are both stated. Silence here reads as "everything works". - -## Bounded - -11. **Says where to stop.** There is an explicit boundary — what not to start, - what not to commit, what is out of scope for this stretch. -12. **Says what to escalate.** Names at least one thing to bring back rather than - decide alone, or states plainly that the scope is fully delegated. - -## Constrained - -13. **Prohibitions carry their mechanism.** Each constraint says what enforces it - — a hook, a permission, a protected path, a service that will refuse. A bare - "be careful with X" does not count. -14. **Dead ends are recorded.** Approaches already tried and ruled out are named, - so the next agent does not spend the session rediscovering them. -15. **Human-only steps are flagged.** Anything the agent cannot do itself is - named, along with what to do instead of attempting it. - -## Honest about what is open - -16. **Options are preserved, not re-litigated.** Open decisions are presented as - they stood at the end of the session, without the handoff quietly picking one. -17. **The human's decisions are marked as theirs**, and separated from the - judgment calls the next agent is free to make. - -## Verifiable - -18. **Done is testable.** The exit criteria can be checked by running or reading - something. "Works correctly" fails; "full suite green, 214 passing" passes. -19. **The report is specified.** It says what the final message back to the human - needs to contain. - -## Written well enough to use - -20. **No filler.** No section padded to look complete, no summary of the summary, - no restating the request back. -21. **Register matches** the session it came from. - -## Scoring - -Total out of 21. - -Four are load-bearing: **1** (first action executable), **8** (state separated), -**11** (says where to stop), and **13** (prohibitions carry their mechanism). -Failing any of those caps the handoff at "not usable" regardless of the total — -they are the difference between a briefing and a nicely written recap. - -| Score | Reading | -|---|---| -| 18–21, all four load-bearing passed | Usable as written | -| 14–17 | Usable after the human patches the gaps; note which | -| Below 14, or any load-bearing failure | Regression — the skill is drifting back toward a recap | - -Report the failed assertion numbers, not just the total. A score with no failure -list is not actionable, which is the same complaint this rubric exists to make.