Skip to content

v3.3.0 — add test-and-verify, and have execute-plan delegate to it - #4

Merged
melconcoast merged 3 commits into
masterfrom
feature/test-and-verify-skill
Aug 23, 2026
Merged

v3.3.0 — add test-and-verify, and have execute-plan delegate to it#4
melconcoast merged 3 commits into
masterfrom
feature/test-and-verify-skill

Conversation

@melconcoast

Copy link
Copy Markdown
Owner

Stacked on #3 (which is stacked on #2). Merge in order; each base retargets automatically.

The first three skills are a pipeline. This one is a service the others call: execute-plan no longer runs tests itself — it hands each task and each Task X.V gate to test-and-verify and acts on the verdict.

The skill

It finds the project's test command, runs the tests relevant to the work, and reads the real output rather than the exit code — a suite that exits 0 having collected no tests, or having skipped the new ones, is a failure wearing a green hat.

On a failure it classifies the bug as application code or test before editing anything, because picking wrong either papers over a real defect or bends correct code to satisfy a broken assertion. One targeted fix per attempt. Three attempts, then it stops — past three the diagnosis is usually what was wrong rather than the fix, so further attempts widen the diff on a premise that was never true.

Two boundaries make it safe to call automatically:

  • It never writes to a plan file. It returns a verdict; execute-plan marks the boxes. Two writers on one plan file is how a plan stops being trustworthy.
  • It never weakens a test to reach green — no deleted assertions, loosened matchers, added skips, widened timeouts, or expected values rewritten to match whatever the code returns.

At a gate it widens to the whole module's suite plus the type-checker and linter, and says so explicitly when the project has neither, since an omitted line reads as a check that passed.

Refactor: test-command discovery has one owner

Discovery lived in execute-plan's verification.md and the new skill would have duplicated it. That file keeps only what is genuinely execute-plan's — bootstrapping a runner when a project has none, which picks tooling and edits a manifest and so is a project decision, not a remediation step — and its "what counts as green" section narrows to the question it actually owns: not whether the run passed, but whether that pass closes a checkbox.

Verification

Gates: four skills, names matching directories, descriptions at 857/861/994/1007 characters (limit 1024), 63/61/50/119 lines, plugin.json at 3.3.0, zero broken links.

Against the installed 3.3.0 plugin in fresh headless sessions:

  • S96 — the regression guard, and the main risk with a fourth description. All twelve phrases belonging to scaffold, plan-module, and execute-plan still reach the skill they always did. 12/12.
  • S95 — all four test-and-verify phrases fire it.
  • S85 / S91 — given a suite with one genuine bug (a <= where the rule says "not before"), it diagnosed application code, checked the claim against docs/product.md and the plan's own scenario, changed exactly one character in one file, and went green. It did not touch the test.
  • S86 — on an already-green suite it reported and changed nothing.
  • S89 — the integration. execute-plan fired, delegated to test-and-verify, took its verdict, and wrote the plan file itself. Both skills in one run, 10 tests green, real code built.
  • S92 — at the gate it ran the full suite and reported "The project has no type-checker and no linter… I'm not claiming checks that never ran."
  • S83 / S67 — the same run closed two gaps left open in v3.2.0 — add execute-plan, closing the scaffold → plan → build chain #3: the header Status: flipped to Done at [2/2 Phases Closed], and the roadmap write-back used bare vocabulary words throughout.

One defect the runs caught, fixed here

"run the tests" and "verify this" originally did the right thing but did it inline, without loading the skill — so the circuit breaker, the no-weakening rule, and the gate widening were never in play. Correct by luck, not construction. The request simply looks too small to warrant a skill. The description now says the discipline is the point rather than the command, and to use it even when the ask looks like a one-liner.

Known limitation, stated honestly: standalone triggering is now reliable when tool access is constrained, but under full permissions the model still sometimes runs a trivial suite inline rather than loading the skill. The path that matters — execute-plan delegating — is driven by an explicit instruction in its Step 3 rather than description matching, and that fired reliably.

🤖 Generated with Claude Code

@melconcoast
melconcoast force-pushed the feature/execute-plan-skill branch from 5283677 to 77cd437 Compare August 23, 2026 07:47
@melconcoast
melconcoast force-pushed the feature/test-and-verify-skill branch from 5b9ea2b to be8dd47 Compare August 23, 2026 07:47
@melconcoast
melconcoast changed the base branch from feature/execute-plan-skill to master August 23, 2026 07:48
melconcoast and others added 3 commits August 23, 2026 01:48
The first three skills are a pipeline. This one is a service the others call:
`execute-plan` no longer runs tests itself, it hands each task and each
`Task X.V` gate to `test-and-verify` and acts on the verdict.

It reads the real output rather than the exit code — a suite that exits 0
having collected no tests, or having skipped the new ones, is a failure wearing
a green hat. Every failure gets classified as an application bug or a test bug
before anything is edited, because picking wrong either papers over a real
defect or bends correct code to satisfy a broken assertion.

Three remediation attempts, then it stops. Past three the diagnosis is usually
what was wrong rather than the fix, so further attempts just widen the diff on
a premise that was never true. `execute-plan` may not re-invoke it to get past
a red verdict; that is the same attempt with the safety removed, and it is
called out in both skills.

Two boundaries make it safe to call automatically. It never writes to a plan
file — it returns a verdict and `execute-plan` marks the boxes, because two
writers on one plan file is how a plan stops being trustworthy. And it never
weakens a test to reach green: no deleted assertions, loosened matchers, added
skips, widened timeouts, or expected values rewritten to whatever the code
happens to return. If the honest outcome is "still failing", that is what it
reports.

Test-command discovery now has one owner. It lived in `execute-plan`'s
verification.md and the new skill would have duplicated it, so that file keeps
only what is genuinely execute-plan's — bootstrapping a runner when a project
has none, which picks tooling and edits a manifest and so is a project decision
rather than a remediation step — and its "what counts as green" section narrows
to the question it actually owns: not whether the run passed, but whether that
pass closes a checkbox.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the scenarios caught this: "run the tests" and "verify this" did the
right thing but did it *inline*, without loading the skill — so the circuit
breaker, the never-weaken-a-test rule, and the gate's type-check widening were
never in play. The outcome was correct by luck, not by construction.

The cause is that the request looks too small to warrant a skill. The
description now says the discipline is the point rather than the command, and
says explicitly to use it even when the ask looks like a one-liner. All four
trigger phrases fire it now, and the twelve phrases belonging to the other
three skills still reach the skill they always did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixture G was doing two incompatible jobs. Its plan calls Phase 1 closed while
only Task 1.1 was ever built, and its Files Modified lists three .ts paths that
don't exist — so every execute-plan run against it stopped at Step 0 on the
false closure and never reached the behavior the scenario meant to test. Four
separate runs did exactly that.

That is the right behavior and S84 exists to check it, so Fixture G keeps the
inconsistency and is now documented as deliberate, with a note not to "fix" it.
The sixteen other scenarios that were keyed to it move to Fixture I — Fixture G
with Phase 1 told truthfully: one development task, two scenarios, both covered
by passing tests, and a Files Modified list of files that genuinely exist.

Fixture H now derives from I rather than G, since a failing-suite scenario has
no reason to inherit a broken plan. Its bug description also said `<` where the
seeded bug is `<=`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@melconcoast
melconcoast force-pushed the feature/test-and-verify-skill branch from be8dd47 to 7715458 Compare August 23, 2026 07:48
@melconcoast
melconcoast merged commit 96ac0e0 into master Aug 23, 2026
@melconcoast
melconcoast deleted the feature/test-and-verify-skill branch August 23, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant