v3.3.0 — add test-and-verify, and have execute-plan delegate to it - #4
Merged
Conversation
melconcoast
force-pushed
the
feature/execute-plan-skill
branch
from
August 23, 2026 07:47
5283677 to
77cd437
Compare
melconcoast
force-pushed
the
feature/test-and-verify-skill
branch
from
August 23, 2026 07:47
5b9ea2b to
be8dd47
Compare
The first three skills are a pipeline. This one is a service the others call: `execute-plan` no longer runs tests itself, it hands each task and each `Task X.V` gate to `test-and-verify` and acts on the verdict. It reads the real output rather than the exit code — a suite that exits 0 having collected no tests, or having skipped the new ones, is a failure wearing a green hat. Every failure gets classified as an application bug or a test bug before anything is edited, because picking wrong either papers over a real defect or bends correct code to satisfy a broken assertion. Three remediation attempts, then it stops. Past three the diagnosis is usually what was wrong rather than the fix, so further attempts just widen the diff on a premise that was never true. `execute-plan` may not re-invoke it to get past a red verdict; that is the same attempt with the safety removed, and it is called out in both skills. Two boundaries make it safe to call automatically. It never writes to a plan file — it returns a verdict and `execute-plan` marks the boxes, because two writers on one plan file is how a plan stops being trustworthy. And it never weakens a test to reach green: no deleted assertions, loosened matchers, added skips, widened timeouts, or expected values rewritten to whatever the code happens to return. If the honest outcome is "still failing", that is what it reports. Test-command discovery now has one owner. It lived in `execute-plan`'s verification.md and the new skill would have duplicated it, so that file keeps only what is genuinely execute-plan's — bootstrapping a runner when a project has none, which picks tooling and edits a manifest and so is a project decision rather than a remediation step — and its "what counts as green" section narrows to the question it actually owns: not whether the run passed, but whether that pass closes a checkbox. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the scenarios caught this: "run the tests" and "verify this" did the right thing but did it *inline*, without loading the skill — so the circuit breaker, the never-weaken-a-test rule, and the gate's type-check widening were never in play. The outcome was correct by luck, not by construction. The cause is that the request looks too small to warrant a skill. The description now says the discipline is the point rather than the command, and says explicitly to use it even when the ask looks like a one-liner. All four trigger phrases fire it now, and the twelve phrases belonging to the other three skills still reach the skill they always did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixture G was doing two incompatible jobs. Its plan calls Phase 1 closed while only Task 1.1 was ever built, and its Files Modified lists three .ts paths that don't exist — so every execute-plan run against it stopped at Step 0 on the false closure and never reached the behavior the scenario meant to test. Four separate runs did exactly that. That is the right behavior and S84 exists to check it, so Fixture G keeps the inconsistency and is now documented as deliberate, with a note not to "fix" it. The sixteen other scenarios that were keyed to it move to Fixture I — Fixture G with Phase 1 told truthfully: one development task, two scenarios, both covered by passing tests, and a Files Modified list of files that genuinely exist. Fixture H now derives from I rather than G, since a failing-suite scenario has no reason to inherit a broken plan. Its bug description also said `<` where the seeded bug is `<=`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
melconcoast
force-pushed
the
feature/test-and-verify-skill
branch
from
August 23, 2026 07:48
be8dd47 to
7715458
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3 (which is stacked on #2). Merge in order; each base retargets automatically.
The first three skills are a pipeline. This one is a service the others call:
execute-planno longer runs tests itself — it hands each task and eachTask X.Vgate totest-and-verifyand acts on the verdict.The skill
It finds the project's test command, runs the tests relevant to the work, and reads the real output rather than the exit code — a suite that exits 0 having collected no tests, or having skipped the new ones, is a failure wearing a green hat.
On a failure it classifies the bug as application code or test before editing anything, because picking wrong either papers over a real defect or bends correct code to satisfy a broken assertion. One targeted fix per attempt. Three attempts, then it stops — past three the diagnosis is usually what was wrong rather than the fix, so further attempts widen the diff on a premise that was never true.
Two boundaries make it safe to call automatically:
execute-planmarks the boxes. Two writers on one plan file is how a plan stops being trustworthy.At a gate it widens to the whole module's suite plus the type-checker and linter, and says so explicitly when the project has neither, since an omitted line reads as a check that passed.
Refactor: test-command discovery has one owner
Discovery lived in
execute-plan'sverification.mdand the new skill would have duplicated it. That file keeps only what is genuinelyexecute-plan's — bootstrapping a runner when a project has none, which picks tooling and edits a manifest and so is a project decision, not a remediation step — and its "what counts as green" section narrows to the question it actually owns: not whether the run passed, but whether that pass closes a checkbox.Verification
Gates: four skills, names matching directories, descriptions at 857/861/994/1007 characters (limit 1024), 63/61/50/119 lines,
plugin.jsonat 3.3.0, zero broken links.Against the installed 3.3.0 plugin in fresh headless sessions:
scaffold,plan-module, andexecute-planstill reach the skill they always did. 12/12.test-and-verifyphrases fire it.<=where the rule says "not before"), it diagnosed application code, checked the claim againstdocs/product.mdand the plan's own scenario, changed exactly one character in one file, and went green. It did not touch the test.execute-planfired, delegated totest-and-verify, took its verdict, and wrote the plan file itself. Both skills in one run, 10 tests green, real code built.Status:flipped toDoneat[2/2 Phases Closed], and the roadmap write-back used bare vocabulary words throughout.One defect the runs caught, fixed here
"run the tests" and "verify this" originally did the right thing but did it inline, without loading the skill — so the circuit breaker, the no-weakening rule, and the gate widening were never in play. Correct by luck, not construction. The request simply looks too small to warrant a skill. The description now says the discipline is the point rather than the command, and to use it even when the ask looks like a one-liner.
Known limitation, stated honestly: standalone triggering is now reliable when tool access is constrained, but under full permissions the model still sometimes runs a trivial suite inline rather than loading the skill. The path that matters —
execute-plandelegating — is driven by an explicit instruction in its Step 3 rather than description matching, and that fired reliably.🤖 Generated with Claude Code