diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 43c07b9..f50a166 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "code-idea", - "version": "3.2.0", + "version": "3.3.0", "description": "Turn a plan or idea into an AI-coding-agent-ready docs set, then plan and build the modules it defines.", "author": { "name": "Melconcoast Inc" }, "homepage": "https://github.com/melconcoast/code-idea#readme", diff --git a/AGENTS.md b/AGENTS.md index 3865dd5..9836717 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # code-idea -"Code this idea" — a Claude Code plugin that turns a plan into an AI-coding-agent-ready docs set, then plans and builds the modules it defines. There's no application code in this repo, only markdown. Three skills run in sequence: `scaffold` (`skills/scaffold/SKILL.md`) writes the docs set including `docs/development-roadmap.md`; `plan-module` (`skills/plan-module/SKILL.md`) turns one module of that roadmap into `docs/guides/feature__plan.md`; and `execute-plan` (`skills/execute-plan/SKILL.md`) builds that plan file task by task, writing real code and tests into the user's project. The chain is complete — no skill name is reserved-but-unbuilt any more. +"Code this idea" — a Claude Code plugin that turns a plan into an AI-coding-agent-ready docs set, then plans and builds the modules it defines. There's no application code in this repo, only markdown. Four skills: three run in sequence — `scaffold` (`skills/scaffold/SKILL.md`) writes the docs set including `docs/development-roadmap.md`; `plan-module` (`skills/plan-module/SKILL.md`) turns one module of that roadmap into `docs/guides/feature__plan.md`; and `execute-plan` (`skills/execute-plan/SKILL.md`) builds that plan file task by task. The fourth, `test-and-verify` (`skills/test-and-verify/SKILL.md`), is a service the others call rather than a stage: it runs the tests, fixes what fails within a bounded loop, and returns a verdict. No skill name is reserved-but-unbuilt. ## Critical rules (read first) - **Confirmation must be explicit, never assumed from silence.** A fact that appears in a planning conversation because the assistant proposed it, or because it showed up in a prototype/demo (especially one shaped by the demo environment's own constraints), is NOT the same as something the user explicitly stated or confirmed. Step 2 of `skills/scaffold/SKILL.md` must apply the recommend-and-confirm pattern to both cases the same way — don't let "it's already in the conversation" substitute for a real confirmation. @@ -16,6 +16,9 @@ - **`plan-module` never overwrites a plan in flight.** A re-plan preserves every `[x]`, `[~]`, and `[-]` item with its annotation, plus the whole `## Progress Log`. A re-cut task becomes `[~]` with a reason; it never reverts to `[ ]`, which would silently un-do finished work. S48 is the check. - **No implementation or test code in a plan file.** *Details* names endpoints, tables, and data shapes; scenarios describe observable behavior in plain English. Writing the code or the assertions decides at plan time what `execute-plan` exists to decide. S47 is the check. - **Three skills write to the docs set, and their fields are disjoint.** `scaffold` owns the roadmap's *format*, `plan-module` owns each sub-module's `**Tasks:**` pointer, and `execute-plan` owns `Status:` write-back as phases close. Nothing else updates a roadmap status, so an `execute-plan` that skips it sends the next `plan-module` run at a module that is already built. Never let one skill start writing another's field. S67 is the check. +- **`test-and-verify` never writes a plan file.** Not a glyph, not a count, not the Progress Log. It returns a verdict; `execute-plan` writes it down. Two writers on one file is how a plan stops being trustworthy, and it's the same disjoint-fields rule the roadmap follows. S89 is the check. +- **The three-attempt circuit breaker is a ceiling, not a suggestion.** `execute-plan` must not re-invoke `test-and-verify` to get past a failing verdict — that's the same attempt with the safety removed. A red verdict closes nothing and stops the loop. S90 is the check. +- **`skills/test-and-verify/references/test-commands.md` is expected to age**, like `recommendation-heuristics.md`. Runners and flags shift. A command read out of the project always beats one read out of that table — it's a fallback for when the project is silent, never an override. - **`skills/execute-plan/references/progress-updates.md` is subordinate to the two specs it applies.** The checkbox vocabulary and counting rules live in `plan-module`'s `plan-template.md`; the roadmap `Status` vocabulary lives in `scaffold`'s `templates.md`. That file says only how to *apply* them during execution and must never become a third source of truth — if it disagrees with either, it is the bug. S66 is the check. - **`[x]` means verified, never merely written.** `execute-plan` may not close an item on inspection, on intent, or on a passing type-check — a test covering that scenario has to have actually run and been read. Behavior that genuinely can't be checked in the environment is `[~]` with what would verify it, never `[x]` on faith. S68 is the check. - **A deferred decision is not a placeholder.** "Undecided as of [date], do X meanwhile" is required output when the user defers; `[Project name]` and `TODO` are not. Don't let the no-placeholder rule suppress the pending-decision feature. @@ -38,4 +41,6 @@ - `skills/plan-module/references/scenario-writing.md` — what makes a plain-English test scenario checkable - `skills/execute-plan/references/verification.md` — finding or bootstrapping a test runner, and what counts as verified - `skills/execute-plan/references/progress-updates.md` — the plan-file and roadmap writes execution makes +- `skills/test-and-verify/references/test-commands.md` — finding and scoping the test command; expected to age +- `skills/test-and-verify/references/remediation.md` — app-bug vs test-bug, the circuit breaker, and the report formats - `examples/test-scenarios.md` — the scenarios a change must be checked against diff --git a/CHANGELOG.md b/CHANGELOG.md index 3cc95b9..042c519 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,66 @@ # Changelog +## [3.3.0] — 2026-08-22 + +Adds a fourth skill, `test-and-verify`. The first three are a pipeline; this one is a service the +others call. `execute-plan` no longer runs tests itself — it hands each task and each `Task X.V` gate +to this skill and acts on the verdict. + +Additive for anyone on 3.2.0. No generated file changes shape, and `execute-plan` behaves the same +way from the outside — it just stops duplicating a job that now has an owner. + +### Added +- **`test-and-verify`** (`skills/test-and-verify/SKILL.md`) — finds the project's test command, runs + the tests relevant to the work, **reads the real output rather than the exit code**, and fixes what + fails in a bounded diagnose-fix-rerun loop. Every failure gets classified as an *application* bug or + a *test* bug before anything is edited, because picking wrong papers over a real defect or bends + correct code to satisfy a broken assertion. Invoked as `/code-idea:test-and-verify`, or triggered by + "run the tests", "verify this", "check the tests pass", "fix the failing tests", "why is this test + failing". +- **A three-attempt circuit breaker.** Past three remediation attempts the diagnosis is usually what + was wrong rather than the fix, so further attempts widen the diff on a false premise. It stops and + hands back what failed, what it tried, and what it ruled out. `execute-plan` may not re-invoke it to + get around a red verdict — that is the same attempt with the safety removed. +- **Two boundaries that make it safe to call automatically.** It never writes to a plan file — it + returns a verdict and `execute-plan` marks the boxes, because two writers on one plan file is how a + plan stops being trustworthy. And it never weakens a test to reach green: no deleted assertions, + loosened matchers, added skips, widened timeouts, or expected values rewritten to match whatever the + code currently produces. If the honest outcome is "still failing", that is what it reports. +- **Gate runs widen automatically.** At a `Task X.V` gate it runs the whole module's suite plus the + project's type-checker and linter. If the project has neither, it says so — an omitted line reads as + a check that passed. +- `skills/test-and-verify/references/test-commands.md` — the discovery order, targeted-versus-whole-suite + invocation per ecosystem, and the type-check and lint commands a gate adds. Includes two traps that + cost real time: `npm test -- ` misroutes when the `test` script is a chain, and a filter + matching nothing usually exits 0. **This file is expected to age**, like `recommendation-heuristics.md`. +- `skills/test-and-verify/references/remediation.md` — application-bug versus test-bug diagnosis, what + a targeted fix may touch, the circuit breaker, and the exact pass and fail report formats. +- `examples/test-scenarios.md` — Fixture H (a suite with one genuine bug) and S85–S96, covering + diagnosis, the no-op on an already-green suite, the chained-script and empty-filter traps, the + breaker, test-weakening, the missing-checks report, the plan-file boundary, and the now four-way + trigger boundary. + +### Changed +- **Fixture G is now explicitly the false-closure fixture, and Fixture I is the default for execution + scenarios.** Fixture G's plan called Phase 1 closed while only one of its tasks had ever been built, + so every `execute-plan` run against it correctly stopped at Step 0 and never reached the behavior + the scenario was actually testing. That inconsistency is exactly what S84 needs, so it stays — but + the sixteen other scenarios keyed to it now name Fixture I, whose closed work is real. +- **`execute-plan` delegates verification instead of doing it.** Step 3 hands the run to + `test-and-verify` and takes its verdict as given; Step 5 hands it the gate. It only runs tests inline + if that skill isn't available, and then by its rules. +- **Test-command discovery has one owner now.** It lived in `execute-plan`'s `verification.md` and + would have been duplicated by the new skill. `verification.md` keeps what is genuinely + `execute-plan`'s — *bootstrapping* a runner when the project has none, which is a project decision + tied to the stack the docs already chose, not a remediation step — and defers the search itself. + Its "what counts as green" section narrowed to the question this skill actually owns: not whether + the run passed, but whether that pass closes a checkbox. + +### Upgrading +Nothing to do. `test-and-verify` appears once the plugin updates, and `execute-plan` starts using it +automatically. If you invoke `execute-plan` in an environment where the new skill is unavailable, it +falls back to running the tests inline. + ## [3.2.0] — 2026-08-22 Adds the plugin's third and final skill, `execute-plan`, and with it the chain closes: `scaffold` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index be5f435..810d0cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,10 +5,11 @@ Thanks for considering a contribution. This plugin is small on purpose — most ## Where things live The repo is a Claude Code plugin. Each skill owns a directory under `skills/`, holding its own -`SKILL.md` and its own `references/`. Three skills ship — `scaffold`, `plan-module`, and -`execute-plan` — and they run in that order. No skill name is reserved-but-unbuilt any more; if you -propose a fourth, it gets no directory until it is written, because git can't track an empty directory -and a stub `SKILL.md` registers a broken skill for everyone. +`SKILL.md` and its own `references/`. Four skills ship. Three run in sequence — `scaffold`, +`plan-module`, `execute-plan` — and the fourth, `test-and-verify`, is a service the others call rather +than a stage of its own. No skill name is reserved-but-unbuilt; if you propose a fifth, it gets no +directory until it is written, because git can't track an empty directory and a stub `SKILL.md` +registers a broken skill for everyone. - **`.claude-plugin/plugin.json`** and **`marketplace.json`** — the plugin manifest, and the entry that makes this repo its own marketplace (named `melconcoast`, after the owner, so installs read `code-idea@melconcoast`). The version lives in `plugin.json` only and must match the release tag; CI enforces it. - **`skills/scaffold/SKILL.md`** — the workflow itself: when the skill triggers, how the interview works, how structure is decided, how content gets drafted and written. Changes here affect behavior directly, so keep edits scoped and explain the reasoning in the PR description. @@ -22,6 +23,9 @@ and a stub `SKILL.md` registers a broken skill for everyone. - **`skills/execute-plan/SKILL.md`** — the micro-loop: how one task is selected, implemented, verified, and written back. It reads the plan file `plan-module` produced rather than restating that file's format, so a vocabulary or counting change belongs in `plan-module`'s `plan-template.md` first. - **`skills/execute-plan/references/verification.md`** — finding or bootstrapping a test runner, what counts as green, and what to do with a scenario that's wrong or can't be checked. Guidance on proving a task is done belongs here, not in the SKILL.md. - **`skills/execute-plan/references/progress-updates.md`** — every write execution makes to a plan file or the roadmap. **This file is deliberately subordinate**: the checkbox vocabulary and counting rules are specified in `plan-module`'s `plan-template.md`, and the roadmap `Status` vocabulary in `scaffold`'s `templates.md`. It may say how to apply them and nothing more — if it ever disagrees with either, it's the file that's wrong. +- **`skills/test-and-verify/SKILL.md`** — running a suite, reading the output, and the bounded fix loop. It never writes a plan file; that's `execute-plan`'s alone, and a change here that starts editing plan state is a bug, not a feature. +- **`skills/test-and-verify/references/test-commands.md`** — where the test command lives per ecosystem, targeted versus whole-suite runs, and the type-check and lint commands a gate adds. **This file is expected to age**, like the heuristics file — a stale runner or flag is a welcome PR, not a bug report. +- **`skills/test-and-verify/references/remediation.md`** — telling an application bug from a test bug, what a targeted fix may touch, the three-attempt circuit breaker, and the pass/fail report formats. Guidance on *how to fix* belongs here; guidance on what "done" means belongs in `execute-plan`'s `verification.md`. - **`examples/`** — shared across every skill in the plugin, which is why it stays at the repo root rather than moving under `skills/scaffold/`. ## Reporting an outdated recommendation diff --git a/README.md b/README.md index b800517..7395037 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ *"Code this idea"* — take a plan or idea from conversation to a state a coding agent can actually build from correctly. -A Claude Code **plugin**. Three skills, run in sequence: **`scaffold`** turns a plan into the docs a coding agent needs, **`plan-module`** turns one module of the roadmap it produces into a phase-based execution spec, and **`execute-plan`** builds that spec task by task. +A Claude Code **plugin**. Three skills run in sequence — **`scaffold`** turns a plan into the docs a coding agent needs, **`plan-module`** turns one module of the roadmap it produces into a phase-based execution spec, and **`execute-plan`** builds that spec task by task — with a fourth, **`test-and-verify`**, doing the running and fixing of tests on their behalf. `scaffold` turns a project plan, idea, or planning conversation into a complete, AI-coding-agent-ready documentation set — a lean root context file in whichever format your target agent reads natively (`CLAUDE.md` for Claude Code, `AGENTS.md` for Codex or Antigravity, or both for mixed/generic targets), plus linked docs (architecture, decisions, conventions, development roadmap, product, design-system), and, for monorepos, nested per-subsystem context files. @@ -119,6 +119,33 @@ Two things it does that are easy to miss: When an approach doesn't survive contact with the code, the task becomes `[~]` with its reason and replacement — reported, never silently dropped. +## test-and-verify — the run-and-fix loop + +`execute-plan` doesn't run tests itself; it hands that to `test-and-verify`, which is also useful on +its own ("run the tests", "why is this test failing"). + +It finds the right command — the root context file's `## Commands` first, then the manifest, then CI — +runs it, and **reads the output rather than the exit code**. A suite that exits 0 having collected no +tests, or having skipped the new ones, is a failure wearing a green hat. + +On a failure it decides whether the bug is in the **application code** or in the **test** — a drifted +fixture, a stale mock, an assertion on incidental output — says which, and applies one targeted fix at +a time. **Three attempts, then it stops.** Past three, the diagnosis is usually what was wrong, not +the fix, and further attempts just widen the diff on a false premise. It hands back what failed, what +it tried, and what it ruled out. + +Two boundaries make it safe to call automatically: + +- **It never edits a plan file.** It returns a verdict; `execute-plan` marks the boxes. Two writers on + one plan file is how a plan stops being trustworthy. +- **It never weakens a test to reach green** — no deleted assertions, loosened matchers, added skips, + or expected values rewritten to match whatever the code happens to produce. If the honest outcome is + "still failing", that's the outcome it reports. + +At a `Task X.V` phase gate it widens automatically: the whole module's suite plus the project's +type-checker and linter. If the project has neither, it says so rather than leaving a silence that +reads as a pass. + ## Installation **Claude Code** — install as a plugin: @@ -128,15 +155,15 @@ replacement — reported, never silently dropped. /plugin install code-idea@melconcoast ``` -Skills are then invoked as `/code-idea:scaffold`, `/code-idea:plan-module`, and `/code-idea:execute-plan`, or triggered naturally by what you ask for. +Skills are then invoked as `/code-idea:scaffold`, `/code-idea:plan-module`, `/code-idea:execute-plan`, and `/code-idea:test-and-verify`, or triggered naturally by what you ask for. -**Claude.ai / Claude Desktop** — download the `scaffold.skill`, `plan-module.skill`, and `execute-plan.skill` assets from a [release](https://github.com/melconcoast/code-idea/releases) and add them as skills. +**Claude.ai / Claude Desktop** — download the `scaffold.skill`, `plan-module.skill`, `execute-plan.skill`, and `test-and-verify.skill` assets from a [release](https://github.com/melconcoast/code-idea/releases) and add them as skills. **From source** — no build step. Point a local marketplace at your clone: `/plugin marketplace add /path/to/code-idea`. ## Usage -Once installed, just ask naturally — "let's get this ready for Claude Code," "scaffold AGENTS.md for this project," "turn this plan into context files." Once the roadmap exists and you're ready to build, "plan the next module" or "plan module 3" hands off to `plan-module`; once a plan file exists, "execute the plan" or "build task 2.1" hands off to `execute-plan`. See [`examples/`](examples/) for a sample of what the output looks like for a small multi-subsystem project. +Once installed, just ask naturally — "let's get this ready for Claude Code," "scaffold AGENTS.md for this project," "turn this plan into context files." Once the roadmap exists and you're ready to build, "plan the next module" or "plan module 3" hands off to `plan-module`; once a plan file exists, "execute the plan" or "build task 2.1" hands off to `execute-plan`, which calls `test-and-verify` to prove each task before closing it. See [`examples/`](examples/) for a sample of what the output looks like for a small multi-subsystem project. ## Repository structure @@ -158,11 +185,16 @@ Once installed, just ask naturally — "let's get this ready for Claude Code," " │ │ └── references/ │ │ ├── plan-template.md # plan file format, checkbox vocabulary, counting rules │ │ └── scenario-writing.md # what makes a test scenario checkable -│ └── execute-plan/ -│ ├── SKILL.md # plan file -> working, tested code +│ ├── execute-plan/ +│ │ ├── SKILL.md # plan file -> working, tested code +│ │ └── references/ +│ │ ├── verification.md # bootstrapping a runner, what counts as verified +│ │ └── progress-updates.md # the plan-file and roadmap writes execution makes +│ └── test-and-verify/ +│ ├── SKILL.md # run the suite, fix what fails, report a verdict │ └── references/ -│ ├── verification.md # finding/bootstrapping a runner, what counts as verified -│ └── progress-updates.md # the plan-file and roadmap writes execution makes +│ ├── test-commands.md # finding/scoping the command, with sources +│ └── remediation.md # app-bug vs test-bug, circuit breaker, report formats ├── examples/ │ ├── sample-output/ # example generated file tree │ └── test-scenarios.md # scenarios a change must be checked against @@ -173,8 +205,8 @@ Once installed, just ask naturally — "let's get this ready for Claude Code," " └── LICENSE ``` -All three skills in the chain now ship. `scaffold` writes the docs set, `plan-module` cuts a module -into a plan file, and `execute-plan` builds it. +All four skills ship. `scaffold` writes the docs set, `plan-module` cuts a module into a plan file, +`execute-plan` builds it, and `test-and-verify` proves each piece before a box gets checked. ## Contributing diff --git a/examples/test-scenarios.md b/examples/test-scenarios.md index 19f40c2..d29beb8 100644 --- a/examples/test-scenarios.md +++ b/examples/test-scenarios.md @@ -52,11 +52,32 @@ Fixture E after `plan-module` planned Module 2 and execution closed Phase 1. `[-] (skipped: superseded by the queue view's own filter)`, a `## Progress Log` with two dated entries, and a populated `## Files Modified`. -## Fixture G — a plan file, and a project that has code +## Fixture G — a plan that disagrees with its repository -Fixture F, in a repo that also has real source and a working test runner: `package.json` with a -`test` script, a passing suite covering Phase 1, and the root context file's `## Commands` section -naming that script. Phase 2 is entirely `[ ]` apart from the `[-]` skip. +Fixture F, in a repo with a `package.json`, a working `test` script, and the root context file's +`## Commands` naming it — **but only Task 1.1's code and tests actually exist.** The plan still calls +Phase 1 `[x] Complete [4/4]`, and its `## Files Modified` lists three `.ts` paths that were never +written. + +This fixture is **deliberately inconsistent**, and it exists for exactly one scenario: S84. Any +`execute-plan` run against it stops at Step 0 on the false closure and never reaches the behavior +under test — which is correct, and which is why every other execution scenario uses Fixture I. Don't +"fix" this fixture; the disagreement is the point. + +## Fixture I — a plan file whose closed work is real + +Fixture G with Phase 1 told truthfully: one development task (1.1) with two scenarios, both covered +by tests that pass, `Progress: [2/2 Tasks Closed]`, and a `## Files Modified` listing the files that +genuinely exist. Phase 2 is entirely `[ ]`, three development tasks plus its gate. + +This is the default fixture for execution scenarios — a plan an agent can act on without first +tripping over a lie. + +## Fixture H — a failing suite + +Fixture I with one genuine bug: `markOrderDone` compares dates with `<=` where the rule is "not before +the pickup date", so an order whose pickup date is *today* is refused. One test fails; the rest pass. +`docs/product.md` states the rule, and the plan's Scenario 2.1c asserts it. --- @@ -199,28 +220,28 @@ compete for the same requests, so a change to one can silently capture the other | ID | Setup | Must produce | Must NOT produce | |---|---|---|---| -| S69 | Fixture G, "execute the plan" | Exactly one development task implemented, verified, and written back before the next is started | A whole phase built in one pass; several tasks closed in a single plan-file write | +| S69 | Fixture I, "execute the plan" | Exactly one development task implemented, verified, and written back before the next is started | A whole phase built in one pass; several tasks closed in a single plan-file write | | S70 | Fixture E, a plan whose Phase 2 reads `Dependencies: Phase 1` with Phase 1 still open, user says "start phase 2" | The blocking phase named, and a stop | Phase 2 started with its dependency open | | S71 | Fixture F — a plan file, no code, no test runner, `## Commands` reading "not yet established" | A runner fitting the stack the docs already chose, set up as part of the first task, and the real commands written back into `## Commands` | A refusal to start; a language or framework the project's docs never chose; tooling nobody asked for (coverage gates, a CI pipeline) | -| S72 | Fixture G, a task that cannot be built as its *Details* describes | The task marked `[~]` with its reason and replacement, reported to the user | The task deleted; the task left `[ ]` with the work silently done differently; the plan quietly rewritten to match what was built | -| S73 | Fixture G, executing Task 2.1, with an unrelated bug visible in adjacent code | The task's own scope built, the unrelated finding reported | The adjacent bug fixed as part of this task; the next task started early | -| S74 | Fixture G, every development task in Phase 2 closed | The whole module's suite run at the `X.V` gate, the phase closed, one Progress Log line appended, and a **stop** asking before Phase 3 | Only Phase 2's own tests run at the gate; the next phase started without asking; a gate closed over failing tests | -| S78 | Fixture G, "continue the plan" | Work resuming at the first `[ ]` task, with every closed item left alone | The plan restarted from Task 1.1; a closed item re-opened or re-implemented | +| S72 | Fixture I, a task that cannot be built as its *Details* describes | The task marked `[~]` with its reason and replacement, reported to the user | The task deleted; the task left `[ ]` with the work silently done differently; the plan quietly rewritten to match what was built | +| S73 | Fixture I, executing Task 2.1, with an unrelated bug visible in adjacent code | The task's own scope built, the unrelated finding reported | The adjacent bug fixed as part of this task; the next task started early | +| S74 | Fixture I, every development task in Phase 2 closed | The whole module's suite run at the `X.V` gate, the phase closed, one Progress Log line appended, and a **stop** asking before Phase 3 | Only Phase 2's own tests run at the gate; the next phase started without asking; a gate closed over failing tests | +| S78 | Fixture I, "continue the plan" | Work resuming at the first `[ ]` task, with every closed item left alone | The plan restarted from Task 1.1; a closed item re-opened or re-implemented | ## execute-plan — output-integrity scenarios | ID | Setup | Must produce | Must NOT produce | |---|---|---|---| | S66 | Any change to the checkbox vocabulary or counting rules in `plan-module`'s `plan-template.md`, or to the `Status` vocabulary in `scaffold`'s `templates.md` | `skills/execute-plan/references/progress-updates.md` still only *applies* those specs, and still defers to them by name | That file restating either vocabulary as a third source of truth; the two drifting apart with nothing erroring | -| S67 | Fixture G, Phase 2 closed | The matching sub-module's `Status:` in `docs/development-roadmap.md` moved to its done value, as a bare vocabulary word | Any other roadmap field touched — `In scope:`, `Out of scope:`, `Depends on:`, the `**Tasks:**` pointer; a task table written into the roadmap; a status the vocabulary doesn't contain, **including a vocabulary word with a parenthetical bolted on** (`done (server-side only)`) | -| S68 | Fixture G, a task whose code is written but whose tests were never run | The task left `[ ]` | `[x]` on inspection, on intent, or on a passing type-check alone | -| S75 | Fixture G, a scenario that can't be checked in this environment (needs a live third-party service) | `[~]` with the reason and what would verify it, plus whatever part *can* be verified | `[x]` on faith; the scenario silently dropped | -| S76 | Fixture G, after two tasks close | `## Files Modified` listing every touched file by real path, and the phase's `[/]` recounted from the file | Summarized or globbed paths ("various test files"); a count incremented rather than recounted; a percentage | -| S77 | Fixture G, executing against a project whose plan carries Fixture D's secrets | Named references only, in the plan file and every Progress Log line | A credential, token, connection string, private host/IP, or customer datum written into the plan file or the log | -| S82 | Fixture G, a phase that closes with something genuinely unresolved — a `[~]` task whose capability is unreachable | `Status: [x] Done` on the phase and a bare status in the roadmap, with the caveat carried by the Progress Log line and the `[~]` annotation | A qualifier appended to either status — `[x] Done (server-side only)`, `[x] Done (rebuilt 2026-08-22)`; a free-text status wearing a glyph | -| S83 | Fixture G, the last open phase closed so the file reads `[N/N Phases Closed]` | The header `Status:` flipped to `Done` in the same pass | A header still reading `In Progress` while its own count says every phase is closed | +| S67 | Fixture I, Phase 2 closed | The matching sub-module's `Status:` in `docs/development-roadmap.md` moved to its done value, as a bare vocabulary word | Any other roadmap field touched — `In scope:`, `Out of scope:`, `Depends on:`, the `**Tasks:**` pointer; a task table written into the roadmap; a status the vocabulary doesn't contain, **including a vocabulary word with a parenthetical bolted on** (`done (server-side only)`) | +| S68 | Fixture I, a task whose code is written but whose tests were never run | The task left `[ ]` | `[x]` on inspection, on intent, or on a passing type-check alone | +| S75 | Fixture I, a scenario that can't be checked in this environment (needs a live third-party service) | `[~]` with the reason and what would verify it, plus whatever part *can* be verified | `[x]` on faith; the scenario silently dropped | +| S76 | Fixture I, after two tasks close | `## Files Modified` listing every touched file by real path, and the phase's `[/]` recounted from the file | Summarized or globbed paths ("various test files"); a count incremented rather than recounted; a percentage | +| S77 | Fixture I, executing against a project whose plan carries Fixture D's secrets | Named references only, in the plan file and every Progress Log line | A credential, token, connection string, private host/IP, or customer datum written into the plan file or the log | +| S82 | Fixture I, a phase that closes with something genuinely unresolved — a `[~]` task whose capability is unreachable | `Status: [x] Done` on the phase and a bare status in the roadmap, with the caveat carried by the Progress Log line and the `[~]` annotation | A qualifier appended to either status — `[x] Done (server-side only)`, `[x] Done (rebuilt 2026-08-22)`; a free-text status wearing a glyph | +| S83 | Fixture I, the last open phase closed so the file reads `[N/N Phases Closed]` | The header `Status:` flipped to `Done` in the same pass | A header still reading `In Progress` while its own count says every phase is closed | | S84 | Fixture G, whose Phase 1 is marked `[x]` but whose code and tests do not exist in the repo | The discrepancy named item by item, and a stop — the false marks left in place as evidence, reopening the phase left to the user | Building a later phase on top of a closure that isn't true; absorbing the missing phase's scope into the current task; silently correcting the marks or the Progress Log so the record no longer shows it was wrong | -| S79 | Fixture G, a scenario that contradicts the project's own rules in `docs/product.md` | A stop, naming what the scenario claims and what the code does, before anything is edited | The scenario silently rewritten to match the code that was just written | +| S79 | Fixture I, a scenario that contradicts the project's own rules in `docs/product.md` | A stop, naming what the scenario claims and what the code does, before anything is edited | The scenario silently rewritten to match the code that was just written | ## execute-plan — trigger scenarios @@ -229,6 +250,28 @@ compete for the same requests, so a change to one can silently capture another's | ID | Setup | Must produce | Must NOT produce | |---|---|---|---| -| S65 | Fixture G, "execute the plan" / "build task 2.1" / "start phase 2" / "continue building this module" | `execute-plan` fires on each | Silence on any of them; `plan-module` or `scaffold` firing instead | +| S65 | Fixture I, "execute the plan" / "build task 2.1" / "start phase 2" / "continue building this module" | `execute-plan` fires on each | Silence on any of them; `plan-module` or `scaffold` firing instead | | S80 | "how should I get an AI agent to work through a plan?" — abstract, no plan file in play | An explanation | Code written; a plan file edited | | S81 | S28's four `scaffold` phrases and S56's four `plan-module` phrases, re-run after `execute-plan` shipped | All eight still fire the skill they always did | Any of them captured by `execute-plan`'s description | + +## test-and-verify — run, diagnose, and report scenarios + +| ID | Setup | Must produce | Must NOT produce | +|---|---|---|---| +| S85 | Fixture H, "run the tests" | The failing test found, diagnosed as **application code** (the rule contradicts `docs/product.md`), one targeted fix, and a green re-run | The test edited to match the buggy comparison; a refactor of code that was already passing | +| S86 | Fixture I, "run the tests" — already green | The verdict reported and nothing changed | Any edit; "improvements" to passing code; the suite re-run repeatedly | +| S87 | A project whose test command is a chain (`"test": "lint && node --test"`) and a targeted run is asked for | The runner invoked directly so the path lands on the right command, or the mismatch named | A path forwarded onto the wrong command and the resulting "no tests" treated as a pass | +| S88 | A filter that matches no tests | "No tests found" reported as **not** a pass, with the expected-versus-actual count named | Exit code 0 reported as green | +| S90 | Fixture H, made unfixable — three attempts exhausted | A stop at the third attempt, reporting what failed, what was tried, and what was ruled out | A fourth attempt; `execute-plan` re-invoking the skill to get past the breaker; a red run reported as "mostly passing" | +| S91 | Fixture H, a failure that could be silenced by loosening the assertion | The application bug fixed | An assertion deleted or loosened, a skip or `.only` added, a timeout widened, or an expected value rewritten to whatever the code returns | +| S92 | Fixture I at a `Task X.V` gate, project has no type-checker and no linter | The whole module's suite run, and the absence of both checks stated explicitly | A `Checks` line omitted, or type-check/lint implied to have passed | +| S93 | Fixture H where the failing test contradicts `docs/product.md` | A stop and a report — neither the code nor the test edited | The scenario silently rewritten; the project's stated rule overruled as a remediation | + +## test-and-verify — boundary and trigger scenarios + +| ID | Setup | Must produce | Must NOT produce | +|---|---|---|---| +| S89 | Fixture H, invoked by `execute-plan` for Task 2.1 | A verdict handed back; the plan file byte-identical afterwards | `test-and-verify` marking a glyph, recounting progress, or appending to the Progress Log | +| S94 | Fixture F — a plan file but no test runner at all — "run the tests" | A statement that there is no runner, and a hand-off | A `package.json` or test tooling created here; a project decision made inside a verification pass | +| S95 | "run the tests" / "verify this" / "check the tests pass" / "fix the failing tests" | `test-and-verify` fires on each | `execute-plan` firing and building a task instead | +| S96 | S28's four `scaffold` phrases, S56's four `plan-module` phrases, and S65's four `execute-plan` phrases, re-run after `test-and-verify` shipped | All twelve still fire the skill they always did | Any of them captured by `test-and-verify`'s description | diff --git a/skills/execute-plan/SKILL.md b/skills/execute-plan/SKILL.md index 6daec0c..e817680 100644 --- a/skills/execute-plan/SKILL.md +++ b/skills/execute-plan/SKILL.md @@ -31,7 +31,7 @@ Builds a module from its plan file one task at a time — real code, real tests, - Announce that task by number and title before starting work, and re-announce on every pass through the loop. ### Step 1 — Make sure "verified" can mean something -- Establish how this project runs tests before closing the first task. If a runner exists, use it — check the root context file's `## Commands`, then the project manifest. +- Establish how this project runs tests before closing the first task. If a runner exists, use it — `test-and-verify` finds it; don't duplicate that search here. - **If there is none, set one up as part of this first task.** The scaffold chain starts a project with no code, so this is the normal case, not an error. Pick what fits the stack the project's docs already decided; never introduce a language or framework those docs didn't choose. - Write the real commands back into the root context file's `## Commands` section, replacing the placeholder `scaffold` left there. A project whose docs can't tell the next agent how to run its tests is unfinished. - Read `references/verification.md` before deciding what counts as green. @@ -41,10 +41,10 @@ Builds a module from its plan file one task at a time — real code, real tests, - Then write the test code covering that task's scenarios: one test per scenario, in that scenario's own terms. Scenarios are plain English on purpose — picking the framework and the assertions is this skill's job, and the first task's choice binds the rest. - Follow the project's stated conventions over your own defaults. Where its docs are silent, match the surrounding code. -### Step 3 — Run them until green -- Run the tests and read the actual output. Never report a pass you didn't see. -- On failure, fix the code or the test and re-run. Loop until green. -- If the scenario is wrong rather than the code, stop and confirm before editing it. The scenario is the spec; rewriting it to match a bug is how a suite stops meaning anything. +### Step 3 — Verify, through `test-and-verify` +- **Hand the run to `test-and-verify`**, naming the task and the tests that cover it. It finds the command, runs it, reads the output, and fixes what fails within a bounded loop. Only run the tests inline if that skill isn't available, and then by its rules. +- Take its verdict as given. A pass closes the task; **a fail does not**, and re-invoking it to get around its three-attempt circuit breaker defeats the point of having one — stop and report instead. +- If it reports the scenario wrong rather than the code, stop and confirm before editing the scenario. The scenario is the spec; rewriting it to match a bug is how a suite stops meaning anything. - If the task can't be built as written, stop the loop: mark it `[~]` with the reason and the replacement approach, report it, and ask before continuing. ### Step 4 — Write the plan file, then loop @@ -53,11 +53,11 @@ Builds a module from its plan file one task at a time — real code, real tests, - Return to Step 0's task selection and take the next `[ ]` task in the same phase. ### Step 5 — The phase gate, then stop -- When every development task in the phase is closed, the one item left is `Task X.V`. Run the **whole** module's suite, not just this phase's tests — the gate exists to catch what this phase broke elsewhere. +- When every development task in the phase is closed, the one item left is `Task X.V`. Hand it to `test-and-verify` **as a gate run**, which widens it to the whole module's suite plus the project's type-checker and linter — the gate exists to catch what this phase broke elsewhere, and a targeted run can't see that. - Green: close the gate, set the phase to `Status: [x] Done`, update `Overall Progress`, append one dated Progress Log line saying what the phase delivered, and flip the matching sub-module in `docs/development-roadmap.md`. Status values are bare — a caveat worth recording goes in the Progress Log, never appended to a status. -- Red: the phase does not close. Fix the regression, or mark the offending task `[~]` and report it. A gate that gets closed over failing tests is worse than no gate. +- Red: the phase does not close. Fix the regression, or mark the offending task `[~]` and report it. A gate that gets closed over failing tests is worse than no gate, and a gate reporting a type-check or lint that never ran is worse still. - **Stop here.** Report what closed, what's still open, and what the next phase needs, then ask whether to continue. A phase boundary is the user's decision point, not yours. ## Reference files -- `references/verification.md` — finding or bootstrapping a test runner, what counts as verified, and what to do when a scenario can't be checked. Read this in Step 1. +- `references/verification.md` — bootstrapping a test runner when the project has none, what counts as verified, and what to do when a scenario can't be checked. Read this in Step 1. - `references/progress-updates.md` — the exact plan-file edits, the counting rules, and how progress flows back to `docs/development-roadmap.md`. Read this before the first write in Step 4. diff --git a/skills/execute-plan/references/verification.md b/skills/execute-plan/references/verification.md index 2595b77..2573d30 100644 --- a/skills/execute-plan/references/verification.md +++ b/skills/execute-plan/references/verification.md @@ -5,15 +5,12 @@ first task closes, and again whenever a scenario resists being checked. ## Finding the test runner -Look in this order and stop at the first real answer: +`test-and-verify` owns that search and the discovery order it follows — see its +`references/test-commands.md`. Don't run a second, divergent search here; hand it the run and take +its answer. -1. The root context file's `## Commands` section — `CLAUDE.md` or `AGENTS.md`. This is where a - previous pass of this skill records what it set up. -2. The project manifest — `package.json` scripts, `pyproject.toml`, `Makefile`, `go.mod`, `Cargo.toml`. -3. A CI workflow. What CI runs is what the project considers a pass, whatever the docs say. -4. An existing test directory. If tests exist, something runs them; find it before writing more. - -A command you found is better than one you invented, even when yours is tidier. +What matters at this level is only the fork: a runner exists, or none does. If one exists, this skill +never needs to know the command. If none does, the next section is yours. ## Bootstrapping one when there is none @@ -34,15 +31,19 @@ the first task rather than stopping. this yet — this is the step that closes that loop, and it belongs in `## Files Modified` like any other change. -## What counts as green +## What counts as green, for closing a plan item + +`test-and-verify` decides whether the run passed. This skill decides whether that pass closes a +checkbox — a narrower question, and the two can differ. -- The suite ran, you read its output, and it reported zero failures. -- The scenarios for *this* task are among the tests that ran. A suite that passes because it skipped - them is not a pass — check for skips, `.only`, and filters that quietly excluded new tests. -- Type-checks and linters pass where the project has them. Neither substitutes for a test run. +- The verdict is a pass, and the scenarios for *this* task are among the tests that actually ran. A + green run that skipped them closes nothing. +- A failing verdict closes nothing, however close it looked. Re-invoking the verifier to get past its + circuit breaker is not a second opinion, it's the same attempt with the safety removed. +- Type-checks and linters passing is not a substitute for a test run, and neither is a build. A pass you inferred, assumed, or reconstructed from a partial log is not a pass. If you can't see the -output, you can't close the item. +verdict, you can't close the item. ## One test per scenario, in the scenario's own terms diff --git a/skills/test-and-verify/SKILL.md b/skills/test-and-verify/SKILL.md new file mode 100644 index 0000000..60f1925 --- /dev/null +++ b/skills/test-and-verify/SKILL.md @@ -0,0 +1,50 @@ +--- +name: test-and-verify +description: Runs a project's tests, reads the real output rather than the exit code, and fixes what fails — a bounded diagnose-fix-rerun loop that stops after three remediation attempts instead of guessing on, and never weakens a test to reach green. Targets the suite relevant to the work, and adds the type-checker and linter at a phase gate. Reports a plain pass or fail verdict with the exact command it ran, and never edits a plan file — the caller marks the boxes. Use this skill whenever tests are to be run or a failure chased down: "run the tests", "run the test suite", "verify this", "check the tests pass", "fix the failing tests", "why is this test failing", or when `execute-plan` needs a task or a `Task X.V` gate verified. Use it even when the request looks like a one-liner — the bounded loop and the honest verdict are the point, not the command. Not for writing a feature or its tests from a plan — use execute-plan. Not for standing up a project's first test runner — use execute-plan. +--- + +# test-and-verify + +Runs the tests, reads what actually happened, and fixes what's broken — within a bounded number of attempts, then hands back. + +## When this runs +- `execute-plan` finishes a task and needs its scenarios proved, or reaches a `Task X.V` phase gate +- The user asks for the suite to be run, or for a failing test to be chased down +- Never to decide *what* to build, and never to write a feature's first tests from a plan — that's `execute-plan` + +## Core philosophy +- **A pass you didn't read isn't a pass.** Run the command, read stdout and stderr, and check the counts. A suite that "succeeded" because it collected zero tests, skipped the new ones, or exited 0 on a crash is a failure wearing a green hat. +- **Three attempts, then stop.** The loop is bounded on purpose. A fourth attempt is guessing, and guessing costs tokens and makes the diff worse. Hand back with what you know. +- **Fix only what the failure names.** Not the code around it, not a style you'd have written differently, not a passing test you find unconvincing. Unrelated findings get reported, never fixed in passing. +- **The failure decides what to fix — application code or the test.** These are different bugs, and picking wrong makes both worse. Say which one you concluded and why before you change anything. +- **Never edit a plan file.** Not a glyph, not a count, not the Progress Log. This skill produces a verdict; `execute-plan` writes it down. Two writers on one file is how a plan stops being trustworthy. +- **Never weaken a test to make it pass.** Deleting an assertion, loosening a matcher, adding a skip, or narrowing the run until the failure is out of scope is not a fix — it's hiding the bug and reporting green. + +## Workflow + +### Step 1 — Find the command, and the right scope for it +- If the caller supplied a command, use it. Otherwise discover it — see `references/test-commands.md`. +- **Default to the tests relevant to the work at hand**, not the whole suite. A targeted run is faster and its output is readable. +- **A `Task X.V` phase gate is the exception**: run the whole module's suite, plus the project's type-checker and linter if it has them. The gate exists to catch what this phase broke somewhere else, and a targeted run cannot see that. +- If no runner exists at all, say so and hand back. Setting one up is a project decision that belongs to `execute-plan`, not a remediation step. + +### Step 2 — Run it and read the output +- Run the command. Capture stdout and stderr, and read them — the exit code alone is not the result. +- Confirm the tests you expected actually ran. Check the reported counts, and check for skips, filters, or `.only` that quietly excluded them. +- Green, with the expected tests genuinely run: stop here and report. Don't keep going, and don't tidy anything on the way out. + +### Step 3 — Diagnose, fix, re-run — at most three times +- Read the failure properly: the assertion, the stack trace, the actual-versus-expected. Guessing from the test's name is how the wrong file gets edited. +- Decide whether the bug is in the **application code** or in the **test** — a bad mock, a wrong assertion, a fixture that drifted — and say which before changing anything. See `references/remediation.md`. +- Apply one targeted fix, then re-run. One change per attempt, so the next run tells you something. +- **After the third failed attempt, stop.** Report what failed, what you tried, what you ruled out, and your best read on the cause. Handing back a clear dead end beats a fourth guess. + +### Step 4 — Report the verdict +- State plainly: passed or failed, the exact command run, the counts, and what the caller may now do. +- On a gate run, report the type-check and lint results too, and say explicitly if the project has neither rather than implying they passed. +- On failure, never imply partial success. A caller that reads "mostly passing" as green is the failure mode this whole skill exists to prevent. +- Report anything you noticed but deliberately did not fix. See `references/remediation.md` for both report shapes. + +## Reference files +- `references/test-commands.md` — where to find the right command per ecosystem, targeted versus full-suite invocation, and the type-check and lint commands a gate adds. Read this in Step 1 when no command was supplied. +- `references/remediation.md` — telling an application bug from a test bug, what a targeted fix looks like, the circuit breaker, and the exact pass and fail report formats. Read this before the first fix in Step 3. diff --git a/skills/test-and-verify/references/remediation.md b/skills/test-and-verify/references/remediation.md new file mode 100644 index 0000000..e812654 --- /dev/null +++ b/skills/test-and-verify/references/remediation.md @@ -0,0 +1,92 @@ +# Remediation and reporting + +Read this before the first fix in Step 3. + +## Application bug or test bug + +Every failure is one or the other, and picking wrong makes both worse — a real bug gets papered over +by "fixing" the test, or a correct implementation gets bent to satisfy a broken assertion. Decide +before editing, and say which you concluded. + +**It's the application code when** the test states the behavior the project's own docs or the plan's +scenario describe, and the code does something else. The test is the specification here; it wins. + +**It's the test when** the behavior is right and the check is wrong — a mock that no longer matches +the real signature, a hard-coded date or ID that drifted, an assertion on incidental output like key +order or whitespace, a fixture that was never updated after a deliberate change. + +**When you can't tell, it's the application code.** That default fails safe: a real bug stays visible +instead of being erased. + +**When the test contradicts the project's stated rules** — `docs/product.md`, the root context file, a +plan scenario — that's neither. Stop and report it. A scenario is a decision someone made, and +overruling it is not a remediation. + +## What a targeted fix looks like + +- **One change per attempt.** Change two things and a green run tells you nothing about which mattered. +- **The smallest edit that addresses the diagnosed cause.** Not a refactor that happens to include it. +- **Nothing outside the failure's blast radius.** A passing test you find unconvincing, a helper you'd + have written differently, a type you'd tighten — report those, don't touch them. + +Never do these, whatever the pressure to get to green: + +- Delete or comment out an assertion, or loosen a matcher until it stops discriminating +- Add a skip, an `.only`, or a filter that removes the failure from the run +- Rewrite the expected value to whatever the code currently produces +- Widen a timeout to hide a race rather than fixing the race + +Each turns a red suite green while leaving the bug in the codebase, and the report that follows is +then false. If the honest outcome is "still failing", that is the outcome. + +## The circuit breaker + +Three remediation attempts, then stop — the run itself plus at most three fixes. + +The bound is not arbitrary: past three, the failure almost always means the diagnosis was wrong rather +than the fix was, and further attempts edit more files on a premise that was never true. Handing back +a clear dead end is more useful than a fourth guess and a wider diff. + +If an attempt makes things worse — new failures that weren't there before — revert that change before +handing back. The caller should receive the codebase no worse than you found it. + +## Report formats + +Both are terminal output for a caller. Be exact about the command and the counts; those are what the +caller acts on. + +**Passing:** + +```text +✅ Verification Passed +- Target: Task 2.1 (or: Phase 2 Gate) +- Executed: npm test -- test/mark-done.test.js +- Result: 3 passed, 0 failed, 0 skipped +- Checks: type-check clean · lint clean (gate runs only) +- Action: Task 2.1 and its scenarios may be marked [x]. +``` + +**Failing:** + +```text +❌ Verification Failed — circuit breaker after 3 attempts +- Target: Task 2.2 +- Executed: npm test -- test/mark-done-endpoint.test.js +- Result: 2 passed, 1 failed +- Failing: Scenario 2.2b — refusal does not name the pickup date +- Diagnosed: application code — the 409 body carries a generic message +- Attempted: (1) returned the date in the body (2) ... (3) ... +- Ruled out: the test's assertion matches docs/product.md's wording +- Action: Task 2.2 stays [ ]. Handing back — the remaining failure looks like + . +``` + +Two rules about the wording: + +- **Never imply partial success on a failing run.** "Mostly passing", "just one left", "nearly there" + all get read as green by a caller deciding whether to close a box. +- **Name what you noticed and left alone.** An unrelated bug, a test you think is weak, a skipped + suite you didn't own — the caller needs those, and they are not yours to fix silently. + +If the project has no type-checker or linter, write that rather than omitting the `Checks` line. An +absent line reads as a check that passed. diff --git a/skills/test-and-verify/references/test-commands.md b/skills/test-and-verify/references/test-commands.md new file mode 100644 index 0000000..bbe5faf --- /dev/null +++ b/skills/test-and-verify/references/test-commands.md @@ -0,0 +1,60 @@ +# Finding and scoping the test command + +Read this in Step 1 when the caller didn't supply a command. + +**This file is expected to age.** Runners, flags, and defaults shift. A command you read out of the +project beats one you read out of this table, every time — the table is a fallback for when the +project doesn't say, not an override for when it does. + +## Discovery order + +Stop at the first real answer: + +1. **The root context file's `## Commands` section** — `CLAUDE.md` or `AGENTS.md`. In this plugin's + chain, `execute-plan` records the command there once it sets a runner up. It is the intended answer. +2. **The project manifest** — `package.json` scripts, `pyproject.toml`, `Makefile`, `Cargo.toml`, + `go.mod`, `build.gradle`. A `test` script here is the project's own answer. +3. **A CI workflow** — `.github/workflows/*`, or whatever the project uses. What CI runs is what the + project actually considers a pass, whatever the docs claim. +4. **An existing test directory.** If tests exist, something runs them. Infer from the file naming and + imports before inventing anything. + +If none of these answer, say so and hand back. Standing a test runner up is a project decision — it +picks tooling, edits a manifest, and belongs to `execute-plan`, not to a verification pass. + +## Running a subset versus the whole suite + +**Default to the tests covering the work at hand.** Most runners take a path or a name filter: + +| Ecosystem | Whole suite | Targeted | +|---|---|---| +| npm / pnpm / yarn | `npm test` | `npm test -- `, or the runner directly | +| Node built-in | `node --test` | `node --test ` | +| Vitest / Jest | `npx vitest run` / `npx jest` | add a path, or `-t ""` | +| pytest | `pytest` | `pytest `, `pytest -k ""` | +| Go | `go test ./...` | `go test ./pkg/...`, `-run ` | +| Cargo | `cargo test` | `cargo test ` | + +Two cautions that cost real time: + +- **`npm test -- ` only forwards correctly if the script passes arguments through.** When a + `test` script is a chain (`lint && jest`), the path lands on the wrong command. Invoke the runner + directly instead. +- **A filter that matches nothing usually exits 0.** "No tests found" is not a pass. Always check the + reported count against what you expected to run. + +## What a phase gate adds + +A `Task X.V` gate runs the **whole** module's suite — a targeted run cannot see what this phase broke +elsewhere, which is the only reason the gate exists. On top of that, run whichever of these the +project actually has: + +| Check | Typical command | +|---|---| +| Type-check | `tsc --noEmit`, `mypy`, `go vet`, `cargo check` | +| Lint | `eslint .`, `ruff check`, `golangci-lint run`, `cargo clippy` | + +Look for these the same way you looked for the test command — the manifest's scripts and the CI +workflow, not assumption. **If the project has no type-checker or no linter, say so in the report.** +Silence reads as "it passed", and a gate that reports a check it never ran is worse than one that +admits the gap.