Minitest by plan, not by luck — and a W2.B baseline that can see

[pstrzalk]

Production project 40 (2026-09-06) was asked for "all business logic covered by automated tests" and got an RSpec suite: spec/ paths in all six revision prompts, rspec-rails in the Gemfile, and 62 examples that verification never ran — rails_test is gated on test/**/*_test.rb, and there were none. The same runs showed the W2.B route-smoke baseline dying in ~240 ms with Bundler::GemNotFound on every revision after the gem was added, so the baseline was blind exactly when it mattered.

Four commits, ordered by where the decision is actually made.

1. The planner picks the framework (1cca8b2)

The code agent's rules already said "Minitest, not RSpec". The log shows it naming the conflict — "The conventions say 'Minitest, not RSpec' but the task says to create spec/models/user_spec.rb — this is a conflict" — and then following the task. So the rule goes where the choice is made, in the positive-only register both planner prompts already use: Minitest under test/, the file layout spelled out, and the test stack declared complete so the planner stops inventing SimpleCov.

Measured with bin/inspect-plan-application-creation on project 40's exact intent, same model (anthropic/claude-haiku-4.5). Baseline was 3 of 5 runs producing RSpec:

creation run	spec/	test/	rspec	extra gems
1	0	12	0	0
2	0	8	0	0
3	0	7	0	0
4	0	8	0	0
5	0	6	0	0

Modification planner (project 43, ×2): test/ only, spec/ and RSpec absent from both. 7 of 7 clean, no escalation needed. The five plans still read concrete and file-level — the rule did not make the planner terse.

2. The code agent translates rather than follows (7443727)

RevisionPrompt gains the anti-reflex line (Tests → Minitest + fixtures in test/, NOT RSpec/FactoryBot) and a rule that answers the conflict instead of restating the preference: write the equivalent Minitest tests under test/, the task's wording does not override this, add no test gem.

Verified with a real dev revision (project 37, instruction 61) whose prompt names spec/models/score_spec.rb. The commit carries test/models/score_test.rb — ScoreTest < ActiveSupport::TestCase, five cases covering both validations plus the zero boundary — no spec/ directory, no rspec in the Gemfile, and W2.4 ran all five checks including rails test. This time the agent did not report a conflict; it just wrote test/.

3. W2.B repairs the bundle before it smokes (9986ea3)

Each revision runs in a fresh docker run --rm that mounts the workspace and nothing else, and BUNDLE_PATH=/usr/local/bundle dies with the container. A gem an earlier revision's agent installed is therefore in the lockfile and nowhere else, and W2.B — which runs before the agent — is the first step to walk into the hole: no per-route data, which is the one thing the baseline exists to produce, plus a re-download and a wasted agent turn every revision.

AutoRemediate.ensure_bundle runs bundle check and, on failure, the recipe W2.AR already applies after a failed W2.4 — the same install, moved before the damage instead of after it. W2.B smokes only once the bundle is whole, and says so in the sentinel when it is not. VerifyRevision.tally gives the three callers that build {checks, passed, failed} one place to do it.

Verified on a scratch copy of project 43 whose lockfile names an uninstalled gem: bundle check FAIL (174 ms) → ensure_bundle → false, ["bundler missing gems: ran \bundle install`"]→bundle checkPASS (225 ms) →route smokePASS (1 891 ms, 5 runs — Rails booted, which it could not have done before). The install-failed branch was exercised too, by accident: a Gemfile/lockfile mismatch under frozen mode makesbundle installrefuse,ensure_bundlereturns[]fixes, and W2.B skips the smoke and records that rather than running it intoBundler::GemNotFound`.

4. Docs (bc25098)

W2.B in the canon, both CLAUDE.md bullets, and three follow-ups: the per-project gem volume (with its two open decisions — lifecycle on project deletion, behaviour across image upgrades), the mechanical Minitest guard the measurement did not justify building, and the fact that a revision writing no tests leaves no trace in metrics["verify"] at all.

Verification

• bin/rails test — 700 runs, 2 706 assertions, 0 failures
• bin/rubocop — 187 files, no offenses
• E2E_GENERATE=1 bin/rails test test/integration/generate_todo_list_test.rb — green, 652 s (budget 1 200), 42 assertions. Every revision: W2.B: PASS bundle check, PASS route smoke | W2.4: PASS bundle check, PASS db:prepare, PASS zeitwerk check, PASS route smoke, PASS rails test, applied absent as asserted
• bin/verify-workspace ~/projects/hifumi-dev-workspaces/project_* — 26/29, unchanged; the same three pre-existing failures (27, 39, 7)

Not in this PR

• Phase 5, the mechanical spec/ guard — its trigger never fired. Recorded in follow-ups as the fallback if the prompt rule regresses.
• Apps that already use RSpec (production project 40). Their future modifications get the Minitest rule; the mixed framework in that one app is accepted.
• Running RSpec in verification. The fix is that RSpec stops being produced.
• The per-project gem volume. W2.B reinstalls correctly today; the volume only saves the repeated download, and it has two lifecycle decisions to make first.

Still open after merge

The production proof: start a project whose plan legitimately adds a gem, then check that the first revision after it shows bundle check=false, applied=["bundler missing gems: ran \bundle install`"]androute smoke=trueabove 800 ms, with noFetching ` inside the agent's turn.

Plan: thoughts/shared/plans/2026-09-07/minitest-planning-and-baseline-bundle.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01RJNJrvTLzXwvxpgDVXEnKp