Minitest by plan, not by luck — and a W2.B baseline that can see

CLAUDE.md

@@ -57,6 +57,6 @@ Additional sources from the Phase 1 spike:
 - **LLM model selection**: never hardcode a model id at a call site — `lib/llm/stages.rb` (`LLM::Stages`, note the `LLM` acronym inflection) is the registry of the six LLM stages (chat, plan_creation, plan_modification, template, code, docs), their labels, factory defaults, and the curated `AVAILABLE_MODELS` list (full OpenRouter ids only, never `sonnet`/`haiku` aliases). Profiles store per-user defaults (`default_<stage>_model`), projects snapshot their own selection (`<stage>_model`) at creation; selectors live in the build tab (`model_selections/_pane`), the new-project form, and the account integrations pane. A new stage = registry entry + migration on both tables + threading at the call site. A new **model** = confirm the OpenRouter slug supports `structured_outputs` (plan/template stages) and `tools` (chat) → populate the RubyLLM registry in **every** environment → only then add the id to `AVAILABLE_MODELS`; that order is load-bearing, because RubyLLM owns its registry: it resolves against the `ruby_llm_models` table (its store) and falls back to the gem's bundled `models.json` only when that table is *empty*, so an id absent from both raises `ModelNotFoundError` on the four RubyLLM-backed stages. Check with `bin/verify-model-registry [candidate-id]`, fix with `RubyLLM.models.refresh!` plus a process restart (the registry is memoized per process); the refresh only ever adds or updates rows — per-provider fetch failures are logged and skipped, never allowed to empty the store. Full procedure in `docs/05-runbooks/03-llm-model-registry.md`. Selection applies on the OpenRouter path only — `roast_model_env` keeps claudesubscription runs on the operator's ENV/alias defaults, and an explicit `HIFUMI_DEV_MODEL`/`HIFUMI_DEV_DOCS_MODEL` always wins.
 - **Agent sandbox (tenant isolation)**: the codegen agent runs `claude` with `skip_permissions!` on user-controlled prompts — treat it as untrusted code execution, so in production it must NOT run in the shared generator container. `ExecuteInstructionJob#execute_revision` wraps the roast invocation via `Roast::Sandbox.wrap` (`lib/roast/sandbox.rb`, plain builder, returns the `docker run` argv) into a throwaway `--rm` container that mounts ONLY this project's workspace (no `workspace_root`, no `/var/run/docker.sock`), runs entirely as the unprivileged `generator` user (`--user`, `--cap-drop=ALL`, zero cap-adds — uniform uid avoids the capless-root/mixed-ownership deadlock of issue #24; the workspace is re-relaxed `a+rwX` before every sandboxed run), and forwards env (incl. `OPENROUTER_API_KEY` and the per-project `HIFUMI_DEV_*` model selection) **by name** so secrets never hit argv. Gated by `sandboxed?` (production OR `FORCE_AGENT_SANDBOX=1`); dev stays direct (single-tenant, Claude-subscription transport, no Docker). Image = the generator's own, from `HIFUMI_AGENT_IMAGE` (set in `deploy.yml`). Not runtime-verifiable on macOS — see the verification checklist + residuals (generator-side socket, egress) in `docs/09-ideas/05-followups.md`.
 - **Preview infrastructure**: `lib/preview/preview_manager.rb` (plain Ruby, not a Roast workflow) drives Docker. `lib/preview/Dockerfile{,.base}` are owned by this repo — never read from generated apps. `lib/preview/skeleton/` is the canonical fresh-Rails-app baseline copied into every workspace; regenerate with `bin/preview-regen-skeleton` when bumping Rails. Rebuild the base image with `bin/preview-rebuild-base` after Gemfile changes. The `preview-internal` Docker network is created without `--internal` on Docker Desktop (host port mapping wouldn't work otherwise) — Phase 4 reintroduces strict egress isolation on a Linux production host. In remote mode `PreviewManager#run_container` passes `PREVIEW_HOST=<id>.preview.<domain>` to the container; the skeleton-overlay's `preview_iframe.rb` initializer appends it to `Rails.application.config.hosts` so Rails 8's dev HostAuthorization doesn't 403 the kamal-proxy request.
-- **Modification planner context**: `PlanApplicationModification::AdHocLLM` plans against a snapshot of the real workspace, built by `AppState.build(workspace:)` in `lib/app_state.rb` — gems beyond the default Gemfile, database tables (or the `db/migrate/` listing while `db/schema.rb` is not written yet), `config/routes.rb` verbatim, every `.rb/.erb/.js/.css` under `app/` minus `app/assets/builds/`, and the four `docs/` files. `ModifyApplication#execute` assembles it (inside its `rescue StandardError`, so an unreadable file degrades to a chat-safe error hash) and passes `context: { app_state: }`; the planner renders it after `Intent:`. The docs cap is **per file** (`AppState::DOC_FILE_CAP`, 8 000 chars), never on the total — a body cap evicts whichever file is last, and that is `frontend.md`, the only record of the app's palette. The W2.6 docs-writer prompt in `lib/roast/revision_workflow.rb` hardcodes the same 8 000 (it runs as a Roast subprocess, outside the autoloader) — change both together. The **creation** planner stays blind by ordering, not choice: `CreateApplication` persists the plan before `ExecuteInstructionJob` runs `rails new`, so no workspace exists yet and its system prompt is its only lever. Both planner prompts say only what IS there ("default Rails 8", Tailwind, Hotwire, `has_secure_password` for sign-in) — no gem-absence claims, no Propshaft/Importmap/Solid enumeration, no `--accent`-style token names. Verify any planner change with `bin/inspect-plan-application-modification <project_id> "<intent>" [--blind]` and `bin/inspect-plan-application-creation "<intent>"`: both dry-run the planner and persist nothing, and `--blind` is the same-session A/B against planning from the intent alone.
-- **W2.4 verification**: `VerifyRevision` (`lib/roast/verify_revision.rb`) runs five checks in order — `bundle check` (lockfile satisfied; the only short-circuit, everything after it would repeat the same error), `db:prepare` (boot + migrations; writes `db/schema.rb`, which the test-env checks need), `zeitwerk:check` (every `app/` file loads — the only check that sees code no route and no test touches), route smoke (every static GET page requested once in the test env, 10 s per request) and `rails test` (when tests exist). All are blocking except route smoke, which is **advisory**: it still fails W2.4 and enters W2.AR/W2.R with the exact exception, but if two fix attempts don't resolve it the revision commits anyway (W2.F0) instead of being reset. **W2.B** runs route smoke alone at the parent commit before the agent, and W2.4 skips the pages it recorded — so only breakage new in this revision reaches the fix agent and a page committed broken is never re-remediated. Route smoke copies `lib/roast/route_smoke.rb` + `route_smoke_check.rb` into the workspace's `tmp/hifumi/` for one `bin/rails test tmp/hifumi/route_smoke_test.rb` and removes the directory afterwards (four April-era workspaces don't gitignore `tmp/`); a fix-prompt hint tells the agent that file is the verifier's. Every verification run (W2.B, W2.4, W2.AR, W2.RV × 2) is printed as a `[HIFUMI:VERIFY]` JSON line — Roast relays cog output through its own logger on **stderr**, decorated (`I, [ts]  INFO -- ruby(:verify) ❯ …`), so `VerifyReport.parse_line` finds the prefix anywhere in the line and `ExecuteInstructionJob` scans both streams — and lands in `revision.metrics["verify"]` with per-check pass/fail, tier, duration, capped error text and failing paths. Fix-agent error input is capped head-and-tail at `ERROR_CAP_CHARS` (4 000). Standalone, **dev only** (it executes the workspace's code unsandboxed): `bin/verify-workspace [--check NAME] [--known-failing PATH,PATH] <workspace>...`. `boot_check` and `herb_lint` were removed 2026-09-06 (dominated / never ran); the as-built note in `docs/02-architecture/01-workflows-and-decisions.md` records why herb was rejected rather than revived.
+- **Modification planner context**: `PlanApplicationModification::AdHocLLM` plans against a snapshot of the real workspace, built by `AppState.build(workspace:)` in `lib/app_state.rb` — gems beyond the default Gemfile, database tables (or the `db/migrate/` listing while `db/schema.rb` is not written yet), `config/routes.rb` verbatim, every `.rb/.erb/.js/.css` under `app/` minus `app/assets/builds/`, and the four `docs/` files. `ModifyApplication#execute` assembles it (inside its `rescue StandardError`, so an unreadable file degrades to a chat-safe error hash) and passes `context: { app_state: }`; the planner renders it after `Intent:`. The docs cap is **per file** (`AppState::DOC_FILE_CAP`, 8 000 chars), never on the total — a body cap evicts whichever file is last, and that is `frontend.md`, the only record of the app's palette. The W2.6 docs-writer prompt in `lib/roast/revision_workflow.rb` hardcodes the same 8 000 (it runs as a Roast subprocess, outside the autoloader) — change both together. The **creation** planner stays blind by ordering, not choice: `CreateApplication` persists the plan before `ExecuteInstructionJob` runs `rails new`, so no workspace exists yet and its system prompt is its only lever. Both planner prompts say only what IS there ("default Rails 8", Tailwind, Hotwire, `has_secure_password` for sign-in) — no gem-absence claims, no Propshaft/Importmap/Solid enumeration, no `--accent`-style token names. In the same positive register, both pin tests to **Minitest under `test/`** with the file layout spelled out and the test stack declared complete ("plan no additional testing gems, frameworks or coverage tools") — the framework is chosen in the plan, and the code agent follows a task's file paths over its own rules, which is how production project 40 got an RSpec suite `rails test` never ran. The prompt tests pin the framing by refusing `rspec|spec/|factory_bot|simplecov` in either prompt. Verify any planner change with `bin/inspect-plan-application-modification <project_id> "<intent>" [--blind]` and `bin/inspect-plan-application-creation "<intent>"`: both dry-run the planner and persist nothing, and `--blind` is the same-session A/B against planning from the intent alone.
+- **W2.4 verification**: `VerifyRevision` (`lib/roast/verify_revision.rb`) runs five checks in order — `bundle check` (lockfile satisfied; the only short-circuit, everything after it would repeat the same error), `db:prepare` (boot + migrations; writes `db/schema.rb`, which the test-env checks need), `zeitwerk:check` (every `app/` file loads — the only check that sees code no route and no test touches), route smoke (every static GET page requested once in the test env, 10 s per request) and `rails test` (when tests exist). All are blocking except route smoke, which is **advisory**: it still fails W2.4 and enters W2.AR/W2.R with the exact exception, but if two fix attempts don't resolve it the revision commits anyway (W2.F0) instead of being reset. **W2.B** runs at the parent commit before the agent — `AutoRemediate.ensure_bundle` (`bundle check`, then the W2.AR install recipe if it fails) and then route smoke — and W2.4 skips the pages it recorded, so only breakage new in this revision reaches the fix agent and a page committed broken is never re-remediated. The bundle step is not optional: the sandbox container is throwaway, so a gem an earlier revision's agent installed is in the lockfile but not in this container's `BUNDLE_PATH`, and the smoke would die on `Bundler::GemNotFound` with no per-route data — blind exactly when a gem has been added (production project 40). When the install itself fails, W2.B skips the smoke and records that; the baseline's sentinel carries both checks plus `applied`. Route smoke copies `lib/roast/route_smoke.rb` + `route_smoke_check.rb` into the workspace's `tmp/hifumi/` for one `bin/rails test tmp/hifumi/route_smoke_test.rb` and removes the directory afterwards (four April-era workspaces don't gitignore `tmp/`); a fix-prompt hint tells the agent that file is the verifier's. Every verification run (W2.B, W2.4, W2.AR, W2.RV × 2) is printed as a `[HIFUMI:VERIFY]` JSON line — Roast relays cog output through its own logger on **stderr**, decorated (`I, [ts]  INFO -- ruby(:verify) ❯ …`), so `VerifyReport.parse_line` finds the prefix anywhere in the line and `ExecuteInstructionJob` scans both streams — and lands in `revision.metrics["verify"]` with per-check pass/fail, tier, duration, capped error text and failing paths. Fix-agent error input is capped head-and-tail at `ERROR_CAP_CHARS` (4 000). Standalone, **dev only** (it executes the workspace's code unsandboxed): `bin/verify-workspace [--check NAME] [--known-failing PATH,PATH] <workspace>...`. `boot_check` and `herb_lint` were removed 2026-09-06 (dominated / never ran); the as-built note in `docs/02-architecture/01-workflows-and-decisions.md` records why herb was rejected rather than revived.
 - **RubyLLM tools must be idempotent within a user turn**. RubyLLM's tool loop will sometimes call the same tool twice in adjacent assistant messages before either result lands, producing the order `assistant(use_X) → assistant(use_Y) → user(result_X) → user(result_Y)` — illegal for Anthropic, and the chat permanently rejects every subsequent message. **There is no tool-side guard any more** — the tool that carried one was deleted in `13e9c1c`. What stands today is a prompt rule (*"Call each tool AT MOST ONCE per user turn"*, `app/prompts/generator_agent/instructions.txt.erb:19`) plus the `RubyLLM::BadRequestError` banner in `ChatRespondJob::FRIENDLY_ERRORS` that turns an already-corrupt chat into *"This conversation can't be continued by the model. Please start a new project."* So a new tool needs its own in-band guard: return an error hash rather than raising, so the second `tool_use` still gets a `tool_result` and history stays valid. Diagnose corrupt chats with `bin/inspect-chat <project_id>` (works on prod via `kamal app exec`).

app/prompts/plan_application_creation_system.md

@@ -4,6 +4,7 @@ Rules for the plan:
 - 3 to 6 revisions.
 - Each revision is one atomic, testable change ("add Product model with name/price", not "set up the shop").
 - The workspace is a default Rails 8 app with Tailwind and Hotwire, on the default Gemfile. Do NOT include `rails new`. If the app needs sign-in, plan a revision that adds it — `has_secure_password` plus sessions is the Rails-native default.
+- Tests are Minitest under `test/`, the Rails default, run with `bin/rails test`: `test/models/<name>_test.rb`, `test/controllers/<name>_controller_test.rb`, `test/integration/<flow>_test.rb`, fixtures in `test/fixtures/`. The default test stack is complete — plan no additional testing gems, frameworks or coverage tools. Every revision that adds or changes behaviour names the Minitest file(s) it adds.
 - Prefer Rails Way: scaffolds, concerns, validations over custom abstractions.
 - Mount the primary user-facing feature at the root path (`root to: "...#index"`) unless the user explicitly asked for a different landing page. The first revision that introduces that feature must set the root route.
 - When the plan introduces more than one user-facing feature, include a revision that adds a top or side navigation menu in `app/views/layouts/application.html.erb` linking to each feature — unless the user explicitly asked for a different navigation pattern (e.g. single-page, dashboard-only).

app/prompts/plan_application_modification_system.md

@@ -9,6 +9,7 @@ Rules for the plan:
 - DO NOT re-introduce models, controllers, or views that already exist. Reference existing files by path; describe modifications rather than scaffolds.
 - DO NOT add a navigation menu unless the user explicitly asks for one. Modify the existing navigation only when relevant.
 - The app is a default Rails 8 installation — everything Rails ships with is there and needs no setup. On top of that, Tailwind and Hotwire (Turbo + Stimulus) ARE installed: use them, and reach for Turbo Frames / Turbo Streams / Stimulus rather than hand-written fetch or a JS framework.
+- Tests are Minitest under `test/`, the Rails default, run with `bin/rails test`: `test/models/<name>_test.rb`, `test/controllers/<name>_controller_test.rb`, `test/integration/<flow>_test.rb`, fixtures in `test/fixtures/`. The default test stack is complete — plan no additional testing gems, frameworks or coverage tools. Every revision that adds or changes behaviour names the Minitest file(s) it adds.
 - Plan only with what the "Gems" section names plus what a default Rails 8 install ships. For sign-in, the Rails-native route is `has_secure_password` plus sessions.
 - Style with the literal Tailwind classes and hex values the app already uses. `docs/frontend.md` is the palette source; the file list tells you which views exist, not what is in them. Do NOT introduce CSS variables the app does not define.
 - Do NOT hedge. The application state is given to you. Never write "if it uses X…", "whichever matches your design", "assuming Y exists", or "verify that…". Name the actual file, the actual class, the actual colour.