Modification planner workspace context - #44
Merged
Conversation
The modification planner (PlanApplicationModification::AdHocLLM) plans against nothing but the user's sentence and invents file paths, columns and CSS variables — stored plans reference app/assets/stylesheets/application.tailwind.css (does not exist) and "assume a User model with Devise" on apps with no users. AppState.build(workspace:) describes a generated app from the workspace itself: gems beyond the skeleton Gemfile, tables condensed from db/schema.rb (falling back to the db/migrate/ listing when the schema has not been written yet — project_3 has two migrations and no schema, so "no tables" would be a confident falsehood), config/routes.rb verbatim, every .rb/.erb/.js/.css under app/ minus app/assets/builds/, and the four docs/ files with the init_docs_baseline placeholders filtered out. The docs cap is per file (DOC_FILE_CAP = 8_000 chars), deliberately not on the total: a body cap evicts whichever file is last, and on the two largest workspaces that is frontend.md, the only record of the app's palette. Character slice rather than byteslice — the docs are full of em dashes and an invalid encoding fails JSON serialization, not the LLM call. Deliberately not RevisionPrompt: that builds the implementer's prompt, globs only app/controllers + app/models (misses project_30's app/concerns/), and carries no schema or Gemfile diff. The W2.6 docs-writer prompt gains one rule naming the same 8000-character budget, so the docs agent condenses instead of appending. Hardcoded there because revision_workflow.rb runs as a Roast subprocess outside the autoloader. Not wired to anything yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
The context: seam has been plumbed and consumed nowhere since 2026-04-18 —
ModifyApplication passed { project_id: } and AdHocLLM bound it as _context.
It now carries { app_state: AppState.build(...) }, and the planner renders it
after "Intent:" (request first, reference material after, as RevisionPrompt
does).
Built in the tool rather than the planner so AdHocLLM stays a pure function of
its arguments, and so the file reads sit inside the existing rescue
StandardError: an unreadable workspace file degrades to a chat-safe error hash
instead of orphaning the tool_use. The new tool test exercises exactly that path
(chmod 000 Gemfile → error hash, Rails.error.report, nothing persisted, planner
never called).
The no-context prompt is byte-identical to before, asserted. This is the first
deliberate divergence from the PlanApplicationCreation twin, which cannot have
workspace context — CreateApplication persists the plan before
ExecuteInstructionJob runs rails new.
Verified live on project_42: the plan now names app/views/standups/index.html.erb
with its real markup and the office template's #0052CC, where the blind planner
produced app/views/entries/*.html.erb and var(--accent). Tool call 3.7s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
bin/inspect-plan-application-creation has been broken since the RubyLLM v2 upgrade: require "ruby_llm/schema" raises LoadError (the constant moved to Schematist::Schema, which app/schemas/plan_schema.rb requires itself), and it called PlanApplicationCreation.call without the now-required model: kwarg. It now loads the environment itself (so it runs directly and under kamal app exec), defaults model: to the plan_creation stage default from LLM::Stages, and takes --model and --help. The no-argument three-intent suite stays. bin/inspect-plan-application-modification is new: it dry-runs the modification planner against a real project's workspace, printing the AppState payload with its size before the plan, so a prompt change can be read next to its output. --blind omits the snapshot — the same-session A/B that gathered the evidence for this work; it reproduces the phantom-authorization and guessed-path defects on demand. Persists nothing. bin/inspect-plans was untracked despite being the source of the stored evidence and the header convention the new script mirrors; committed here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
Line 11 instructed the planner to "assume Tailwind, Hotwire, Devise, and the template's design tokens are already wired" and to reference --accent and --paper-100. Both claims are false for generated apps: Devise is not in the skeleton Gemfile (the implementer's prompt says "NOT Devise"), and those tokens are hifumi.dev's own, not the app's. With ground truth now in the user turn the line did not merely mislead, it contradicted the payload. The prompt now points at the "Current application state" block, and says only what IS there: a default Rails 8 installation plus Tailwind and Hotwire, plan with what the Gems section names, has_secure_password for sign-in. No gem-absence claims (a Gemfile scan cannot back "no bcrypt" on a Devise app), no Propshaft/Importmap/Solid enumeration (they are what a default install is), and no token name quoted even negatively — Haiku mimics literals. Adds an explicit anti-hedging rule aimed at the "if it uses a CSS class… whichever matches your design" failure. Verified with bin/inspect-plan-application-modification on projects 42, 30 and 39: no phantom authorization, no Devise on the has_secure_password app, office-template hexes instead of var(--accent), no "if it uses" / "whichever" / "assuming" in any output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
The creation planner cannot be given workspace context (the workspace does not exist when it runs), so its prompt is the only lever, and line 6 carried the same false "Devise gems available" claim as the modification prompt. It now says what the workspace IS — a default Rails 8 app with Tailwind and Hotwire on the default Gemfile — and names has_secure_password plus sessions as the route for sign-in, matching RevisionPrompt.stack_inventory_section so planner and implementer stop disagreeing. Verified with bin/inspect-plan-application-creation: "a blog with authors who sign in" plans has_secure_password + a SessionsController; the built-in three-intent suite produces 5/6/6 revisions with no Devise mention. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
W4.1 "Load app manifest (docs/)" was annotated "delegated to CreatePlan in Phase 2" since the canon was written and never built; D3 listed three research tiers without recording which one exists. Both now state what is there: W4.1 is AppState.build assembled by ModifyApplication, only D3[a] is implemented, and D3[c] (reads files) was measured — same plan quality at 3–8× the latency, with un-sandboxed file access in the generator container — and deferred. CLAUDE.md gains a Conventions bullet so a reader learns where planner context comes from, why the creation planner stays blind, that the 8000 budget lives in two places, and which bin/inspect-plan-application-* script verifies a prompt change. The user-journey W4 table's step label follows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
Every brakeman run has reported one obsolete ignore entry since the gem moved past 8.0.4: the "Command Injection" fingerprint for ExportToGithubJob's array-form Open3.capture3 push. Brakeman 8.0.6 no longer raises that warning — the argv form involves no shell — so the entry matched nothing. CI does not fail on obsolete entries, but the noise hid whether the ignore file was current. Three entries remain, all for the Roast-side shell-outs whose interpolations come from hardcoded tables. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
It had failed in 0.11s since Phase 4 made sign-in mandatory: the POST redirected to the login page, no project was created, and Project.order(:id).last landed on a fixture whose instruction is "implementing". The test now signs in through Devise's helpers and finds the project through the signed-in user — fixture ids are large hashes, so ordering by id would still pick a fixture. Two more things had rotted underneath it. Templates::Picker.pick is a RubyLLM call the test's fake key cannot make — pinned to "office", with apply left real so frontend.md and the font <link> land as in production. And the auto-recap nudge that instruction.completed injects re-entered the Chat#complete stub, which would have started a second identical build; the stub now answers that turn with text only, as the real prompt requires. perform_enqueued_jobs is scoped to ChatRespondJob and ExecuteInstructionJob: StopPreviewJob would drive Docker for a preview that was never started. Verified 2026-09-03: E2E_GENERATE=1 run green in 550s (budget 900s), three revisions completed, generated app's own suite passing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
Minor bump: change requests are now planned against the real workspace, two planner probe scripts arrived, and both planner prompts stopped asserting Devise and hifumi.dev's own design tokens exist in generated apps. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBkAFYRa8XRPp2KMJDoQYy
5cbe8b2 repaired test/integration/generate_todo_list_test.rb and ran it green (550s, three revisions, generated suite passing), so the 2026-05-14 entry describing it as broken no longer holds. The other entry under that date stays.
…er review Nine review findings on the modification-planner workspace context, each discussed and fixed in place: - AppState.read_workspace_file backs every workspace read: realpath containment (a symlink out of the workspace reads as absent) and encoding scrub (a stray byte no longer raises and dead-ends the project) - docs/ bodies are fenced with four backticks so their `#`/`##` headings cannot outrank the payload's `###` sections - AppState::PLACEHOLDER is the single source (the job and the test helper interpolate it) and detection is a whole-body match, so a populated doc carrying the baseline line is kept - the Rails version in the gems line comes from the skeleton Gemfile - modification prompt: `app/` is a list of paths, `docs/frontend.md` is the palette source; AdHocLLM appends NO_STATE_NOTE when no snapshot is given, so the "do not hedge" rule is conditional at the moment it is false; `--blind` header says what it measures - W2.6 length rule states its precedence over "do not rewrite whole files" and protects frontend.md - E2E generator test: budget 1200 asserted before the generated suite runs, frontend.md checked for the template name rather than the H1 - both planner test files assert the prompt invariant (no Devise-class gems, no hifumi tokens, no default-stack enumeration; positive framing present) - docs: post-launch review no longer calls the E2E test broken, the prompt-injection idea records the planner as a second channel, W4/D3 canon states the real [a]/[c] distinction, CHANGELOG describes the snapshot and the three plan scripts accurately, README documents them - tests tightened: file-listing order, DOC_FILE_CAP <-> W2.6 prompt guard, stray schema line, all-empty build, rescue-scope via stub instead of chmod Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117StvHb26xv3MUepWHhcWy
Owner
Author
Review summaryMulti-agent review of this PR against the plan ( Fixed
Considered and not flagged
Full suite 627 runs green, rubocop and brakeman clean. Review generated with multi-agent analysis |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.