From 332e683e12a64dd1541d52c0b616f033423df30d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Strza=C5=82kowski?= Date: Thu, 3 Sep 2026 23:17:36 +0200 Subject: [PATCH] fix: bake the workspace skeleton's bundle into the generator image Every workspace starts as a copy of lib/preview/skeleton, lockfile and frozen .bundle/config included, and the agent sandbox is the generator image itself. The generator's own lock had moved past the skeleton's (rails 8.1.3.1 vs 8.1.3, puma 8.0.2 vs 8.0.1, sqlite3 2.9.6 vs 2.9.3), so `bundle check` inside every sandbox failed and Bundler re-downloaded all 116 gems into the throwaway container, revision after revision. On project 36 (2026-09-03) that was ~9 of 32 minutes and an extra fix-agent call per step, with verify never reaching the tests before remediation; the follow-up modification paid 97s of its 173s. The build stage now installs the skeleton's bundle next to the generator's and runs `bundle check` against it, so a future drift fails the image build instead of billing every revision. Verified locally: both bundles resolve, and the skeleton's `bundle check` passes as root and as uid 1000. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0117StvHb26xv3MUepWHhcWy --- CHANGELOG.md | 16 ++++++++++++++++ CLAUDE.md | 2 +- Dockerfile | 19 ++++++++++++++++++- docs/09-ideas/05-followups.md | 3 +-- 4 files changed, 36 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 013b725..5bc5a9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,22 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [semantic versioning](https://semver.org/) (minor for new functionality, patch for fixes and internal changes). +## [1.5.1] - 2026-09-03 + +### Fixed + +- Every build step on hifumi.dev was silently paying for a full `bundle + install`. The generator image's gems had moved past the versions pinned in + the blank-app skeleton every project starts from, and the agent sandbox is + that same image, so Bundler inside the sandbox could not satisfy the + project's lockfile and downloaded all 116 gems again — in a container that + is discarded a minute later. On a six-step build that was about nine of + thirty-two minutes, an extra fix-agent call per step, and the verification + step never reaching the tests before remediation. The image now carries the + skeleton's bundle as well, and its build fails if the two ever drift again. + Self-hosters: rebuild the image; the build takes a little longer, every + build step afterwards is faster and cheaper. + ## [1.5.0] - 2026-09-03 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 3db86f2..6379251 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -55,7 +55,7 @@ Additional sources from the Phase 1 spike: - **Pin `.ruby-version` by writing the file** (Write tool), not via the version manager CLI (`frum local`, `rbenv local`, etc.). User wants to verify state from the file itself. - **Roast runner**: `bin/roast-claudesubscription` is the dev default (uses Claude Code subscription — wrapper unsets `ANTHROPIC_*` ENV + pins PATH to `.ruby-version` via frum). `bin/roast-openrouter` is the per-token alternative used in production and when `FORCE_OPENROUTER=1` in dev. `bin/roast` (the bundler binstub) calls `bundle exec roast` raw, no env setup — for direct testing only. `ExecuteInstructionJob` picks `-openrouter` in production / when `FORCE_OPENROUTER=1` / whenever sandboxed, else `-claudesubscription`. - **LLM model selection**: never hardcode a model id at a call site — `lib/llm/stages.rb` (`LLM::Stages`, note the `LLM` acronym inflection) is the registry of the six LLM stages (chat, plan_creation, plan_modification, template, code, docs), their labels, factory defaults, and the curated `AVAILABLE_MODELS` list (full OpenRouter ids only, never `sonnet`/`haiku` aliases). Profiles store per-user defaults (`default__model`), projects snapshot their own selection (`_model`) at creation; selectors live in the build tab (`model_selections/_pane`), the new-project form, and the account integrations pane. A new stage = registry entry + migration on both tables + threading at the call site. A new **model** = confirm the OpenRouter slug supports `structured_outputs` (plan/template stages) and `tools` (chat) → populate the RubyLLM registry in **every** environment → only then add the id to `AVAILABLE_MODELS`; that order is load-bearing, because RubyLLM owns its registry: it resolves against the `ruby_llm_models` table (its store) and falls back to the gem's bundled `models.json` only when that table is *empty*, so an id absent from both raises `ModelNotFoundError` on the four RubyLLM-backed stages. Check with `bin/verify-model-registry [candidate-id]`, fix with `RubyLLM.models.refresh!` plus a process restart (the registry is memoized per process); the refresh only ever adds or updates rows — per-provider fetch failures are logged and skipped, never allowed to empty the store. Full procedure in `docs/05-runbooks/03-llm-model-registry.md`. Selection applies on the OpenRouter path only — `roast_model_env` keeps claudesubscription runs on the operator's ENV/alias defaults, and an explicit `HIFUMI_DEV_MODEL`/`HIFUMI_DEV_DOCS_MODEL` always wins. -- **Agent sandbox (tenant isolation)**: the codegen agent runs `claude` with `skip_permissions!` on user-controlled prompts — treat it as untrusted code execution, so in production it must NOT run in the shared generator container. `ExecuteInstructionJob#execute_revision` wraps the roast invocation via `Roast::Sandbox.wrap` (`lib/roast/sandbox.rb`, plain builder, returns the `docker run` argv) into a throwaway `--rm` container that mounts ONLY this project's workspace (no `workspace_root`, no `/var/run/docker.sock`), runs entirely as the unprivileged `generator` user (`--user`, `--cap-drop=ALL`, zero cap-adds — uniform uid avoids the capless-root/mixed-ownership deadlock of issue #24; the workspace is re-relaxed `a+rwX` before every sandboxed run), and forwards env (incl. `OPENROUTER_API_KEY` and the per-project `HIFUMI_DEV_*` model selection) **by name** so secrets never hit argv. Gated by `sandboxed?` (production OR `FORCE_AGENT_SANDBOX=1`); dev stays direct (single-tenant, Claude-subscription transport, no Docker). Image = the generator's own, from `HIFUMI_AGENT_IMAGE` (set in `deploy.yml`). Not runtime-verifiable on macOS — see the verification checklist + residuals (generator-side socket, egress, bundle vendoring) in `docs/09-ideas/05-followups.md`. +- **Agent sandbox (tenant isolation)**: the codegen agent runs `claude` with `skip_permissions!` on user-controlled prompts — treat it as untrusted code execution, so in production it must NOT run in the shared generator container. `ExecuteInstructionJob#execute_revision` wraps the roast invocation via `Roast::Sandbox.wrap` (`lib/roast/sandbox.rb`, plain builder, returns the `docker run` argv) into a throwaway `--rm` container that mounts ONLY this project's workspace (no `workspace_root`, no `/var/run/docker.sock`), runs entirely as the unprivileged `generator` user (`--user`, `--cap-drop=ALL`, zero cap-adds — uniform uid avoids the capless-root/mixed-ownership deadlock of issue #24; the workspace is re-relaxed `a+rwX` before every sandboxed run), and forwards env (incl. `OPENROUTER_API_KEY` and the per-project `HIFUMI_DEV_*` model selection) **by name** so secrets never hit argv. Gated by `sandboxed?` (production OR `FORCE_AGENT_SANDBOX=1`); dev stays direct (single-tenant, Claude-subscription transport, no Docker). Image = the generator's own, from `HIFUMI_AGENT_IMAGE` (set in `deploy.yml`). Not runtime-verifiable on macOS — see the verification checklist + residuals (generator-side socket, egress) in `docs/09-ideas/05-followups.md`. - **Preview infrastructure**: `lib/preview/preview_manager.rb` (plain Ruby, not a Roast workflow) drives Docker. `lib/preview/Dockerfile{,.base}` are owned by this repo — never read from generated apps. `lib/preview/skeleton/` is the canonical fresh-Rails-app baseline copied into every workspace; regenerate with `bin/preview-regen-skeleton` when bumping Rails. Rebuild the base image with `bin/preview-rebuild-base` after Gemfile changes. The `preview-internal` Docker network is created without `--internal` on Docker Desktop (host port mapping wouldn't work otherwise) — Phase 4 reintroduces strict egress isolation on a Linux production host. In remote mode `PreviewManager#run_container` passes `PREVIEW_HOST=.preview.` to the container; the skeleton-overlay's `preview_iframe.rb` initializer appends it to `Rails.application.config.hosts` so Rails 8's dev HostAuthorization doesn't 403 the kamal-proxy request. - **Modification planner context**: `PlanApplicationModification::AdHocLLM` plans against a snapshot of the real workspace, built by `AppState.build(workspace:)` in `lib/app_state.rb` — gems beyond the default Gemfile, database tables (or the `db/migrate/` listing while `db/schema.rb` is not written yet), `config/routes.rb` verbatim, every `.rb/.erb/.js/.css` under `app/` minus `app/assets/builds/`, and the four `docs/` files. `ModifyApplication#execute` assembles it (inside its `rescue StandardError`, so an unreadable file degrades to a chat-safe error hash) and passes `context: { app_state: }`; the planner renders it after `Intent:`. The docs cap is **per file** (`AppState::DOC_FILE_CAP`, 8 000 chars), never on the total — a body cap evicts whichever file is last, and that is `frontend.md`, the only record of the app's palette. The W2.6 docs-writer prompt in `lib/roast/revision_workflow.rb` hardcodes the same 8 000 (it runs as a Roast subprocess, outside the autoloader) — change both together. The **creation** planner stays blind by ordering, not choice: `CreateApplication` persists the plan before `ExecuteInstructionJob` runs `rails new`, so no workspace exists yet and its system prompt is its only lever. Both planner prompts say only what IS there ("default Rails 8", Tailwind, Hotwire, `has_secure_password` for sign-in) — no gem-absence claims, no Propshaft/Importmap/Solid enumeration, no `--accent`-style token names. Verify any planner change with `bin/inspect-plan-application-modification "" [--blind]` and `bin/inspect-plan-application-creation ""`: both dry-run the planner and persist nothing, and `--blind` is the same-session A/B against planning from the intent alone. - **RubyLLM tools must be idempotent within a user turn**. RubyLLM's tool loop will sometimes call the same tool twice in adjacent assistant messages before either result lands, producing the order `assistant(use_X) → assistant(use_Y) → user(result_X) → user(result_Y)` — illegal for Anthropic, and the chat permanently rejects every subsequent message. **There is no tool-side guard any more** — the tool that carried one was deleted in `13e9c1c`. What stands today is a prompt rule (*"Call each tool AT MOST ONCE per user turn"*, `app/prompts/generator_agent/instructions.txt.erb:19`) plus the `RubyLLM::BadRequestError` banner in `ChatRespondJob::FRIENDLY_ERRORS` that turns an already-corrupt chat into *"This conversation can't be continued by the model. Please start a new project."* So a new tool needs its own in-band guard: return an error hash rather than raising, so the second `tool_use` still gets a `tool_result` and history stays valid. Diagnose corrupt chats with `bin/inspect-chat ` (works on prod via `kamal app exec`). diff --git a/Dockerfile b/Dockerfile index 59646d5..09c2e6f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -89,10 +89,27 @@ COPY vendor/* ./vendor/ COPY Gemfile Gemfile.lock ./ RUN bundle install && \ - rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \ # -j 1 disable parallel compilation to avoid a QEMU bug: https://github.com/rails/bootsnap/issues/495 bundle exec bootsnap precompile -j 1 --gemfile +# Bake the workspace skeleton's bundle next to the generator's. Every project +# workspace starts as a copy of lib/preview/skeleton — Gemfile.lock and a +# frozen .bundle/config included — and the agent sandbox is this very image, so +# `bundle check` inside a sandbox passes only if the skeleton's exact gem +# versions already sit in BUNDLE_PATH. When they did not (the generator's own +# lock had moved on: rails 8.1.3.1 vs the skeleton's 8.1.3), every revision on +# production paid a fresh ~80s `bundle install` into the throwaway container +# plus a fix-agent call, and verify never reached the tests before remediation +# (project 36, 2026-09-03: ~9 of 32 minutes). Two bundles share one BUNDLE_PATH +# without conflict — each resolves against its own lock and ignores the other's +# specs. BUNDLE_WITHOUT=development applies to both, which is exactly what the +# sandbox's `bundle check` expects. The `bundle check` below turns a future +# drift into an image-build failure instead of a per-revision bill. +COPY lib/preview/skeleton/Gemfile lib/preview/skeleton/Gemfile.lock ./lib/preview/skeleton/ +RUN BUNDLE_GEMFILE=/rails/lib/preview/skeleton/Gemfile bundle install && \ + BUNDLE_GEMFILE=/rails/lib/preview/skeleton/Gemfile bundle check && \ + rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git + # Copy application code COPY . . diff --git a/docs/09-ideas/05-followups.md b/docs/09-ideas/05-followups.md index 3b7c0d3..8b710b8 100644 --- a/docs/09-ideas/05-followups.md +++ b/docs/09-ideas/05-followups.md @@ -140,8 +140,7 @@ The codegen agent now runs each revision in a per-instruction throwaway containe 1. **Verify on the host.** Remaining: one full sandboxed generation green on the Linux host under the uniform-uid sandbox (`--user generator`, zero cap-adds — issue #24 replaced the earlier root+`runuser` design after the 2026-06-12 prod E2E hit the capless-root/mixed-uid SQLite deadlock). C1/C2 probes, gem resolution, and `HIFUMI_AGENT_IMAGE` pinning were confirmed on prod 2026-06-12. 2. **Generator-side Docker socket.** The generator container still runs `USER root` with `/var/run/docker.sock` bound — it has to, to launch throwaways + previews. The agent no longer runs there, but an RCE in the generator process itself is still host-root. Put a socket-proxy (e.g. tecnativa/docker-socket-proxy) in front, allowing only the `containers/images/networks` verbs PreviewManager + Sandbox use. (Already a Phase 5 candidate.) 3. **Agent egress is unrestricted.** The throwaway uses the default bridge (needs OpenRouter + rubygems). It can therefore also reach the host's published ports / other bridge containers. Move it to a dedicated network that allows only the egress it needs (DNS + 443 to OpenRouter + the gem source), or front gem installs with a local mirror so the network can be `--internal`. -4. **Bundle vendoring (perf).** The throwaway reconciles the app's gems via `AutoRemediate`'s `bundle install` on first verify, re-doing it per revision for agent-added gems. If first-revision wall time suffers (Step-7 budget is already near the edge), vendor the workspace bundle into `vendor/bundle` (already gitignored in the skeleton) so installed gems travel via the mount — but this requires untangling the `BUNDLE_PATH=/usr/local/bundle` global vs the workspace bundle (see the warning comment in `lib/roast/verify_revision.rb`), so it's deliberately deferred. (`/usr/local/bundle` is generator-owned since issue #24, so the uid-permission half of that tangle is gone; the perf half remains.) -5. **Dev/prod parity.** Sandboxing is prod-only; dev runs roast directly (no Docker, Claude-subscription transport). `FORCE_AGENT_SANDBOX=1` exercises the container path locally (needs Docker + `HIFUMI_AGENT_IMAGE` + an OpenRouter key) — wire it into CI or a manual smoke step once a Linux runner is available, since the macOS dev box can't run it. +4. **Dev/prod parity.** Sandboxing is prod-only; dev runs roast directly (no Docker, Claude-subscription transport). `FORCE_AGENT_SANDBOX=1` exercises the container path locally (needs Docker + `HIFUMI_AGENT_IMAGE` + an OpenRouter key) — wire it into CI or a manual smoke step once a Linux runner is available, since the macOS dev box can't run it. ---