fix(dev_env): namespace the Compose project and stop deleting the dev volume - #2396
Open
jakebromberg wants to merge 2 commits into
Open
fix(dev_env): namespace the Compose project and stop deleting the dev volume#2396jakebromberg wants to merge 2 commits into
jakebromberg wants to merge 2 commits into
Conversation
… volume `dev_env/docker-compose.yml` declared no top-level `name:`, so Compose fell back to the directory holding the file — `dev_env` — for every clone and every worktree of this repo. All of them silently addressed one Compose project, and `db:stop` ran `down -v --remove-orphans`, so a routine stop in one tree tore down another tree's containers and deleted its seeded `pg-data` volume. That is not hypothetical: it destroyed a dev database mid-demo, surfacing downstream as `ECONNREFUSED` and an opaque "Internal server error" in the UI. The compose file now declares `name: wxyc-backend`, which makes the project one explicit value rather than a function of where the repo happens to live, and `COMPOSE_PROJECT_NAME` overrides it (verified working both as a shell export and from the `--env-file` that every one of these scripts passes) for anyone who wants a genuinely separate stack. Containers are namespaced by project but the host ports they publish are not, so a second stack also needs its own `DB_PORT` and friends — documented alongside the override. The four `container_name:` pins are gone. Container names are global to the Docker daemon rather than scoped to a Compose project, so a pin makes two otherwise well-namespaced projects mutually exclusive: the second to start fails with `Conflict. The container name "/wxyc-db-init" is already in use`. Compose now prefixes the project name instead. Two stacks brought up side by side from this file — different project names, different host ports — were confirmed to seed and serve independently. Dropping the pin invalidates every command that reached a container by name, so each of those moved to a Compose-resolved form. `db:start` no longer ends in `docker attach wxyc-db-init`; it brings Postgres up with `up -d --wait db` and then runs the initializer with `run --rm db-init`, which streams the migration and seeding log to the terminal and propagates the container's exit status as the script's — the seeding output is how a developer knows migrations ran, and `up -d` for the whole profile would swallow both. `test:etl:env` waits on the MySQL healthcheck via `up --wait` instead of polling `docker exec dev_env-etl-mysql-1`; `scripts/run-library-etl.sh` asks `docker compose ps -q db` which container backs the service; the etl e2e suite takes its container name from `ETL_MYSQL_CONTAINER` with the declared project's name as the default; and the album-reviews suite's rate-limit hint names `docker compose restart e2e-auth`. `db:stop` keeps the volume — it is now `down --remove-orphans` — and the destructive form moved to a new `db:reset`. A routine stop should not be one keystroke from destroying a seeded database; wanting a clean fixture is a deliberate act and now has its own verb. Every caller and doc that relied on `db:stop` dropping `pg-data` was updated. `tests/unit/scripts/docker-compose-project-isolation.test.ts` pins all of it: the compose file declares a project name and pins no `container_name`, no package.json script addresses a container with a bare `docker attach`/`exec`/`restart`/`rm`, no file still carries a `dev_env`-project container name, the etl suite's default tracks the declared project, and `db:stop` carries no `-v` while `db:reset` does.
…e stale down -v `test:etl:env` and `test:etl:clean` were the only two Compose invocations in the repo that never passed `--env-file .env`. Compose resolves a bare `.env` against the compose file's own directory (`dev_env/`), not the caller's cwd, so neither script ever saw a worktree's `COMPOSE_PROJECT_NAME` — the override the Compose-project documentation tells that worktree to set. A worktree following those instructions got etl containers in the declared `wxyc-backend` project on the default host ports, and `test:etl:clean` then ran `down -v` against the shared project rather than its own: the same shape as the incident the project rename fixes, surviving inside the fix. `.env.example` now also carries `E2E_DB_PORT`, `ETL_PG_PORT`, and `ETL_MYSQL_PORT`, which the Compose-project note tells a worktree to claim but the file never defined. Two stale instructions the rename sweep missed. `jobs/library-etl/README.md` still told the reader to drop the volume with a raw `docker compose ... --profile dev down -v`, which carries no `--env-file` and so targets the declared project rather than the reader's; it now names `npm run db:reset`. `plans/bs1522-recurring-pollution-check.md` still asserted the dev DB is disposable via `db:stop`, which stopped being true when `db:stop` kept its volume — leaving the synthetic rotation row in place to re-fire a Sentry event on the next run. The regression test grows two guards: every Compose invocation in `package.json` and `scripts/*.sh` passes `--env-file`, and no markdown file spells out a destructive compose command. Both fail on the parent commit.
jakebromberg
force-pushed
the
dev-env-compose-isolation
branch
from
September 8, 2026 23:10
f114f20 to
f5f6ecc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2395.
dev_env/docker-compose.ymldeclared no top-levelname:, so Compose fell back to the directory holding the file —dev_env— for every clone and every worktree of this repo. All of them silently addressed one Compose project, anddb:stoprandown -v --remove-orphans, so a routine stop in one tree tore down another tree's containers and deleted its seededpg-datavolume. #2395 has the incident.What changed
Project naming. The compose file declares
name: wxyc-backend. That makes the project one explicit value instead of a function of where the repo happens to live, andCOMPOSE_PROJECT_NAMEoverrides it — verified working both as a shell export and from the--env-file .envthat every one of these scripts already passes, which is what makes the override usable per worktree rather than per shell. Containers are namespaced by project but the host ports they publish are not, so a second stack also needs its ownDB_PORT/CI_DB_PORT/E2E_DB_PORT/ETL_PG_PORT/ETL_MYSQL_PORT; that is documented next to the override in.env.example,docs/env-vars.md, anddocs/dev-db-fixture.md.A declared name does not by itself hand every worktree its own database, and that is deliberate. The dev DB is a single container on a single host port seeded from a ~14 MB prod clone; forking it per worktree silently multiplies disk and produces "where did my migration go?" confusion, and a path-derived project name would also desync from a bare
docker compose -f dev_env/docker-compose.yml ...typed by hand (which thewxyc-sharedteardown script relies on). So the default is one shared, explicitly-named stack, and a second one is opt-in.Container names. All four
container_name:pins (wxyc-db-init,wxyc-ci-db-init,wxyc-e2e-db-init,wxyc-etl-db-init) are gone. Container names are global to the Docker daemon rather than scoped to a Compose project, so a pin makes two otherwise well-namespaced projects mutually exclusive — the second to start fails withConflict. The container name "/wxyc-db-init" is already in use.db:startwithoutdocker attach. The old script ended indocker attach wxyc-db-init, which is how the migration and seeding log reached the terminal. It is now:up -d --wait dbblocks on the Postgres healthcheck;run --rm db-initruns the initializer in the foreground, so its log streams and its exit status becomes the script's.up -dfor the whole profile would detach db-init and swallow both.runbuilds the image on demand exactly likeupdoes, so a fresh clone still works with no--buildand an existing clone pays no rebuild.Every other by-name reference.
test:etl:envwaits on the MySQL healthcheck withup --waitinstead of pollingdocker exec dev_env-etl-mysql-1.scripts/run-library-etl.shasksdocker compose ps -q dbwhich container backs the service.tests/e2e/etl.test.tsreadsETL_MYSQL_CONTAINER, defaulting to the declared project's generated name.tests/e2e/album-reviews-pipeline.test.ts's rate-limit hint namesdocker compose restart e2e-auth.db:stopkeeps the data. It is nowdown --remove-orphans; the destructive form moved to a newdb:reset(down -v --remove-orphans). This is a deliberate developer-workflow change, not a silent one: a routine stop should not be one keystroke from destroying a seeded database, and wanting a clean fixture is a deliberate act that deserves its own verb.ci:clean,e2e:clean, andtest:etl:cleankeep their-v— those tear down ephemeral CI/e2e stacks that have no data worth keeping. Every caller and doc that relied ondb:stopdroppingpg-datawas updated:README.md,CLAUDE.md,docs/dev-db-fixture.md,docs/testing.md,scripts/run-library-etl.sh, and the fixture-drift comment intests/integration/flowsheet-upcoming-show-support.spec.js.Test
tests/unit/scripts/docker-compose-project-isolation.test.ts(source-grep, no docker, no PG — same shape as the adjacentdocker-compose-db-port.test.ts) pins the compose file declaring a project name and pinning nocontainer_name; every package.json script reaching containers throughdocker composerather than a baredocker attach/exec/restart/rm; no file still carrying adev_env-project container name; the etl suite's default tracking the declared project; anddb:stopcarrying no-vwhiledb:resetdoes. It fails 11 assertions on the pre-change tree.Verification
Beyond the test, run locally against a throwaway project name:
<project>-db-1, nowxyc-db-initanywhere, the initializer ran as<project>-db-init-run-<hash>and was removed by--rm.libraryrows loaded from the clone.db:startreturns 0, and a deliberately-failingrun --rmreturns its container's 42.container_namepins made impossible.db:stopthendb:start: data survived (Database already contains data, skipping seed).db:resetdropped the volume.npm run typecheck,npm run lint(0 errors, no new warnings), the full unit suite (534 suites / 9,492 tests),check:docs,check:auth-tables-doc,check:better-auth-mock-sync, andlint:envall pass.Follow-up worth a look, not in this PR
wxyc-shared'sscripts/teardown-dev-environment.shstill runsdocker compose -f <backend>/dev_env/docker-compose.yml --profile dev down -v --remove-orphanswhile iterating over Backend-Service and its worktrees. It keeps working, and it now resolves to the declared project rather than a directory-derived one, but it is still a-von the dev volume — the same hazard this PR removed fromdb:stop, in another repo.