Skip to content

fix(dev_env): namespace the Compose project and stop deleting the dev volume - #2396

Open
jakebromberg wants to merge 2 commits into
mainfrom
dev-env-compose-isolation
Open

fix(dev_env): namespace the Compose project and stop deleting the dev volume#2396
jakebromberg wants to merge 2 commits into
mainfrom
dev-env-compose-isolation

Conversation

@jakebromberg

Copy link
Copy Markdown
Member

Closes #2395.

dev_env/docker-compose.yml declared no top-level name:, so Compose fell back to the directory holding the file — dev_env — for every clone and every worktree of this repo. All of them silently addressed one Compose project, and db:stop ran down -v --remove-orphans, so a routine stop in one tree tore down another tree's containers and deleted its seeded pg-data volume. #2395 has the incident.

What changed

Project naming. The compose file declares name: wxyc-backend. That makes the project one explicit value instead of a function of where the repo happens to live, and COMPOSE_PROJECT_NAME overrides it — verified working both as a shell export and from the --env-file .env that every one of these scripts already passes, which is what makes the override usable per worktree rather than per shell. Containers are namespaced by project but the host ports they publish are not, so a second stack also needs its own DB_PORT / CI_DB_PORT / E2E_DB_PORT / ETL_PG_PORT / ETL_MYSQL_PORT; that is documented next to the override in .env.example, docs/env-vars.md, and docs/dev-db-fixture.md.

A declared name does not by itself hand every worktree its own database, and that is deliberate. The dev DB is a single container on a single host port seeded from a ~14 MB prod clone; forking it per worktree silently multiplies disk and produces "where did my migration go?" confusion, and a path-derived project name would also desync from a bare docker compose -f dev_env/docker-compose.yml ... typed by hand (which the wxyc-shared teardown script relies on). So the default is one shared, explicitly-named stack, and a second one is opt-in.

Container names. All four container_name: pins (wxyc-db-init, wxyc-ci-db-init, wxyc-e2e-db-init, wxyc-etl-db-init) are gone. Container names are global to the Docker daemon rather than scoped to a Compose project, so a pin makes two otherwise well-namespaced projects mutually exclusive — the second to start fails with Conflict. The container name "/wxyc-db-init" is already in use.

db:start without docker attach. The old script ended in docker attach wxyc-db-init, which is how the migration and seeding log reached the terminal. It is now:

docker compose ... --profile dev up -d --wait db && docker compose ... --profile dev run --rm db-init

up -d --wait db blocks on the Postgres healthcheck; run --rm db-init runs the initializer in the foreground, so its log streams and its exit status becomes the script's. up -d for the whole profile would detach db-init and swallow both. run builds the image on demand exactly like up does, so a fresh clone still works with no --build and an existing clone pays no rebuild.

Every other by-name reference. test:etl:env waits on the MySQL healthcheck with up --wait instead of polling docker exec dev_env-etl-mysql-1. scripts/run-library-etl.sh asks docker compose ps -q db which container backs the service. tests/e2e/etl.test.ts reads ETL_MYSQL_CONTAINER, defaulting to the declared project's generated name. tests/e2e/album-reviews-pipeline.test.ts's rate-limit hint names docker compose restart e2e-auth.

db:stop keeps the data. It is now down --remove-orphans; the destructive form moved to a new db:reset (down -v --remove-orphans). This is a deliberate developer-workflow change, not a silent one: a routine stop should not be one keystroke from destroying a seeded database, and wanting a clean fixture is a deliberate act that deserves its own verb. ci:clean, e2e:clean, and test:etl:clean keep their -v — those tear down ephemeral CI/e2e stacks that have no data worth keeping. Every caller and doc that relied on db:stop dropping pg-data was updated: README.md, CLAUDE.md, docs/dev-db-fixture.md, docs/testing.md, scripts/run-library-etl.sh, and the fixture-drift comment in tests/integration/flowsheet-upcoming-show-support.spec.js.

Test

tests/unit/scripts/docker-compose-project-isolation.test.ts (source-grep, no docker, no PG — same shape as the adjacent docker-compose-db-port.test.ts) pins the compose file declaring a project name and pinning no container_name; every package.json script reaching containers through docker compose rather than a bare docker attach/exec/restart/rm; no file still carrying a dev_env-project container name; the etl suite's default tracking the declared project; and db:stop carrying no -v while db:reset does. It fails 11 assertions on the pre-change tree.

Verification

Beyond the test, run locally against a throwaway project name:

  • Brought the dev stack up: container came up as <project>-db-1, no wxyc-db-init anywhere, the initializer ran as <project>-db-init-run-<hash> and was removed by --rm.
  • Seeding output streamed to the terminal; 159 migrations applied and 64,193 library rows loaded from the clone.
  • Exit status propagates: the real db:start returns 0, and a deliberately-failing run --rm returns its container's 42.
  • Brought a second stack up from the same compose file concurrently — different project name, different host port. Both databases served independently, which is the case the old container_name pins made impossible.
  • db:stop then db:start: data survived (Database already contains data, skipping seed). db:reset dropped the volume.

npm run typecheck, npm run lint (0 errors, no new warnings), the full unit suite (534 suites / 9,492 tests), check:docs, check:auth-tables-doc, check:better-auth-mock-sync, and lint:env all pass.

Follow-up worth a look, not in this PR

wxyc-shared's scripts/teardown-dev-environment.sh still runs docker compose -f <backend>/dev_env/docker-compose.yml --profile dev down -v --remove-orphans while iterating over Backend-Service and its worktrees. It keeps working, and it now resolves to the declared project rather than a directory-derived one, but it is still a -v on the dev volume — the same hazard this PR removed from db:stop, in another repo.

… volume

`dev_env/docker-compose.yml` declared no top-level `name:`, so Compose fell back to the directory holding the file — `dev_env` — for every clone and every worktree of this repo. All of them silently addressed one Compose project, and `db:stop` ran `down -v --remove-orphans`, so a routine stop in one tree tore down another tree's containers and deleted its seeded `pg-data` volume. That is not hypothetical: it destroyed a dev database mid-demo, surfacing downstream as `ECONNREFUSED` and an opaque "Internal server error" in the UI. The compose file now declares `name: wxyc-backend`, which makes the project one explicit value rather than a function of where the repo happens to live, and `COMPOSE_PROJECT_NAME` overrides it (verified working both as a shell export and from the `--env-file` that every one of these scripts passes) for anyone who wants a genuinely separate stack. Containers are namespaced by project but the host ports they publish are not, so a second stack also needs its own `DB_PORT` and friends — documented alongside the override.

The four `container_name:` pins are gone. Container names are global to the Docker daemon rather than scoped to a Compose project, so a pin makes two otherwise well-namespaced projects mutually exclusive: the second to start fails with `Conflict. The container name "/wxyc-db-init" is already in use`. Compose now prefixes the project name instead. Two stacks brought up side by side from this file — different project names, different host ports — were confirmed to seed and serve independently.

Dropping the pin invalidates every command that reached a container by name, so each of those moved to a Compose-resolved form. `db:start` no longer ends in `docker attach wxyc-db-init`; it brings Postgres up with `up -d --wait db` and then runs the initializer with `run --rm db-init`, which streams the migration and seeding log to the terminal and propagates the container's exit status as the script's — the seeding output is how a developer knows migrations ran, and `up -d` for the whole profile would swallow both. `test:etl:env` waits on the MySQL healthcheck via `up --wait` instead of polling `docker exec dev_env-etl-mysql-1`; `scripts/run-library-etl.sh` asks `docker compose ps -q db` which container backs the service; the etl e2e suite takes its container name from `ETL_MYSQL_CONTAINER` with the declared project's name as the default; and the album-reviews suite's rate-limit hint names `docker compose restart e2e-auth`.

`db:stop` keeps the volume — it is now `down --remove-orphans` — and the destructive form moved to a new `db:reset`. A routine stop should not be one keystroke from destroying a seeded database; wanting a clean fixture is a deliberate act and now has its own verb. Every caller and doc that relied on `db:stop` dropping `pg-data` was updated.

`tests/unit/scripts/docker-compose-project-isolation.test.ts` pins all of it: the compose file declares a project name and pins no `container_name`, no package.json script addresses a container with a bare `docker attach`/`exec`/`restart`/`rm`, no file still carries a `dev_env`-project container name, the etl suite's default tracks the declared project, and `db:stop` carries no `-v` while `db:reset` does.
…e stale down -v

`test:etl:env` and `test:etl:clean` were the only two Compose invocations in the repo that never passed `--env-file .env`. Compose resolves a bare `.env` against the compose file's own directory (`dev_env/`), not the caller's cwd, so neither script ever saw a worktree's `COMPOSE_PROJECT_NAME` — the override the Compose-project documentation tells that worktree to set. A worktree following those instructions got etl containers in the declared `wxyc-backend` project on the default host ports, and `test:etl:clean` then ran `down -v` against the shared project rather than its own: the same shape as the incident the project rename fixes, surviving inside the fix.

`.env.example` now also carries `E2E_DB_PORT`, `ETL_PG_PORT`, and `ETL_MYSQL_PORT`, which the Compose-project note tells a worktree to claim but the file never defined.

Two stale instructions the rename sweep missed. `jobs/library-etl/README.md` still told the reader to drop the volume with a raw `docker compose ... --profile dev down -v`, which carries no `--env-file` and so targets the declared project rather than the reader's; it now names `npm run db:reset`. `plans/bs1522-recurring-pollution-check.md` still asserted the dev DB is disposable via `db:stop`, which stopped being true when `db:stop` kept its volume — leaving the synthetic rotation row in place to re-fire a Sentry event on the next run.

The regression test grows two guards: every Compose invocation in `package.json` and `scripts/*.sh` passes `--env-file`, and no markdown file spells out a destructive compose command. Both fail on the parent commit.
@jakebromberg
jakebromberg force-pushed the dev-env-compose-isolation branch from f114f20 to f5f6ecc Compare September 8, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dev_env Compose stack is unnamespaced: db:stop in one checkout deletes another checkout's database

1 participant