fix(scripts): complete teardowns and truthful update aborts (NEH-214, NEH-215)

[juancobo]

Fixes the two urgent bugs from the 2026-08-03 Rionegro production incident.

What happened

update.sh's pre-stop up -d db recreated the db container under the live stack; network removal wedged on an orphaned container from a service dropped between releases (dtk-docs-1); the db was left stopped and detached from its published port while the abort path printed "nothing changed". Compose exits 0 on that partial teardown (reproduced locally), so set -e never fires — the failure is invisible to the script and to the operator.

Changes

• --remove-orphans on every compose down (update.sh, stop.sh, rollback-update.sh, setup.sh, the restore path) and on start.sh's boot-time up, so containers from services dropped between releases can never hold the project network hostage.
• update.sh step 5 no longer mutates the running stack: it probes container state first and only runs up -d --no-recreate db when db is not already running. The full recreate happens post-stop, from a clean slate.
• update.sh step 6 tears down explicitly after systemctl stop dtk — systemd only runs ExecStop for an active unit — and a new assert_stack_gone verifies no project containers or the network survive, aborting loudly if they do (this is what catches the exit-0 partial teardown). Same assertion after the post-migration down.
• New STACK_TOUCHED tracking: "nothing changed" now prints only when container state is genuinely untouched. Otherwise the abort reports the service state, per-container states, whether 127.0.0.1:5432 is listening on the host, and the recovery command.
• generate-secrets.sh's rotation path gets the same gating (no recreate from the boot-time oneshot; never stops a db it did not start).
• check-stack-health.sh now also requires the db's published port to be listening on the host (ss, no connection made; skipped if ss is absent) — the incident's exact signature was a container reporting healthy while nothing listened on the host.

Validation

• bash -n clean on all seven scripts.
• Local Docker simulation: reproduced the wedge with a scratch project (orphaned service survives a bare down, network removal fails "Resource is still in use", exit code 0); down --remove-orphans clears container and network; the project-name derivation used by assert_stack_gone works on compose v5.3.0.
• Profile-gated services (dev backend) are NOT treated as orphans — verified a running profile-gated container survives down --remove-orphans, so this cannot destroy a developer's backend container.
• Pending hardware validation on the bench unit at Friday's bench test: plant an orphan, run the update end-to-end (bench checklist updated separately).

Linear: NEH-214, NEH-215 (GitHub mirrors #283, #284).

[linear-code]

NEH-214

NEH-215

[Copilot]

Pull request overview

Improves update and teardown reliability following the Rionegro production incident.

Changes:

• Removes orphaned containers during lifecycle operations.
• Verifies update teardowns and reports actual abort state.
• Adds safer database startup gating and host-port health checks.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/update.sh	Adds teardown assertions and state-aware recovery reporting.
scripts/stop.sh	Removes orphaned containers during shutdown.
scripts/start.sh	Removes orphans during startup reconciliation.
scripts/setup.sh	Cleans up orphans during setup teardown paths.
scripts/rollback-update.sh	Cleans up orphans during rollback operations.
scripts/generate-secrets.sh	Avoids unnecessary database recreation and shutdown.
scripts/check-stack-health.sh	Checks for a host-side database listener.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (3)

scripts/update.sh:193

• If docker ps cannot query the daemon, || true converts that failure into an empty container list. If the subsequent network inspection also returns nonzero, this function reports success even though it verified nothing, defeating the fail-loud teardown assertion. Treat an inspection failure as an incomplete teardown.

    leftovers="$(docker ps -aq --filter "label=com.docker.compose.project=$project" 2>/dev/null || true)"

scripts/generate-secrets.sh:166

• When this branch starts the database, a later ALTER USER failure exits immediately because of set -e, before the conditional stop at lines 183–185. The script therefore leaves a previously stopped database running; handle the ALTER failure explicitly and stop only the container started here before exiting.

        $COMPOSE up -d --no-recreate db >/dev/null

scripts/check-stack-health.sh:102

• When ss is unavailable, db_port_listening intentionally returns success without checking the port, but this message still claims the host listener was verified. Report that the port check was skipped so startup diagnostics remain truthful.

        echo "✓ Stack healthy: db healthy and listening on the host, frontend running${NGINX_NOTE}."