From f910368ce4b9dea1d038a9ec5875c46fb6d728ee Mon Sep 17 00:00:00 2001 From: Fernando Frizzatti Date: Fri, 14 Aug 2026 17:35:59 -0300 Subject: [PATCH 01/19] feat(preview): per-PR deployment previews (chart + build pipeline) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the in-repo half of per-PR preview environments: label a PR `preview` and get a throwaway Studio at pr-.preview.studio.decocms.com with its own namespace, database and bucket, destroyed when the PR closes. Today the only way to see a change running is to merge it — staging is a shared, serialized, post-merge resource, and a non-author has no path short of cloning the monorepo. Chart (0.13.2 -> 0.14.0), everything gated on preview.enabled: - HTTPRoute attaching to a shared wildcard Gateway (one DNS record, one cert; per-host ACME would be an order per PR). - Three Argo sync-hook Jobs: PreSync -10 provision (CREATE DATABASE pr_ + mc mb), PreSync 0 migrate, PostDelete teardown (DROP DATABASE ... WITH (FORCE) + mc rb). Hooks rather than an initContainer because only hooks have a PostDelete phase — otherwise databases leak whenever a teardown job does not run. - validatePreview: five render-time guards, each covering a case that would otherwise produce a healthy-looking object that silently does nothing. - values-preview.yaml, rendered and asserted in CI so it cannot rot even though nothing in this repo deploys it. Migrations run in a Job, never in a pod. The default topology is three processes that would race migrateToLatest on a fresh database. Pods run with --skip-migrations and validatePreview fails the render without it. That flag alone is not sufficient: it only skips studio's own migrations, and DBOS still migrates its `dbos` schema on launch, so parallel boots crash on dbos.dbos_migrations unique-constraint violations (see tests/multi-pod/docker-compose.yml). Hence the new migrate-dbos entry point, bundled alongside migrate.js and run by the same Job. Pod topology is deliberately identical to production — nginx front door plus two API containers. Skipping the nginx tier would have been cheaper but needed new chart keys, and a preview running a different topology than prod can miss exactly the class of bug it exists to catch. Verification: - Default render is unchanged: 0 non-`helm.sh/chart` diff lines vs the pre-change baseline. - CI assertions mutation-tested — removing the HTTPRoute, downgrading the PostDelete hook, reverting DATABASE_POOL_MAX, and misindenting the SQL heredoc each go red. - Hook scripts executed against a stub psql; GC script executed against fixtures; bundled migrate-dbos.js runs and fails only on connection refusal. Deploy side (ApplicationSet, Gateway, cert, oauth2-proxy, shared Postgres and MinIO, TTL/orphan sweepers) lands separately in decocms/deco-apps-cd. Prerequisites are documented in deploy/preview/README.md. Nothing here holds a cluster credential. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/helm-test.yml | 106 +++++++++ .github/workflows/preview-build.yaml | 205 ++++++++++++++++++ .github/workflows/preview-gc.yaml | 103 +++++++++ .github/workflows/release-studio.yaml | 1 + CONTRIBUTING.md | 13 ++ apps/api/package.json | 2 + apps/api/scripts/bundle-server-script.ts | 12 +- apps/api/src/database/migrate-dbos.ts | 64 ++++++ deploy/helm/studio/Chart.yaml | 7 +- deploy/helm/studio/templates/_helpers.tpl | 49 +++++ deploy/helm/studio/templates/httproute.yaml | 33 +++ .../templates/preview-db-provision-job.yaml | 90 ++++++++ .../templates/preview-db-teardown-job.yaml | 83 +++++++ .../studio/templates/preview-migrate-job.yaml | 65 ++++++ deploy/helm/studio/templates/validations.yaml | 1 + deploy/helm/studio/values-preview.yaml | 190 ++++++++++++++++ deploy/helm/studio/values.yaml | 68 ++++++ deploy/preview/README.md | 166 ++++++++++++++ 18 files changed, 1256 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/preview-build.yaml create mode 100644 .github/workflows/preview-gc.yaml create mode 100644 apps/api/src/database/migrate-dbos.ts create mode 100644 deploy/helm/studio/templates/httproute.yaml create mode 100644 deploy/helm/studio/templates/preview-db-provision-job.yaml create mode 100644 deploy/helm/studio/templates/preview-db-teardown-job.yaml create mode 100644 deploy/helm/studio/templates/preview-migrate-job.yaml create mode 100644 deploy/helm/studio/values-preview.yaml create mode 100644 deploy/preview/README.md diff --git a/.github/workflows/helm-test.yml b/.github/workflows/helm-test.yml index 5a6034835d..97a7d400bc 100644 --- a/.github/workflows/helm-test.yml +++ b/.github/workflows/helm-test.yml @@ -156,3 +156,109 @@ jobs: --set database.url=postgresql://ci:ci@postgres.example.com:5432/studio \ -f deploy/helm/studio/examples/values-clickhouse.yaml \ > /dev/null + + # values-preview.yaml is consumed by the Argo ApplicationSet in + # decocms/deco-apps-cd, so nothing in THIS repo would otherwise render it + # and it would rot silently. Assert the properties previews depend on, + # not just that it renders. + - name: Render (per-PR preview) + run: | + set -euo pipefail + helm template studio-pr-42 deploy/helm/studio \ + -f deploy/helm/studio/values-preview.yaml \ + --set preview.prNumber=42 \ + --set preview.host=pr-42.preview.studio.decocms.com \ + --set image.tag=pr-42-abc1234 \ + --set nginx.image.tag=pr-42-abc1234 \ + --set database.url=postgresql://ci:ci@pg.preview.example.com:5432/pr_42 \ + --api-versions gateway.networking.k8s.io/v1 \ + > /tmp/preview.yaml + + fail() { echo "::error::$1"; exit 1; } + + grep -q 'kind: HTTPRoute' /tmp/preview.yaml \ + || fail "preview render has no HTTPRoute — the preview would be unreachable" + grep -q 'pr-42.preview.studio.decocms.com' /tmp/preview.yaml \ + || fail "preview HTTPRoute is missing its hostname" + + for job in provision migrate teardown; do + grep -q "studio-pr-42-preview-${job}" /tmp/preview.yaml \ + || fail "preview render is missing the ${job} Job" + done + grep -q 'argocd.argoproj.io/hook: PostDelete' /tmp/preview.yaml \ + || fail "teardown Job is not a PostDelete hook — databases would leak on PR close" + + # Every studio container (api-0, api-1, worker) must skip migrations: + # the PreSync Job is the single writer. Three, not two. + count=$(grep -c -- '--skip-migrations' /tmp/preview.yaml) + [ "$count" -eq 3 ] \ + || fail "expected 3 containers with --skip-migrations, found ${count}" + + grep -q 'DATABASE_POOL_MAX: "5"' /tmp/preview.yaml \ + || fail "DATABASE_POOL_MAX is not 5 — N previews would exhaust the shared Postgres" + + # PVC finalizers are the most common cause of a namespace that never + # finishes deleting. + if grep -qE '^kind: PersistentVolumeClaim' /tmp/preview.yaml; then + fail "preview render contains a PersistentVolumeClaim" + fi + + # Shell scripts embedded in the hook Jobs — a bad heredoc indent here + # only surfaces at sync time otherwise. + python3 -c 'import yaml' 2>/dev/null || pip install --quiet pyyaml + python3 - <<'PY' + import subprocess, sys, yaml + bad = 0 + for d in yaml.safe_load_all(open("/tmp/preview.yaml")): + if not d or d.get("kind") != "Job": + continue + spec = d["spec"]["template"]["spec"] + for c in (spec.get("initContainers") or []) + spec["containers"]: + if not c.get("args"): + continue + p = subprocess.run(["sh", "-n"], input=c["args"][0], + capture_output=True, text=True) + if p.returncode != 0: + print(f"::error::{d['metadata']['name']}/{c['name']}: {p.stderr.strip()}") + bad += 1 + sys.exit(1 if bad else 0) + PY + + - name: Render (preview validations must fail) + run: | + set -uo pipefail + expect_fail() { + desc="$1"; want="$2"; shift 2 + out=$(helm template studio-pr-42 deploy/helm/studio \ + -f deploy/helm/studio/values-preview.yaml \ + --set preview.prNumber=42 \ + --set database.url=postgresql://ci:ci@pg.example.com:5432/d \ + --api-versions gateway.networking.k8s.io/v1 "$@" 2>&1) + if [ $? -eq 0 ]; then + echo "::error::${desc}: rendered successfully — the guard is missing" + exit 1 + fi + if ! echo "$out" | grep -q "$want"; then + echo "::error::${desc}: failed for the wrong reason. Wanted '${want}', got:" + echo "$out" + exit 1 + fi + echo "ok: ${desc}" + } + + expect_fail "missing preview.host" "preview.host is required" + expect_fail "ingress conflict" "mutually exclusive" \ + --set preview.host=h.example.com \ + --set ingress.enabled=true \ + --set 'ingress.hosts[0].host=h.example.com' \ + --set 'ingress.hosts[0].paths[0].path=/' \ + --set 'ingress.hosts[0].paths[0].pathType=Prefix' + expect_fail "missing gateway name" "preview.gateway.name is required" \ + --set preview.host=h.example.com --set preview.gateway.name= + expect_fail "missing db admin secret" "preview.dbAdminSecret.name is required" \ + --set preview.host=h.example.com --set preview.dbAdminSecret.name= + + # image.command is a list, which --set cannot express cleanly. + printf 'image:\n command: ["bun","run","deco"]\n' > /tmp/no-skip.yaml + expect_fail "missing --skip-migrations" "requires --skip-migrations" \ + --set preview.host=h.example.com -f /tmp/no-skip.yaml diff --git a/.github/workflows/preview-build.yaml b/.github/workflows/preview-build.yaml new file mode 100644 index 0000000000..5f41ac8fa0 --- /dev/null +++ b/.github/workflows/preview-build.yaml @@ -0,0 +1,205 @@ +name: Preview build + +# Builds the two images for a per-PR preview environment and posts (or edits) +# the sticky PR comment carrying the URL. +# +# Deliberately a separate workflow rather than a workflow_call refactor of +# release-studio.yaml: that file is release-consistency logic (npm version +# probing, GHCR existence checks, provenance, the deco-apps-cd dispatch gate, +# the docs webhook), and parameterising it would thread ~8 booleans through +# every `if:` in the PRODUCTION release path. The genuinely shared surface is +# one command (`bun run build:studio`) and two Dockerfiles, already reusable +# as-is — so previews cannot destabilise shipping. +# +# What deploys the images is Argo CD, in decocms/deco-apps-cd: an ApplicationSet +# with a GitHub PR generator, also gated on the `preview` label. Nothing here +# holds a cluster credential. + +on: + pull_request: + types: [opened, synchronize, reopened, labeled] + +concurrency: + group: preview-build-${{ github.event.pull_request.number }} + cancel-in-progress: true + +env: + REGISTRY: ghcr.io + # Separate GHCR packages from the release ones, so the preview GC can be + # aggressive with zero chance of deleting a release tag. + IMAGE_NAME: ${{ github.repository }}/studio-preview + NGINX_IMAGE_NAME: ${{ github.repository }}/studio-nginx-preview + PREVIEW_DOMAIN: preview.studio.decocms.com + +jobs: + build: + name: Build preview images + # Opt-in. `labeled` is in the trigger list so adding the label to an + # already-open PR starts a build; the ApplicationSet enforces the same + # label independently, so neither half alone can create an orphan. + if: contains(github.event.pull_request.labels.*.name, 'preview') + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + pull-requests: write + steps: + - uses: actions/checkout@v4 + + # `pull_request` checks out refs/pull/N/merge, so the image always + # contains main ∪ PR migrations. That invariant is load-bearing: Kysely's + # strict missing-migration check hard-fails a boot against a database + # that knows a migration the image does not ship. Asserted, not assumed. + - name: Assert merge-ref contains main + run: | + git fetch --no-tags --depth=200 origin main + if ! git merge-base --is-ancestor origin/main HEAD; then + echo "::error::HEAD does not contain origin/main. A preview built from" + echo "::error::a stale base can ship fewer migrations than its database" + echo "::error::already has, which hard-fails the boot. Rebase the PR." + exit 1 + fi + + - id: meta + run: | + echo "tag=pr-${{ github.event.pull_request.number }}-$(git rev-parse --short=7 HEAD)" >> "$GITHUB_OUTPUT" + echo "host=pr-${{ github.event.pull_request.number }}.${PREVIEW_DOMAIN}" >> "$GITHUB_OUTPUT" + + # The hidden marker is what makes this comment sticky: find it first, then + # edit in place. Without the lookup every push would append a new comment. + - name: Find existing preview comment + id: find-comment + uses: peter-evans/find-comment@v3 + with: + issue-number: ${{ github.event.pull_request.number }} + comment-author: github-actions[bot] + body-includes: + + # Posted before the build so a reviewer sees "building" at t+15s rather + # than nothing for seven minutes. Edited in place when the images land. + - name: Comment (building) + uses: peter-evans/create-or-update-comment@v4 + continue-on-error: true + with: + issue-number: ${{ github.event.pull_request.number }} + comment-id: ${{ steps.find-comment.outputs.comment-id }} + edit-mode: replace + body: | + + 🔨 **Building preview** for `${{ steps.meta.outputs.tag }}`… + + - uses: actions/setup-node@v4 + with: + node-version: "24" + - name: Setup Bun and install dependencies + uses: ./.github/actions/setup-bun + + # No sourcemaps (nothing un-minifies preview stacks), no npm publish, no + # provenance, no tarball smoke test. Same `build:studio` and same + # `bun add /tmp/decocms.tgz` install as a release, so preview packaging + # cannot drift from production packaging. + - name: Build combined Studio distribution + run: bun run build:studio + env: + BUILD_SOURCEMAPS: "0" + + - name: Strip sourcemaps and pack + run: | + find apps/web/dist apps/api/dist -name '*.map' -delete 2>/dev/null || true + cd apps/api && npm pack + + - name: Stage the tarball for both build contexts + run: | + mkdir -p docker-context + mv apps/api/decocms-*.tgz docker-context/decocms.tgz + # apps/web/Dockerfile builds from the repo root because it also COPYs + # deploy/helm/studio/files/api-nginx.conf. + cp docker-context/decocms.tgz decocms.tgz + + - uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - uses: docker/setup-buildx-action@v3 + + # amd64 only — the preview cluster is amd64, and the release workflow's + # own comments record that emulated arm64 costs ~64s vs ~13s native for + # the tarball-install layer. `cache-from` also reads the release scopes so + # the apt / useradd / duckdb layers are hits on a cold preview. + - name: Build and push API image + uses: docker/build-push-action@v5 + with: + context: ./docker-context + file: ./apps/api/Dockerfile + platforms: linux/amd64 + push: true + provenance: false + tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.tag }} + cache-from: | + type=gha,scope=studio-preview-amd64 + type=gha,scope=studio-api-amd64 + cache-to: type=gha,mode=max,scope=studio-preview-amd64 + + - name: Build and push nginx image + uses: docker/build-push-action@v5 + with: + context: . + file: ./apps/web/Dockerfile + platforms: linux/amd64 + push: true + provenance: false + tags: ${{ env.REGISTRY }}/${{ env.NGINX_IMAGE_NAME }}:${{ steps.meta.outputs.tag }} + cache-from: | + type=gha,scope=studio-preview-web-amd64 + type=gha,scope=studio-web-amd64 + cache-to: type=gha,mode=max,scope=studio-preview-web-amd64 + + # Re-find: on the first run the "building" step created the comment, so + # the id from before the build is empty. + - name: Re-find preview comment + id: find-comment-ready + uses: peter-evans/find-comment@v3 + with: + issue-number: ${{ github.event.pull_request.number }} + comment-author: github-actions[bot] + body-includes: + + - name: Comment (ready) + uses: peter-evans/create-or-update-comment@v4 + continue-on-error: true + with: + issue-number: ${{ github.event.pull_request.number }} + comment-id: ${{ steps.find-comment-ready.outputs.comment-id }} + edit-mode: replace + body: | + + ### 🔍 Preview: https://${{ steps.meta.outputs.host }} + + Built from `${{ steps.meta.outputs.tag }}`. Argo CD picks the image up + within ~60s; the first sync also creates and migrates the database, so + allow another minute or two on a brand-new preview. + + **Sign in:** the database is empty and yours — sign up with any email + and password. It is thrown away when this PR closes. + +
What does not work in a preview + + - **Agent tool execution against a hosted sandbox** — previews run + `STUDIO_SANDBOX_PROVIDER=user-desktop` with no daemon attached, so + dispatch returns `409 link_offline`. Sandbox previews and the + sandbox lifecycle UI are equally out. + - **AI features** until you add your own provider key in org settings. + Previews ship no key, so preview LLM spend is zero by construction. + - **Google / GitHub sign-in** — OAuth callbacks cannot be registered + for a per-PR hostname. The buttons are hidden. + - **Monitoring dashboard** (no ClickHouse), **billing** (no Stripe), + **outbound email** (no mail provider). + - **Multi-pod behaviour** — a preview is one pod. Do not conclude + "it worked in preview" about a distributed-systems change; that is + what `tests/multi-pod/` is for. + +
+ + Remove the `preview` label to tear this down now. Previews expire 72h + after the last push. diff --git a/.github/workflows/preview-gc.yaml b/.github/workflows/preview-gc.yaml new file mode 100644 index 0000000000..99650b3174 --- /dev/null +++ b/.github/workflows/preview-gc.yaml @@ -0,0 +1,103 @@ +name: Preview GC + +# GHCR has no TTL, so preview image tags accumulate forever without this. +# Deletes every `pr--` tag whose pull request is closed, then applies a +# retention floor to whatever survives. +# +# Scoped to the *-preview packages only. Release tags live in different +# packages (studio/studio, studio/studio-nginx) and are never reachable from +# here, which is the whole reason previews got their own packages. + +on: + schedule: + - cron: "17 4 * * *" + workflow_dispatch: + +permissions: + contents: read + packages: write + pull-requests: read + +jobs: + gc: + name: Prune preview image tags + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + package: [studio/studio-preview, studio/studio-nginx-preview] + steps: + - name: Delete tags for closed PRs + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + ORG: ${{ github.repository_owner }} + PACKAGE: ${{ matrix.package }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + # %2F — the package name contains a slash and must be path-escaped. + ENCODED=$(printf '%s' "$PACKAGE" | sed 's#/#%2F#g') + + if ! gh api "orgs/${ORG}/packages/container/${ENCODED}/versions?per_page=100" \ + --paginate > /tmp/versions.json 2>/tmp/api-err; then + # A package that has never been pushed 404s. That is the expected + # state before the first preview, not a failure. + if grep -q "Not Found" /tmp/api-err; then + echo "Package ${PACKAGE} does not exist yet — nothing to prune." + exit 0 + fi + cat /tmp/api-err >&2 + exit 1 + fi + + jq -r '.[] | . as $v | ($v.metadata.container.tags // [])[] | "\($v.id) \(.)"' \ + /tmp/versions.json > /tmp/tags.txt + + # Resolve each PR's state ONCE, not once per tag: a PR routinely has + # a dozen tags and the API is rate-limited. + awk '{ n = $2; sub(/^pr-/, "", n); sub(/-.*$/, "", n); + if (n ~ /^[0-9]+$/) print n }' /tmp/tags.txt | sort -u > /tmp/prs.txt + + : > /tmp/closed.txt + while read -r pr_number; do + [ -n "$pr_number" ] || continue + state=$(gh pr view "$pr_number" --repo "$REPO" --json state --jq .state 2>/dev/null || echo UNKNOWN) + echo "PR #${pr_number}: ${state}" + # UNKNOWN (deleted or inaccessible PR) is deliberately kept — this + # job must never be the thing that removes an image someone is + # still looking at. + if [ "$state" = "CLOSED" ] || [ "$state" = "MERGED" ]; then + echo "$pr_number" >> /tmp/closed.txt + fi + done < /tmp/prs.txt + + deleted=0 + while read -r version_id tag; do + [ -n "$tag" ] || continue + case "$tag" in pr-*) ;; *) continue ;; esac + pr_number="${tag#pr-}" + pr_number="${pr_number%%-*}" + case "$pr_number" in ''|*[!0-9]*) echo "skip malformed tag: $tag"; continue ;; esac + + if grep -qx "$pr_number" /tmp/closed.txt; then + echo "deleting ${PACKAGE}:${tag} (PR #${pr_number})" + gh api --method DELETE \ + "orgs/${ORG}/packages/container/${ENCODED}/versions/${version_id}" || true + deleted=$((deleted + 1)) + fi + done < /tmp/tags.txt + + echo "deleted ${deleted} tag(s) from ${PACKAGE}" + + # Backstop for untagged layers orphaned by tag deletion and for tags whose + # PR lookup kept failing. The floor is generous on purpose: this job is + # about bounding growth, not reclaiming every byte. + - name: Retention floor + uses: actions/delete-package-versions@v5 + continue-on-error: true + with: + owner: ${{ github.repository_owner }} + package-name: ${{ matrix.package }} + package-type: container + min-versions-to-keep: 50 + delete-only-untagged-versions: true diff --git a/.github/workflows/release-studio.yaml b/.github/workflows/release-studio.yaml index 20f62a3372..7e06135a88 100644 --- a/.github/workflows/release-studio.yaml +++ b/.github/workflows/release-studio.yaml @@ -167,6 +167,7 @@ jobs: test -f apps/api/dist/client/index.html test -f apps/api/dist/server/server.js test -f apps/api/dist/server/migrate.js + test -f apps/api/dist/server/migrate-dbos.js test -f apps/api/dist/server/cli.js - name: Pack the npm tarball working-directory: apps/api diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c9474e6c30..51e0ca91ac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,6 +25,19 @@ bun run dev # migrations + web app + API what changed, how to test it, and any migration notes. - Keep PRs focused. Stack dependent PRs rather than bundling unrelated changes. +### Preview environments + +Add the **`preview`** label to a PR and a throwaway Studio is deployed at +`https://pr-.preview.studio.decocms.com`. A bot comment carries the link and +updates itself as the build progresses; expect ~10 minutes on a fresh preview. +The database is empty, so sign up with any email and password. Removing the +label (or closing the PR) destroys the namespace, the database and the bucket. + +Previews deliberately do **not** cover hosted agent sandboxes, OAuth sign-in, +billing, monitoring, or multi-pod behaviour — a green preview says nothing +about any of those. See [`deploy/preview/README.md`](./deploy/preview/README.md) +for the full list and for troubleshooting. + ## The rules that bite hardest ### 1. Tools go through `StudioContext` diff --git a/apps/api/package.json b/apps/api/package.json index 318e9f057b..dffec9f3c1 100644 --- a/apps/api/package.json +++ b/apps/api/package.json @@ -25,9 +25,11 @@ "build:server": "bun run scripts/bundle-server-script.ts --dist ./dist/server", "generate:tool-contracts": "bun run scripts/generate-tool-contracts.ts", "db:migrate": "bun run ./dist/server/migrate.js", + "db:migrate:dbos": "bun run ./dist/server/migrate-dbos.js", "check": "tsc --noEmit", "start": "bun run ./dist/server/server.js", "migrate": "bun run src/database/migrate.ts", + "migrate:dbos": "bun run src/database/migrate-dbos.ts", "test": "bun test src", "better-auth:migrate": "bunx --bun @better-auth/cli migrate -y --config src/auth/index.ts", "prepublishOnly": "bun run --cwd=../.. build:studio" diff --git a/apps/api/scripts/bundle-server-script.ts b/apps/api/scripts/bundle-server-script.ts index 55a25045d7..bd1e40660f 100644 --- a/apps/api/scripts/bundle-server-script.ts +++ b/apps/api/scripts/bundle-server-script.ts @@ -526,6 +526,14 @@ function buildMigrateScript(packagesToExternalize: Set) { ); } +function buildMigrateDbosScript(packagesToExternalize: Set) { + return buildScript( + join(SCRIPT_DIR, "../src/database/migrate-dbos.ts"), + "migrate-dbos.js", + packagesToExternalize, + ); +} + function buildServerScript(packagesToExternalize: Set) { return buildScript( join(SCRIPT_DIR, "../src/index.ts"), @@ -710,7 +718,7 @@ async function verifyBundlesShipExternals() { const failures: { pkg: string; from: Set }[] = []; const byPkg = new Map>(); - const entries = ["cli.js", "server.js", "migrate.js"]; + const entries = ["cli.js", "server.js", "migrate.js", "migrate-dbos.js"]; for (const entry of entries) { const entryPath = join(OUTPUT_DIR, entry); if (!existsSync(entryPath)) continue; @@ -827,6 +835,7 @@ async function main() { // Build migrate.js, server.js, and cli.js await buildMigrateScript(packagesToExternalize); + await buildMigrateDbosScript(packagesToExternalize); await buildServerScript(packagesToExternalize); await buildCliScript(packagesToExternalize); @@ -846,6 +855,7 @@ async function main() { console.log("\n🎉 Build completed successfully!"); console.log(`📦 Output directory: ${OUTPUT_DIR}`); console.log(` - migrate.js`); + console.log(` - migrate-dbos.js`); console.log(` - server.js`); console.log(` - cli.js`); console.log(` - emscripten-module.wasm`); diff --git a/apps/api/src/database/migrate-dbos.ts b/apps/api/src/database/migrate-dbos.ts new file mode 100644 index 0000000000..d9b5269f87 --- /dev/null +++ b/apps/api/src/database/migrate-dbos.ts @@ -0,0 +1,64 @@ +/** + * DBOS System-Schema Migration Runner + * + * Creates/updates the `dbos` schema (which DBOS owns inside studio's Postgres + * database) as a standalone step, so it happens exactly once before any app + * pod boots. + * + * Why this exists as its own entry point: studio's `--skip-migrations` flag + * only skips studio's OWN migrations (see settings/pipeline.ts). DBOS still + * runs its system-schema migrations on `DBOS.launch()`, so N pods booting + * against a fresh database race on inserts into `dbos.dbos_migrations` and the + * losers crash with unique-constraint violations — the failure documented in + * tests/multi-pod/docker-compose.yml. Running this once, ahead of the pods, + * removes the race. + * + * Deliberately separate from migrate.ts rather than folded into it: the two + * fail independently, and `bun run migrate` keeps its exact current behavior + * for dev, e2e and the multi-pod harness. + * + * Usage: + * bun run migrate:dbos (from source) + * bun run dist/server/migrate-dbos.js (from the published bundle/image) + */ + +import { buildDbosConfig } from "../dbos/config"; +import { getSettings } from "../settings"; +import { withSslmode } from "./index"; + +export async function migrateDbos(): Promise { + const settings = getSettings(); + + // Dynamic import mirrors index.ts: setConfig must precede workflow registration. + const { DBOS } = await import("@dbos-inc/dbos-sdk"); + + DBOS.setConfig( + buildDbosConfig({ + systemDatabaseUrl: withSslmode( + settings.databaseUrl, + settings.databasePgSsl, + ), + poolSize: settings.dbosPoolSize, + executorID: settings.podName, + // Dequeue nothing — this process only creates the schema. + listenQueues: [], + }), + ); + + // launch() is what applies the system-schema migrations. + await DBOS.launch(); + await DBOS.shutdown(); +} + +if (import.meta.main) { + (async () => { + try { + await migrateDbos(); + console.log("DBOS system-schema migrations completed."); + process.exit(0); + } catch (error) { + console.error("DBOS system-schema migration failed:", error); + process.exit(1); + } + })(); +} diff --git a/deploy/helm/studio/Chart.yaml b/deploy/helm/studio/Chart.yaml index 3e1ff67e4f..e1cd4a4268 100644 --- a/deploy/helm/studio/Chart.yaml +++ b/deploy/helm/studio/Chart.yaml @@ -2,6 +2,11 @@ apiVersion: v2 name: chart-deco-studio description: Helm chart for deco Studio — supports inline secrets, external secretName, and AWS Secrets Manager via ExternalSecret type: application +# 0.14.0: per-PR preview releases (preview.enabled, default false). Adds an +# HTTPRoute plus three Argo sync-hook Jobs (PreSync provision, PreSync migrate, +# PostDelete teardown) that create and destroy the per-PR database and bucket. +# Renders nothing when disabled, so the default render is byte-identical and +# existing releases are unaffected. Pod topology is deliberately unchanged. # 0.13.2: dedicated emptyDir + FAST_PREVIEW_CACHE_DIR for the sandbox-less # Fast Preview disk cache (fastPreviewCache values block). # 0.13.1: documents the pre-existing `serviceAccount` values block. The @@ -12,7 +17,7 @@ type: application # when disabled, so existing releases are unaffected. # 0.12.4: chart-managed API/worker dispatch roles now use # STUDIO_DISPATCH_ROLE; legacy overrides are rejected. -version: 0.13.2 +version: 0.14.0 appVersion: "latest" dependencies: diff --git a/deploy/helm/studio/templates/_helpers.tpl b/deploy/helm/studio/templates/_helpers.tpl index 5467ec34c4..09f6904fb8 100644 --- a/deploy/helm/studio/templates/_helpers.tpl +++ b/deploy/helm/studio/templates/_helpers.tpl @@ -292,6 +292,55 @@ at render time instead. {{- end }} {{- end }} +{{/* +Per-PR database name. Postgres identifier, so the numeric PR number is prefixed +rather than used bare. +*/}} +{{- define "chart-deco-studio.previewDbName" -}} +{{- printf "pr_%s" (toString .Values.preview.prNumber) -}} +{{- end }} + +{{/* +Per-PR bucket name. Explicit override wins; otherwise derived from the PR number. +*/}} +{{- define "chart-deco-studio.previewBucket" -}} +{{- if .Values.preview.objectStorage.bucket -}} +{{- .Values.preview.objectStorage.bucket | trim -}} +{{- else -}} +{{- printf "preview-pr-%s" (toString .Values.preview.prNumber) -}} +{{- end -}} +{{- end }} + +{{/* +Validates per-PR preview releases. Every failure here is something that would +otherwise render a healthy-looking object that silently does nothing: an +HTTPRoute with no hostname matches no traffic, a pod without --skip-migrations +races the PreSync migration Job, and a provisioner Job without admin +credentials cannot create the database the pods are about to connect to. +*/}} +{{- define "chart-deco-studio.validatePreview" -}} +{{- if .Values.preview.enabled }} +{{- if not .Values.preview.host }} +{{- fail "chart-deco-studio: preview.host is required when preview.enabled=true" -}} +{{- end }} +{{- if not .Values.preview.prNumber }} +{{- fail "chart-deco-studio: preview.prNumber is required when preview.enabled=true — it names the per-PR database and bucket" -}} +{{- end }} +{{- if .Values.ingress.enabled }} +{{- fail "chart-deco-studio: preview.enabled and ingress.enabled are mutually exclusive — previews route through Gateway API (HTTPRoute), not Ingress" -}} +{{- end }} +{{- if not .Values.preview.gateway.name }} +{{- fail "chart-deco-studio: preview.gateway.name is required when preview.enabled=true" -}} +{{- end }} +{{- if not .Values.preview.dbAdminSecret.name }} +{{- fail "chart-deco-studio: preview.dbAdminSecret.name is required when preview.enabled=true — the provisioner Job needs admin credentials to CREATE/DROP the per-PR database" -}} +{{- end }} +{{- if not (has "--skip-migrations" (.Values.image.command | default list)) }} +{{- fail "chart-deco-studio: preview.enabled=true requires --skip-migrations in image.command; the PreSync Job is the single migration writer and neither the API nor the worker containers (which share image.command) may race it" -}} +{{- end }} +{{- end }} +{{- end }} + {{/* Validates public NATS tunnel cluster-creds mount configuration. */}} diff --git a/deploy/helm/studio/templates/httproute.yaml b/deploy/helm/studio/templates/httproute.yaml new file mode 100644 index 0000000000..866f62b285 --- /dev/null +++ b/deploy/helm/studio/templates/httproute.yaml @@ -0,0 +1,33 @@ +{{- if .Values.preview.enabled }} +{{- /* +Gateway API route for a per-PR preview, attaching to a SHARED wildcard Gateway +(one DNS record, one cert) rather than provisioning per-host TLS. + +The Gateway's allowedRoutes selector matches on NAMESPACE labels, so whatever +creates this namespace must label it — with Argo CD that means +syncPolicy.managedNamespaceMetadata. Without the label this route attaches to +nothing and reports no error. + +The `ingress` values block is the wrong tool here and says so; validatePreview +rejects enabling both. +*/ -}} +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: {{ include "chart-deco-studio.fullname" . }} + labels: + {{- include "chart-deco-studio.labels" . | nindent 4 }} +spec: + parentRefs: + - name: {{ .Values.preview.gateway.name }} + namespace: {{ .Values.preview.gateway.namespace }} + {{- with .Values.preview.gateway.sectionName }} + sectionName: {{ . }} + {{- end }} + hostnames: + - {{ .Values.preview.host | quote }} + rules: + - backendRefs: + - name: {{ include "chart-deco-studio.fullname" . }} + port: {{ .Values.service.port }} +{{- end }} diff --git a/deploy/helm/studio/templates/preview-db-provision-job.yaml b/deploy/helm/studio/templates/preview-db-provision-job.yaml new file mode 100644 index 0000000000..fa903386e4 --- /dev/null +++ b/deploy/helm/studio/templates/preview-db-provision-job.yaml @@ -0,0 +1,90 @@ +{{- if .Values.preview.enabled }} +{{- /* +PreSync hook (weight -10): creates the per-PR database and bucket before +anything else in the sync runs. + +An Argo hook rather than an initContainer because an initContainer runs once +per pod (so it re-races on scale-up) and has no counterpart on delete. The +PostDelete teardown Job is the other half of this pair. + +Both steps are idempotent: a resync of an existing preview is a no-op. +*/ -}} +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ include "chart-deco-studio.fullname" . }}-preview-provision + labels: + {{- include "chart-deco-studio.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-10" + helm.sh/hook-delete-policy: before-hook-creation + argocd.argoproj.io/hook: PreSync + argocd.argoproj.io/hook-delete-policy: BeforeHookCreation + argocd.argoproj.io/sync-wave: "-10" +spec: + backoffLimit: 3 + template: + metadata: + labels: + {{- include "chart-deco-studio.podLabels" . | nindent 8 }} + spec: + restartPolicy: OnFailure + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if .Values.preview.objectStorage.manageBucket }} + initContainers: + - name: create-bucket + image: "{{ .Values.preview.mcImage.repository }}:{{ .Values.preview.mcImage.tag }}" + imagePullPolicy: {{ .Values.preview.mcImage.pullPolicy }} + env: + - name: S3_ENDPOINT + value: {{ .Values.preview.objectStorage.endpoint | quote }} + - name: ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.preview.objectStorage.credentialsSecret.name }} + key: {{ .Values.preview.objectStorage.credentialsSecret.accessKeyIdKey }} + - name: SECRET_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.preview.objectStorage.credentialsSecret.name }} + key: {{ .Values.preview.objectStorage.credentialsSecret.secretAccessKeyKey }} + command: ["/bin/sh", "-c"] + args: + - | + set -eu + mc alias set preview "$S3_ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" + mc mb --ignore-existing "preview/{{ include "chart-deco-studio.previewBucket" . }}" + resources: + {{- toYaml .Values.preview.jobResources | nindent 12 }} + {{- end }} + containers: + - name: create-database + image: "{{ .Values.preview.postgresImage.repository }}:{{ .Values.preview.postgresImage.tag }}" + imagePullPolicy: {{ .Values.preview.postgresImage.pullPolicy }} + env: + - name: ADMIN_URL + valueFrom: + secretKeyRef: + name: {{ .Values.preview.dbAdminSecret.name }} + key: {{ .Values.preview.dbAdminSecret.key }} + command: ["/bin/sh", "-c"] + args: + - | + set -eu + # CREATE DATABASE cannot run inside a transaction or a DO block, + # so \gexec is the only idempotent form. + psql "$ADMIN_URL" -v ON_ERROR_STOP=1 -t <<'SQL' + SELECT 'CREATE DATABASE "{{ include "chart-deco-studio.previewDbName" . }}"' + WHERE NOT EXISTS ( + SELECT 1 FROM pg_database + WHERE datname = '{{ include "chart-deco-studio.previewDbName" . }}' + )\gexec + SQL + echo "database {{ include "chart-deco-studio.previewDbName" . }} ready" + resources: + {{- toYaml .Values.preview.jobResources | nindent 12 }} +{{- end }} diff --git a/deploy/helm/studio/templates/preview-db-teardown-job.yaml b/deploy/helm/studio/templates/preview-db-teardown-job.yaml new file mode 100644 index 0000000000..754159ac8d --- /dev/null +++ b/deploy/helm/studio/templates/preview-db-teardown-job.yaml @@ -0,0 +1,83 @@ +{{- if .Values.preview.enabled }} +{{- /* +PostDelete hook: destroys the per-PR database and bucket when the preview goes +away. This is the half of the lifecycle that a GitHub-Actions teardown job +cannot do reliably — if that job is cancelled or never runs, the database +leaks. Argo runs this whenever the Application is deleted. + +DROP DATABASE ... WITH (FORCE) matters: the pods' connection pools may not have +fully drained when this runs, and a plain DROP would fail on the open backends. +*/ -}} +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ include "chart-deco-studio.fullname" . }}-preview-teardown + labels: + {{- include "chart-deco-studio.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-delete + helm.sh/hook-delete-policy: before-hook-creation + argocd.argoproj.io/hook: PostDelete +spec: + backoffLimit: 3 + template: + metadata: + labels: + {{- include "chart-deco-studio.podLabels" . | nindent 8 }} + spec: + restartPolicy: OnFailure + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if .Values.preview.objectStorage.manageBucket }} + initContainers: + - name: remove-bucket + image: "{{ .Values.preview.mcImage.repository }}:{{ .Values.preview.mcImage.tag }}" + imagePullPolicy: {{ .Values.preview.mcImage.pullPolicy }} + env: + - name: S3_ENDPOINT + value: {{ .Values.preview.objectStorage.endpoint | quote }} + - name: ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.preview.objectStorage.credentialsSecret.name }} + key: {{ .Values.preview.objectStorage.credentialsSecret.accessKeyIdKey }} + - name: SECRET_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.preview.objectStorage.credentialsSecret.name }} + key: {{ .Values.preview.objectStorage.credentialsSecret.secretAccessKeyKey }} + command: ["/bin/sh", "-c"] + args: + # Guarded rather than `|| true`: an already-absent bucket is + # expected, any other mc failure should still fail the Job. + - | + set -eu + mc alias set preview "$S3_ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" + if mc ls "preview/{{ include "chart-deco-studio.previewBucket" . }}" >/dev/null 2>&1; then + mc rb --force "preview/{{ include "chart-deco-studio.previewBucket" . }}" + fi + resources: + {{- toYaml .Values.preview.jobResources | nindent 12 }} + {{- end }} + containers: + - name: drop-database + image: "{{ .Values.preview.postgresImage.repository }}:{{ .Values.preview.postgresImage.tag }}" + imagePullPolicy: {{ .Values.preview.postgresImage.pullPolicy }} + env: + - name: ADMIN_URL + valueFrom: + secretKeyRef: + name: {{ .Values.preview.dbAdminSecret.name }} + key: {{ .Values.preview.dbAdminSecret.key }} + command: ["/bin/sh", "-c"] + args: + - | + set -eu + psql "$ADMIN_URL" -v ON_ERROR_STOP=1 \ + -c 'DROP DATABASE IF EXISTS "{{ include "chart-deco-studio.previewDbName" . }}" WITH (FORCE)' + echo "database {{ include "chart-deco-studio.previewDbName" . }} dropped" + resources: + {{- toYaml .Values.preview.jobResources | nindent 12 }} +{{- end }} diff --git a/deploy/helm/studio/templates/preview-migrate-job.yaml b/deploy/helm/studio/templates/preview-migrate-job.yaml new file mode 100644 index 0000000000..07b000257d --- /dev/null +++ b/deploy/helm/studio/templates/preview-migrate-job.yaml @@ -0,0 +1,65 @@ +{{- if .Values.preview.enabled }} +{{- /* +PreSync hook (weight 0): the single migration writer for a preview. + +Runs after the provision Job and before any pod starts, so the Better Auth + +Kysely migrations and the DBOS system-schema migration each happen exactly +once. Pods then boot with --skip-migrations (enforced by validatePreview). + +Both steps are needed. --skip-migrations only covers studio's own migrations; +DBOS still migrates its `dbos` schema on launch, and parallel pod boots race on +inserts into dbos.dbos_migrations — see tests/multi-pod/docker-compose.yml. +*/ -}} +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ include "chart-deco-studio.fullname" . }}-preview-migrate + labels: + {{- include "chart-deco-studio.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation + argocd.argoproj.io/hook: PreSync + argocd.argoproj.io/hook-delete-policy: BeforeHookCreation + argocd.argoproj.io/sync-wave: "0" +spec: + backoffLimit: 3 + template: + metadata: + labels: + {{- include "chart-deco-studio.podLabels" . | nindent 8 }} + spec: + restartPolicy: OnFailure + serviceAccountName: {{ include "chart-deco-studio.serviceAccountName" . }} + securityContext: + {{- include "chart-deco-studio.podSecurityContext" . | nindent 8 }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: migrate + {{- with .Values.securityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + image: "{{ .Values.image.repository }}{{- if and .Values.image.tag (hasPrefix "sha256:" .Values.image.tag) }}@{{ .Values.image.tag }}{{- else }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{- end }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + envFrom: + - configMapRef: + name: {{ include "chart-deco-studio.fullname" . }}-config + - secretRef: + name: {{ include "chart-deco-studio.secretName" . }} + command: ["/bin/sh", "-c"] + args: + # Absolute bundle paths, not `bun run