feat: report migration progress and let a slow one survive its supervisor

[FrameAutomata]

Closes #337.

The problem

runMigrationsOn ran every statement as a bare Exec — no transaction, no timing, no logging — and migrations.Run is called synchronously from cmd/run.go before the HTTP server binds. A boot that was still working looked exactly like one that had hung. #335 measured a single CREATE INDEX on exception_stack_traces at ~10s on a 2M-row / 8.1GB telemetry database.

Being killed part-way through does not repair itself, and I found the two storage families fail differently — the issue described only one of them:

	on kill mid-migration
ClickHouse / Postgres	golang-migrate calls SetVersion(target, **true**) before Run (migrate.go:738), so the database comes back with a dirty schema_migrations and refuses to boot until someone runs force
SQLite / DuckDB	the version is recorded after the statements, and a file's statements are separately auto-committed, so a multi-statement file is left half applied

The SQLite case is worse than "re-runs and is killed again". sqlite_telemetry/0017 is five ALTER TABLE ... ADD COLUMN statements before a CREATE INDEX; those aren't idempotent, so a kill during the index build means the retry dies on duplicate column name — and so does every boot after it. sqlite/0003, sqlite_telemetry/0009 and duckdb_telemetry/0002 have the same shape.

Making that atomic is deliberately not in this PR, per #337's own note: it changes behaviour for every migration, not just the slow ones, and deserves its own look. What this does is stop supervisors killing the process in the first place, and say what is happening while it works.

What changed

Progress logging. Each pending migration is announced before it runs and reported again with elapsed time. The announcement comes first so the newest line names whatever is still running, not what already finished. Already-applied migrations are skipped before the log call, so a steady-state restart is silent — only a boot that actually applies something costs lines.

The golang-migrate paths too. ClickHouse and Postgres don't go through runMigrationsOn, and that's the chart's default variant — fixing only the embedded path would have left the exact deployment shape this is about silent. Verbose() has to be on to get the same guarantee: migrate names a migration before running it only on its verbose stream. The buffering internals riding that stream are dropped by prefix — a skip list rather than a keep list, so migrate's own error lines still reach the log.

A startupProbe in the chart. Kubernetes withholds liveness and readiness until it first succeeds, so the ~70-100s liveness budget only starts once the process is serving. 100 x 5s is deliberately under the Deployment's default progressDeadlineSeconds (600s): a migration the probe tolerates must not simultaneously be reported as a failed rollout, because helm --wait --atomic or a GitOps self-heal answers that by deleting the pod mid-migration — causing precisely what the probe prevents. Raising it past that ceiling means raising progressDeadlineSeconds too, which the values comment says.

Docker had the same bug. The four per-variant images and their compose files gave /health a 5s health-check start period, marking a container unhealthy ~95s in and breaking --wait, depends_on: condition: service_healthy, and orchestrators that restart on health state. Docker ends the start period at the first success, so raising it to 300s only ever costs a slow first boot.

CI. migrate_logger_test.go is build-tagged, and the tagged job only ran ./app/repositories/... ./app/chdb/... — so the test would have run in no job at all. ./app/migrations/... is added to that list, which is the hazard the workflow's own comment already documents for app/chdb.

Validation

• go vet and go test clean on the default tags and on transactional_pg telemetry_ch (the combo CI uses)
• gofmt clean; default go build ./... clean
• helm lint passes; chart renders valid YAML across all three variants, with the probe present by default, removed under startupProbe.enabled=false, and honouring an overridden failureThreshold
• Verified empirically that a fresh SQLite install logs paired lines per migration and a second run is completely silent

Two tests are added: one pins the announce-before-work and silent-re-run properties on the embedded path, the other pins that the logger's skip list keeps migrate's error lines visible — swapping it for a keep list would silently swallow the reason a migration failed.

Deliberately left out

• Atomic migrations (#337 item 3) — the real fix for the half-applied-file case above, but it changes behaviour for every migration
• Boot-phase logging generally — backfill.RunDashboards() blocks on advisory lock 824737001 silently, and cache.ProjectCache.Init and PostStartupHooks are pre-bind and silent too. A step("name", ...) wrapper in run.go would generalize this; the startupProbe budget already covers them
• systemd — notifySystemd() sends READY=1 only after migrations, so a Type=notify unit with the default TimeoutStartSec=90 reproduces the same kill. EXTEND_TIMEOUT_USEC during migration is the equivalent fix
• Making all three probes values-driven — liveness and readiness stay hardcoded; this exposes only the budget knob a slow migration actually needs

Happy to file follow-up issues for any of those.

🤖 Generated with Claude Code

https://claude.ai/code/session_013b8tjKviUGfyVkrF7ccQkm

---

Generated by Claude Code

[dusanstanojeviccs]

Hard to tell, making migrations transactional means we block during a migration, more consistency BUT the extra safety comes with the penalty of downtime while a migration is happening. Right now in case of a blue green release while a migration is going requests from the old instance will still be served (with the old queries) when the new instance is up the traffic will move to it, unless the db changes shape in a way that the old instance can’t read this will be a 0s downtime deployment. By making this transactional no instance can’t safely serve requests while migration is going. Technically more accurate, but functionally potentially a worse experience. I’ll think about it more.

[FrameAutomata]

That tradeoff is real in general, but I think it mostly dissolves here, because the cost and the benefit land on different deployment shapes. Three things I checked:

Postgres / ClickHouse — the blue/green shape you're describing. Every file in pg/ and ch/ is a single statement (the one-statement-per-file rule in CLAUDE.md — I checked pg/, none carries more than one). A lone statement already runs in its own implicit transaction under autocommit, so those migrations are atomic today. golang-migrate doesn't add one of its own either: Run() goes straight to runStatement() → ExecContext (v4.19.1, database/postgres/postgres.go:265); its only transaction is in SetVersion. So "make it transactional" there would just fold the version bookkeeping in with the DDL. The lock is held for as long as the CREATE INDEX / ALTER TABLE takes either way — the delta is one UPDATE. It wouldn't lengthen the window the old instance has to cover.

SQLite / DuckDB — where multi-statement files actually exist, and therefore where the half-applied-file hazard is. But that's the embedded single-process deployment: the database is a local file, so there's no second instance sharing it to serve traffic while the migration runs. Blue/green isn't the model there. And it's opened in WAL (backend/app/db/db_sqlite_open.go:23), where readers aren't blocked by a writer anyway.

So the downtime would be bought mainly on the shape that doesn't need the fix, and the shape that does need it isn't the one doing zero-downtime cutovers.

Two honest caveats. It stops holding for Postgres the moment a pg/ file carries more than one statement — so that convention would need to stay a rule rather than an accident, ideally enforced. And a non-concurrent CREATE INDEX blocks writes on that table regardless of transactions; that's true today and nothing here changes it.

None of this is in this PR either way — it only stops supervisors killing the process mid-migration, and leaves atomicity exactly as it is. Happy to open a separate issue for the deferred piece so it has somewhere to live while you think it over.

---

Generated by Claude Code