test: clone a per-binary schema template per fixture and shard the largest packages - #736
Closed
mariusvniekerk wants to merge 3 commits into
Closed
test: clone a per-binary schema template per fixture and shard the largest packages#736mariusvniekerk wants to merge 3 commits into
mariusvniekerk wants to merge 3 commits into
Conversation
Every push to main since #717 failed in test-postgres and test-pgvector. Small packages died in under a second with "sorry, too many clients already", the pgvector schema migration hit "out of shared memory", and the heavy packages then ran into the 60-minute timeout. Each PostgreSQL-backed test binary opens its own connections: two for the admin handle, four for a warm-pool schema refill, plus the store under test. `go test ./...` starts up to one binary per CPU, and nothing budgets connections across binaries. On the managed runner the twenty-four packages that use PostgreSQL started together and exceeded the service container's 100-connection limit in the first seconds. GitHub-hosted runners have four CPUs, so the same lanes passed there with -p effectively at 4. The Makefile's PostgreSQL targets and the inline pgvector lane now pass -p 4 (PG_TEST_PARALLEL), so the connection footprint no longer depends on the host's CPU count. The pgvector service also gets the max_locks_per_transaction=256 setting the test-postgres service already had; its migration ran out of exactly that. Slower per-package times on the managed runner are a separate matter: the cap reduces contention on the one container but does not change the runner's I/O. Generated with Claude Code (claude-fable-5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every test fixture used to replay InitSchema() into an empty database: a few hundred DDL statements, ~140ms on SQLite and ~1.5s on PostgreSQL per fixture on a laptop, and several times that on a contended CI runner. With about a thousand fixtures across the suite, that replay was most of the wall clock of the largest packages — internal/store spent 39 minutes on the managed runner for tests that do milliseconds of real work each. The initialized schema is a pure function of the test binary, so each binary now builds it once and hands every fixture a copy. On SQLite the template is a file written into the test's temp dir. On PostgreSQL it is a template database and each fixture is a CREATE DATABASE ... TEMPLATE clone, a file-level copy the server makes in tens of milliseconds. Locally internal/store drops from 241s to 120s on SQLite, and a PostgreSQL fixture from ~750ms to ~200ms. A template database has to be reclaimed when the binary that built it is gone. Ownership is a session advisory lock held on a connection pinned for the binary's life; the server releases it on any exit, so the next binary reclaims whatever it can lock. That replaces the warm pool's pid-namespace sweep, which could not run on hosts without /proc and so left macOS on the slow path. A role without CREATEDB falls back to the per-schema fixture. One thing a clone must not copy: InitSchema() mints a durable archive UID, and tests that hand one archive's cursor to another depend on fixtures being distinct archives. The fixture assigns each clone its own UID. Generated with Claude Code (claude-fable-5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cmd/msgvault/cmd, internal/store and internal/api hold 1,300–2,000 tests each, and a Go test binary runs them one at a time. Whichever of the three is slowest sets the wall clock of the whole Linux lane no matter how many cores the runner has — 39 minutes for internal/store on the managed runner before the template fixtures, and still the critical path after them. The Windows lane already solves this with scripts/test-package-shards.ps1: compile the package once, list its tests, deal them into shards, and run each shard as its own process of the same binary. This adds the Linux twin and uses it for those three packages in their own matrix jobs, on both the SQLite lane and the PostgreSQL lane, where each sharded job also gets its own service container so four shards' connections never compete with the rest of the suite. The existing lanes run every other package through the new *-unsharded targets; `make test` and `make test-pg-shipped` still run everything, so a local run is unchanged. Locally the three packages take 225s, 120s and 125s whole; in four shards each they take 173s in total. Generated with Claude Code (claude-fable-5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
roborev: Combined Review (
|
wesm
force-pushed
the
t3code/fix-main-ci-failures-1
branch
from
September 2, 2026 10:32
69aef08 to
3e097ae
Compare
Member
|
Consolidated these changes into #735, which is now rebased directly on main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
InitSchema()per test. Each test binary builds the schema once and clones it: a file copy on SQLite,CREATE DATABASE … TEMPLATEon PostgreSQL (internal/testutil/sqlite_template.go,internal/testutil/pg_template.go). The PostgreSQL warm pool and its/proc-based sweep are gone; template ownership is a session advisory lock the server releases when the binary exits, so it works on macOS too. A role withoutCREATEDBfalls back to the per-schema fixture (MSGVAULT_TEST_PG_TEMPLATE=0forces it).InitSchema()mints one per database and tests that hand one archive's cursor to another rely on fixtures being distinct archives.cmd/msgvault/cmd,internal/storeandinternal/apirun as 4-way shards in their own CI jobs (test-sharded,test-postgres-sharded, each PostgreSQL shard job with its own service container) viascripts/test-package-shards.sh, the Linux twin of the Windows lane's script. Thetestandtest-postgreslanes run everything else (make test-unsharded,make test-pg-shipped-unsharded);make testandmake test-pg-shippedare unchanged.Why
The three largest packages hold 1,300–2,000 tests each, every test built a fresh database from ~330 DDL statements, and a test binary runs its tests one at a time — so
internal/storealone took 39 minutes on the managed runner and set the wall clock of the whole lane.Measured locally: a SQLite fixture goes from ~140 ms to ~12 ms and a PostgreSQL fixture from ~1.5 s to ~60 ms;
internal/storeon SQLite drops from 241 s to 120 s whole, and the three packages together from 470 s whole to 173 s in shards. On PostgreSQL,internal/api's 970 tests run in 69 s across four shards.Usage
Stacked on #735. The new jobs and lane flags were exercised pre-merge with
gh workflow run ci.yml --ref <branch>(PR runs useci.yml@main): https://github.com/kenn-io/msgvault/actions/runs/33529929253 — 21/21 jobs green in 26 minutes on GitHub-hosted runners.generated by a clanker
🤖 Generated with Claude Code