catalog-tests: the catalog test harness, and `flowctl raw test` by jgraettinger · Pull Request #3417 · estuary/flow

jgraettinger · 2026-08-27T00:24:46Z

Description:

Stack 3 of 5, and the core of it: a new catalog-tests crate, layered above runtime-local (#3416) and runtime-next. Plus the one protocol addition it needs, and the CLI that drives it.

Five commits:

proto: add derive Reset / ResetDone messages — the mechanism the harness uses to isolate test cases from one another.
runtime-next: forward Reset to derive connectors — handled shard-locally; the leader is not involved.
derive-sqlite: report no checkpoint for an ephemeral database — a :memory: database is not remote-authoritative, and saying so is what lets the harness run derive-sqlite multi-shard.
catalog-tests: the catalog test harness — the crate.
flowctl: add the raw test subcommand — and CI's ci:catalog-test task now uses it instead of flowctl-go test.

Workflow steps:

flowctl raw test --source examples/flow.yaml

A line per test, keyed by test name. A failure names the source path and JSON pointer of the step it failed at and indents the rendered diff beneath; a case the run never reached says so. Then a tally. Exits non-zero if any test failed.

Documentation links affected:

None user-facing. crates/catalog-tests/README.md is new and carries the design.

Notes for reviewers:

Stack — each PR is based on the one above it, so review only the top commits of each.

Housekeeping and prep for the catalog-test harness #3415 — Housekeeping and prep (merged)
runtime-local: extract the local task-drive layer from flowctl #3416 — runtime-local: extract the local task-drive layer
catalog-tests: the catalog test harness, and flowctl raw test #3417 — catalog-tests: the harness, and flowctl raw test ← you are here
control-plane-api: run publication tests on catalog-tests #3418 — control-plane-api: run publication tests on catalog-tests
Remove the V1 catalog-test machinery #3419 — Remove the V1 catalog-test machinery

Reset asks a derivation shard's connector to reset its internal state to an as-just-initialized condition, mapping onto the connector protocol's existing `derive.Request.Reset`. It is the mechanism the catalog-test harness uses to isolate test cases from one another. Reset is deliberately shard-local — Controller → Shard, with ResetDone coming back Shard → Controller. The leader is not in the path: coordinating a common transaction boundary across shards belongs to the caller, and the test harness already has that property because it drives one transaction per stat() and awaits its commit. ResetDone is nonetheless required. A shard receives transaction messages from its leader but Reset from its controller, so ordering within either stream alone cannot establish that the connector observed the reset before the next transaction's reads.

A derivation shard now answers a controller Reset by queueing a `derive::Request.Reset` to its connector and replying ResetDone. This is handled shard-locally; the leader is not involved.

`Opened.runtime_checkpoint` is how a connector tells the runtime that its checkpoint state is remote-authoritative — that the endpoint, not the recovery log, holds the truth. derive-sqlite reported one unconditionally, including for a `:memory:` database whose "truth" does not outlive the session. Introduce `Database` to decide this once, where the URI is chosen: `Durable` for a file-backed database, whose recorded checkpoint is authoritative and is reported; `Ephemeral` for `:memory:`, which reports none. An explicit `:memory:` from `sqlite_vfs_uri` is now recognized as ephemeral rather than taken at face value as a file path. This is a test-only concern in practice: a production task is always threaded a recorded recovery-log VFS URI by the runtime, and `:memory:` arises only where a local drive deliberately wants a stateless connector. What it buys is that such a drive is no longer forced single-shard by an authority claim the database can't back — which is what lets the catalog-test harness run derive-sqlite multi-shard.

New crate holding what a catalog test *means*, layered above runtime-local and runtime-next. `run_tests` starts a resident session per enabled derivation, then runs every case — sorted by first-step scope, so a run is diffable from one invocation to the next — through the scheduler, with a Reset between cases. Which derivations to start is settled synchronously first, so a malformed catalog fails before any connector boots. Every derivation runs multi-shard, whatever its connector. Sessions thread an empty `Task.sqlite_vfs_uri`, so a derive-sqlite shard runs a session-scoped `:memory:` database and claims no checkpoint authority it couldn't back — which is what leaves the leader free to accept the topology. clock / graph / scheduler carry the scheduling semantics that `go/testing/` worked out — writes cascading through a multi-hop chain, self-cycles reaching a fixed point, read delays against a lazily-advanced synthetic clock, verify steps gated on a forward BFS over pending reads — and follow the Go closely, down to its unit tests, rather than re-deriving them. Two deliberate departures: the graph tracks derivations only, since a test never runs a capture or a materialization; and there is no `ErrAdvanceDisabled`, because synthetic time is a counter local to this crate. Everything else that's non-obvious is written up under "Non-obvious details" in the crate README. Tests run over derive-sqlite and need no containers, covering scheduling, Reset isolation, partition selectors, redaction, collection names nested under one another, read delays, diff rendering, and a run dropped mid-flight. Connector death at startup — while handling Open, or racing the leader's startup commit just after Opened — is covered by `tests/fixtures/dying_connector.py`, since no well-behaved connector produces that fault.

`flowctl raw test --source ...` runs a catalog's tests locally on the runtime-next stack, and CI's catalog-test task now uses it instead of `flowctl-go test`. It sits under `raw` beside `raw preview-next`, which drives the same stack: both are local, developer-facing, and tied to a runtime that is still landing, so neither belongs at the top level yet. `flowctl catalog test` — a remote dry-run publish — is a different command and is unaffected. Output is a line per test, keyed by test name: a failure names the source path and JSON pointer of the step it failed at and indents the rendered diff beneath, and a case the run never reached says so. Then a tally. Exits non-zero if any test failed.

williamhbaker

LGTM

williamhbaker · 2026-09-01T18:17:49Z

+    graph: &mut Graph,
+    driver: &mut D,
+    test: &TestSpec,
+) -> anyhow::Result<String> {


nit: The comment says that the scope of the last step reached is returned and used for error reporting, but as I understand it the error reporting part happens on the error path, and the string value from the Ok result is never used... in crates/catalog-tests/src/run.rs:192 it is discarded. It seems like LiveDriver::last_scope is actually doing the tracking of error scopes.

Thanks, I'll address in a next PR.

jgraettinger requested a review from a team August 27, 2026 00:25

jgraettinger mentioned this pull request Aug 28, 2026

flowctl: preview runs on the V2 runtime, remove V1 preview #3442

Merged

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch from 38a589b to a2d44ab Compare August 30, 2026 17:56

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from 7c143a6 to db1ce76 Compare August 30, 2026 17:56

jgraettinger marked this pull request as draft August 31, 2026 20:38

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch 2 times, most recently from f63e934 to e392a76 Compare August 31, 2026 22:08

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from db1ce76 to df38483 Compare August 31, 2026 22:08

jgraettinger marked this pull request as ready for review August 31, 2026 22:11

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch from e392a76 to a4ad7c4 Compare August 31, 2026 22:19

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from df38483 to 5dd646c Compare August 31, 2026 22:19

jgraettinger mentioned this pull request Aug 31, 2026

flowctl: parse fixture lines once and prepare them on worker threads #3446

Open

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch from a4ad7c4 to 2d160a7 Compare August 31, 2026 22:46

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from 5dd646c to 5106b77 Compare August 31, 2026 22:46

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch from 2d160a7 to 13ac318 Compare September 1, 2026 00:51

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from 5106b77 to 0b5782c Compare September 1, 2026 00:51

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch from 13ac318 to bf1c380 Compare September 1, 2026 04:46

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from 0b5782c to dae189d Compare September 1, 2026 04:46

jgraettinger force-pushed the johnny/v2-tests-2-runtime-local branch from bf1c380 to 8426df8 Compare September 1, 2026 15:34

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from dae189d to 0094e7a Compare September 1, 2026 15:34

Base automatically changed from johnny/v2-tests-2-runtime-local to master September 1, 2026 16:17

jgraettinger added 5 commits September 1, 2026 11:17

runtime-next: forward Reset to derive connectors

2b08085

A derivation shard now answers a controller Reset by queueing a `derive::Request.Reset` to its connector and replying ResetDone. This is handled shard-locally; the leader is not involved.

jgraettinger force-pushed the johnny/v2-tests-3-catalog-tests branch from 0094e7a to a2e9974 Compare September 1, 2026 16:17

williamhbaker approved these changes Sep 1, 2026

View reviewed changes

jgraettinger merged commit c944ab6 into master Sep 1, 2026
11 checks passed

jgraettinger deleted the johnny/v2-tests-3-catalog-tests branch September 1, 2026 19:49

Navigation Menu

Uh oh!

catalog-tests: the catalog test harness, and `flowctl raw test` - #3417

catalog-tests: the catalog test harness, and `flowctl raw test`#3417
jgraettinger merged 5 commits into
masterfrom
johnny/v2-tests-3-catalog-tests

jgraettinger commented Aug 27, 2026 •

edited

Loading

Uh oh!

williamhbaker left a comment

Uh oh!

williamhbaker Sep 1, 2026

Uh oh!

jgraettinger Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jgraettinger commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

williamhbaker left a comment

Choose a reason for hiding this comment

Uh oh!

williamhbaker Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

jgraettinger Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jgraettinger commented Aug 27, 2026 •

edited

Loading