Housekeeping and prep for the catalog-test harness - #3415
Merged
Conversation
Routine refresh of a stale pin — `usage` parses the `#USAGE` specs in `mise/tasks/`, and 3.5.4 is three majors behind. No task's spec changes; the `--help` and flag handling of every task that declares one is unaffected.
These were point-in-time design docs for completed or superseded work; drop the dangling references to plans/runtime-v2/plan.md in the two READMEs that cited it.
The shell-driven end-to-end suite under tests/ has not been exercised by CI: nothing in .github/ references tests/run-all.sh, so these catalogs, row fixtures, and the SSH-forwarding docker-compose scaffolding have been dead weight. The sshforwarding fixtures also carried committed SSH private keys. `flowctl-go api await` existed solely to serve run-end-to-end.sh, and goes with it. ghcr.io/estuary/source-test:dev remains referenced by go/bindings/testdata/build.flow.yaml and crates/shuffle/tests/*.flow.yaml, which are unaffected by removing tests/source-test/.
`shuffle: any` compiles to an empty shuffle key, which both runtimes resolve to the *source* collection key. Only documents of one shuffle key reduce in source order, so a derivation with an order-dependent reduction must shuffle on a key that pins each of its output keys to a single shard -- which the source key does not, in these three cases. `stations` flaked under multi-shard catalog tests (~35% of runs). A bike's relocation-arrival at a station and its ride-departure from that station are separate source keys, so they landed on different shards, and their `stable` set add / remove then reduced in whichever order the two shards' drains raced to. Shuffling on /bike_id keeps a bike's events on one shard and in order; events of *different* bikes commute, because the set is keyed on the bike ID and the arrival / departure counters are sums. `last-seen` (keyed on the bike, last-write-wins) and `stock/daily-stats` (firstWriteWins / lastWriteWins over a day's ticks) carry the same defect. Neither test can expose it today -- neither puts two documents of one output key into a single transaction, so the race window never opens -- but both specs are wrong for multi-shard operation.
`doc::diff` compared floats under a magnitude-scaled epsilon, but only when both sides were floats. An integer against a float fell through to the exact `json::node::compare`, which orders `0` before `0.0` — a distinction JSON Schema does not draw, and neither should a diff. Meanwhile a reduction that lands on `8.000000000000002` where the expectation says `8` is floaty funny bitness, not a difference worth reporting. Both sides are now taken as `f64` and compared under the same scaled epsilon whenever either side is a float. Two integers still compare exactly: the epsilon scales with magnitude, so admitting it there would collapse distinct IDs and neighboring nanosecond timestamps. Also note a location present in the actual but absent from the expectation, which previously reported an unannotated `actual` with no hint of why it differed.
This was referenced Aug 27, 2026
github-actions
Bot
added
pending:agent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Stack 1 of 5, replacing the V1 catalog-test machinery with a Rust harness. This one clears the decks: three commits delete dead scaffolding, two are small correctness fixes that the rest of the stack depends on.
Read these two:
examples: shuffle derivations on the key their reductions require—shuffle: anycompiles to an empty shuffle key, which both runtimes resolve to the source collection key. A derivation with an order-dependent reduction must shuffle on a key that pins each of its output keys to a single shard, which the source key does not in these three cases.citi-bike/stationsflakes ~35% of runs under multi-shard catalog tests;last-seenandstock/daily-statscarry the same defect but can't expose it today.doc: compare numbers of differing representation under an epsilon—doc::diffcompared floats under a magnitude-scaled epsilon only when both sides were floats. An integer against a float fell through to exact comparison, which orders0before0.0— a distinction JSON Schema does not draw. Two integers still compare exactly.Skim these three:
mise: pin usage 6.4.0— routine pin refresh, three majors behind. No task's#USAGEspec changes.plans: retire plans/ directory— point-in-time design docs for completed or superseded work, plus the two READMEs that cited them.tests: retire the dead shell end-to-end suite— nothing in.github/referencestests/run-all.sh, so these catalogs, row fixtures, and the SSH-forwarding docker-compose scaffolding have been dead weight. The sshforwarding fixtures carried committed SSH private keys.flowctl-go api awaitexisted solely to serverun-end-to-end.shand goes with it.Workflow steps:
No user-facing change.
Documentation links affected:
None.
Notes for reviewers:
ghcr.io/estuary/source-test:devremains referenced bygo/bindings/testdata/build.flow.yamlandcrates/shuffle/tests/*.flow.yaml, which are unaffected by removingtests/source-test/.The examples change lands here, ahead of the stack, because PR 3 switches CI's
ci:catalog-testto the multi-shard runtime-next runner — landing the spec fix first keeps every rung green. The one thing to watch on this PR is that the examples still pass under V1flowctl-go test, which is what CI runs at this point in the stack.Stack — each PR is based on the one above it, so review only the top commits of each.
flowctl raw test#3417 — catalog-tests: the harness, andflowctl raw test