Skip to content

flowctl: preview runs on the V2 runtime, remove V1 preview - #3442

Merged
jgraettinger merged 2 commits into
masterfrom
johnny/flowctl-preview-v2-migration-856bc7
Sep 1, 2026
Merged

flowctl: preview runs on the V2 runtime, remove V1 preview#3442
jgraettinger merged 2 commits into
masterfrom
johnny/flowctl-preview-v2-migration-856bc7

Conversation

@jgraettinger

@jgraettinger jgraettinger commented Aug 28, 2026

Copy link
Copy Markdown
Member

Replaces the V1-runtime-backed flowctl preview with the runtime-next + shuffle
implementation developed as flowctl raw preview-next, and deletes V1 preview
along with the runtime harness it was the sole consumer of.

raw preview-next is retained as a hidden, deprecated alias onto the same
clap::Args struct, so existing callers — estuary/connectors CI, the in-repo
pytest suite — keep working untouched.

Commits

  1. flowctl: preview runs on the V2 runtime — re-point Command::Preview,
    hide + deprecate the raw preview-next alias, delete
    flowctl/src/preview/{mod,journal_reader}.rs (−733).
  2. runtime: remove the now-dead V1 preview harness — delete
    runtime/src/harness/ and the private exchange combinator (−1131). Split
    out so the first commit proves the harness has no remaining callers and
    this one reviews as a pure deletion.

Behavior changes

The flag surface is a strict superset (--shards and --debug-port are
additive; nothing is removed). Three behaviors do change:

  • Derivation stdout gains ["collection/name",<doc>] framing, which
    captures and materializations already used. Stripping the wrapper reproduces
    the old output exactly. Nothing downstream parses it. The three site/
    derivation tutorials show bare documents and are now stale — but site/ no
    longer serves docs.estuary.dev (90e8261), so that correction belongs in
    estuary/docs and is deliberately not made here.
  • --delay with --fixture is now an error. A fixture's transaction
    boundaries are fixed by its own commit markers, so a raised minimum
    transaction duration has nothing to batch.
  • Unset --sessions no longer equals --sessions -1 for captures. Unset
    holds an EOF'd session through the capture's poll interval as production does;
    -1 stops at EOF (f259b8f).

Why the module isn't moved to preview/

flowctl preview is backed by a module still named raw/preview_next/. That is
deliberate. #3416 renames five of that module's files out to a new
crates/runtime-local crate and rewrites the rest, so moving now means
rename-vs-rename conflicts on every one — and would land the wrong layout
regardless, since after that split the code deserving the name preview/ is only
the flowctl-specific residue.

git merge-tree against the rebased #3415#3419 stack reports zero conflicts
in either direction. The only file both touch is raw/mod.rs (this PR edits the
PreviewNext arm; #3417 adds a Test arm beside it), and the merged result was
checked by hand to be semantically correct.

A TODO(johnny) on raw::Command::PreviewNext records the deferred move, the
in-repo callers that must migrate first (tests/preview/*.py and the
tests/soak/** comments, both introduced by #3416), and the eventual removal of
the alias.

Verification

  • Both crates build clean under mise with no unused-code warnings — the
    compiler is the proof the harness is dead.
  • flowctl preview --help carries the original doc text with the V2 flag set.
  • raw --help no longer lists preview-next; raw preview-next --help still
    dispatches and reports itself deprecated.

Note that the tests/*.py snapshot suite could not verify anything: it is not
wired into CI, and has been broken since 85253a1 swapped the fixture reader
without converting the checked-in fixtures. The framing claim above was instead
verified directly against the V1 and V2 snapshot artifacts.

@jgraettinger
jgraettinger force-pushed the johnny/flowctl-preview-v2-migration-856bc7 branch from 9306303 to d2606b7 Compare August 28, 2026 21:45
@jgraettinger
jgraettinger requested a review from a team August 28, 2026 21:48
williamhbaker
williamhbaker previously approved these changes Aug 31, 2026

@williamhbaker williamhbaker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jgraettinger
jgraettinger force-pushed the johnny/flowctl-preview-v2-migration-856bc7 branch from d2606b7 to 6ace886 Compare August 31, 2026 23:13
`flowctl preview` now dispatches to the runtime-next + shuffle
implementation developed as `flowctl raw preview-next`, and the legacy
V1-runtime `preview/{mod,journal_reader}.rs` is deleted. `raw
preview-next` remains as a hidden, deprecated alias onto the same
`clap::Args` struct, so existing callers keep working untouched.

The flag surface is a superset -- `--shards` and `--debug-port` are
additive and nothing is removed -- but three behaviors do change:

* Derivation documents on stdout gain the `["collection/name",<doc>]`
  framing that captures and materializations already used; stripping the
  wrapper reproduces the old output exactly. Nothing downstream parses
  it. The three site/ derivation tutorials show bare documents and are
  now stale, but site/ no longer serves docs.estuary.dev (90e8261),
  so that correction belongs in estuary/docs.

* `--delay` combined with `--fixture` is now an error. A fixture's
  transaction boundaries are fixed by its own commit markers, so there
  is nothing for a raised minimum transaction duration to batch.

* An unset `--sessions` no longer means the same thing as `--sessions -1`
  for captures: unset holds an EOF'd session through the capture's poll
  interval as production does, while `-1` stops at EOF (f259b8f).

The module is deliberately NOT moved to `preview/`. The in-flight
`johnny/rebase-runtime-v2-tech-debt-bab0a9` renames five of its files out
to a new `crates/runtime-local` crate and rewrites what remains, so
moving now would mean rename-vs-rename conflicts on every one -- and
would land the wrong layout regardless, since after that split the code
deserving the name `preview/` is only the flowctl-specific residue. A
TODO on `raw::Command::PreviewNext` records the deferred move, along with
the in-repo callers that must migrate before the alias can be dropped.
`flowctl preview` was the sole consumer of `runtime::harness`, and of the
private `exchange` combinator that only the harness's derive and
materialize drivers called. Both die with V1 preview.

No dependency is orphaned: `coroutines` and `tempfile` remain in use
elsewhere in the crate, so Cargo.toml is unchanged.
@jgraettinger
jgraettinger force-pushed the johnny/flowctl-preview-v2-migration-856bc7 branch from 6ace886 to bc2ca1f Compare September 1, 2026 02:53
@jgraettinger
jgraettinger merged commit 08a2aa1 into master Sep 1, 2026
11 checks passed
@jgraettinger
jgraettinger deleted the johnny/flowctl-preview-v2-migration-856bc7 branch September 1, 2026 04:46
@github-actions github-actions Bot added pending:agent Merged, in the control-plane-agent image, and not yet rolled to flow-agent pending:agent-api Merged, ships via Deploy agent-api, and not yet deployed pending:flowctl Merged, changes the flowctl binary, and not in a published release and removed pending:agent-api Merged, ships via Deploy agent-api, and not yet deployed pending:agent Merged, in the control-plane-agent image, and not yet rolled to flow-agent labels Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending:flowctl Merged, changes the flowctl binary, and not in a published release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants