flowctl: parse fixture lines once and prepare them on worker threads - #3446
Open
mdibaiee wants to merge 2 commits into
Open
flowctl: parse fixture lines once and prepare them on worker threads#3446mdibaiee wants to merge 2 commits into
mdibaiee wants to merge 2 commits into
Conversation
added 2 commits
August 30, 2026 11:58
…ows a segment Blocks above the writer's compress threshold are already LZ4-compressed, so compressing a whole sealed segment can grow it. The reclaim stream then credits the larger size on unlink, more than was charged when the segment rolled, and the checked subtraction panicked. Any connector slower than COMPRESS_AFTER per transaction hit it. Closes #3443
The preview-next fixture feeder parsed each line into a serde_json::Value twice, cloned it, rebuilt it as a HeapNode, validated and keyed it, all on one task. Lines now parse once, straight into a HeapNode, and streamed lines are batched and prepared on worker threads, then pushed in fixture order so blocks and clocks are unchanged. --fixture-validate=false skips schema validation for documents known to be valid. Feeding a 25M-document transaction on a 22-core machine: 123s to 46-52s.
jacobmarble
previously approved these changes
Aug 31, 2026
Base automatically changed from
mahdi/preview-next-fixture-backlog-underflow
to
master
August 31, 2026 22:25
Member
|
Please hold off on merging this, until this code refactor move lands? #3416 |
Member
Author
|
@jgraettinger okay 👍🏽 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
The
preview-nextfixture feeder parsed every line into aserde_json::Valuetwice, cloned it, rebuilt it as aHeapNode, validated and keyed it, all on one task. Feeding a 25M-document transaction took ~123s on a 22-core machine, and every shard of a multi-shard preview waited on it.HeapNodewith simd-doc; the synthetic UUID is set on the heap node; one bump allocator is reused.is_commit_lineonly parses lines that can be a marker.--fixture-validate=falseskips schema validation for documents known to be valid, such as generated benchmark fixtures.Same machine, same transaction: 123s → 96s (single parse) → 82s (no validation) → 46–52s (workers).
Workflow steps:
flowctl raw preview-next --fixture ... [--fixture-validate=false]. Output is unchanged.Documentation links affected:
None.
Notes for reviewers: