Conversation
When object keys iterate in sorted order (serde_json without preserve_order), a depth-first walk visits leaf paths in sorted order, so they can be hashed as they are visited instead of being collected in vectors and sorted. The fingerprints are unchanged: golden values and a differential test against the collect-and-sort implementation check it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #6866.
Description
The
structureclustering policy hashes the sorted list of the document's leaf paths. Today it collects every leaf path in its ownVec<&str>, sorts the list, hashes it, and then frees each vector. With document clustering enabled, this was the largest cost of fingerprinting, and fingerprinting was about 40% of DocProcessor time.When
serde_json::Mapiterates in sorted key order (the default, withoutpreserve_order), a depth-first walk already visits the leaf paths in sorted order:The paths are therefore hashed as they are visited, with no allocation or sort. With
preserve_order(enabled by some test dependencies), the previous collect-and-sort implementation is kept.The fingerprints are unchanged. That matters because fingerprints in a split are compared with each other.
How was this PR tested?
serde_json/preserve_order.preserve_order, and ignoring exclusions while streaming, are both caught.Benchmark. Parquet local ingest of OTel logs on top of #6843 and #6866, validation disabled, clustering policies
structure, rawServiceName, rawSeverityText, tokenizedBody. Two alternating runs on an M3 Max: