Skip to content

Faster structure fingerprints for doc clustering - #6867

Draft
fmassot wants to merge 1 commit into
fmassot/borrowed-doc-parsingfrom
fmassot/faster-structure-fingerprint
Draft

fmassot wants to merge 1 commit into
fmassot/borrowed-doc-parsingfrom
fmassot/faster-structure-fingerprint

Conversation

@fmassot

@fmassot fmassot commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #6866.

Description

The structure clustering policy hashes the sorted list of the document's leaf paths. Today it collects every leaf path in its own Vec<&str>, sorts the list, hashes it, and then frees each vector. With document clustering enabled, this was the largest cost of fingerprinting, and fingerprinting was about 40% of DocProcessor time.

When serde_json::Map iterates in sorted key order (the default, without preserve_order), a depth-first walk already visits the leaf paths in sorted order:

  • paths under a key are compared on that key first, and the keys of an object are unique and visited in increasing order;
  • a leaf path is never a prefix of another leaf path, because a leaf has no children.

The paths are therefore hashed as they are visited, with no allocation or sort. With preserve_order (enabled by some test dependencies), the previous collect-and-sort implementation is kept.

The fingerprints are unchanged. That matters because fingerprints in a split are compared with each other.

How was this PR tested?

  • Golden test: pins the structure fingerprints computed by the previous implementation on documents with nested objects, keys sharing prefixes, unicode keys, duplicate keys, and excluded paths. It checks both owned and borrowed documents.
  • Differential test: compares the streaming hash with the collect-and-sort hash on those documents plus 10,000 random ones, for owned and borrowed views, with and without exclusions.
  • Both modes: the tests pass with and without serde_json/preserve_order.
  • Mutation check: always streaming under preserve_order, and ignoring exclusions while streaming, are both caught.

Benchmark. Parquet local ingest of OTel logs on top of #6843 and #6866, validation disabled, clustering policies structure, raw ServiceName, raw SeverityText, tokenized Body. Two alternating runs on an M3 Max:

Pipelines Before (docs/s) After (docs/s)
1 202k–206k 224k–229k (+11%)
8 508k–533k 586k–657k (noisy)

When object keys iterate in sorted order (serde_json without preserve_order),
a depth-first walk visits leaf paths in sorted order, so they can be hashed
as they are visited instead of being collected in vectors and sorted.

The fingerprints are unchanged: golden values and a differential test against
the collect-and-sort implementation check it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant