Prune parquet row groups using fully dictionary-encoded columns - #23851
Draft
DarkWanderer wants to merge 5 commits into
Draft
Prune parquet row groups using fully dictionary-encoded columns#23851DarkWanderer wants to merge 5 commits into
DarkWanderer wants to merge 5 commits into
Conversation
Adds an opt-in `dictionary_filter_on_read` config option (default false) that uses a fully dictionary-encoded BYTE_ARRAY (Utf8/Binary) column chunk's dictionary as an exact row-group membership index. Unlike the existing bloom-filter pruning stage, this is exact rather than probabilistic, so it can prune both `IN`/`=` (value absent) and `NOT IN`/`!=` (value is the chunk's only value) predicates. A column chunk is only used if its page encoding stats prove every data page came from the dictionary (`PLAIN_DICTIONARY` or `RLE_DICTIONARY` only) -- chunks that fell back to `PLAIN` encoding partway through are ignored, since a writer's dictionary fallback means the dictionary is no longer the complete value set. This depends on a new arrow-rs API to decode a dictionary page independent of the row-by-row array reader (arrow-rs#9010), not yet released; see the following commit for the temporary patch.
…anch Switches the [patch.crates-io] block from a local filesystem path to https://github.com/DarkWanderer/arrow-rs/tree/get-dictionary, so the build is reproducible for anyone (including CI) rather than only on one machine. Still temporary -- to be removed once arrow-rs releases a version with the dictionary decode API and the dependency is bumped normally.
github-actions
Bot
added
documentation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
row_groups_pruned_dictionary was sorted after the page-index metrics in EXPLAIN output; group it with the other row-group-level pruning metrics (statistics, bloom filter) instead, and update the golden sqllogictest output accordingly. Also update the temporary Cargo.toml patch comment to point at arrow-rs PR apache#10420, which supersedes the now-closed apache#9011. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23851 +/- ##
==========================================
- Coverage 80.75% 80.71% -0.04%
==========================================
Files 1089 1091 +2
Lines 368809 369903 +1094
Branches 368809 369903 +1094
==========================================
+ Hits 297837 298574 +737
- Misses 53217 53547 +330
- Partials 17755 17782 +27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Thanks @DarkWanderer -- do you have any benchmarks / examples showing this in action / showing how it is faster? The idea in theory sounds reasonable but it would be good to verify it in practice |
Contributor
Author
|
Thanks for feedback @alamb Here is a benchmark on synthetic dataset mimicking the shape of observability traces data. The specific property that highlights the performance gain is - TraceIDs are pseudo-random (so stats-based pruning doesn't help), but are clustered in time. I imagine it would be applicable to any "needle in haystack" scan where the search key is clustered along sorting key. Wall-clock timeCriterion,
Dictionary pruning is ~comparable to bloom filter in wall clock, but importantly, doesn't require extra writes; also, it can be used together with bloom filtering, compounding the win. More importantly, it allows better performance for "in" and "not in" scenarios which degrade bloom performance or cannot be handled at all respectively. Bytes readThis measures the efficiency gain from more precise page pruning. This win is not fully highlighted by using local SSD - but in S3-backed object-store context, I expect that reduction to have a much more pronounced effect.
Grain of salt: this dataset mimics a very specific domain I was optimizing for, I am not sure how well the wins translate to general use cases. |
Contributor
|
Thank you for this @DarkWanderer I am sorry I haven't had a chance to review this -- it sounds great. But I am totally swamped at the moment (and for the forseeable future 😢 ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Dictionary pages are essentially free, exact indexes for
IN/NOT INqueries. Bloom filters can prove a target value is absent from a row group (useful for pruning=/IN), but they are probabilistic (non-zero false-positive rate) and cannot prove aNOT IN/!=exclusion, since that would require confirming every distinct value in the row group is excluded -- information a bloom filter doesn't retain. A fully dictionary-encoded column chunk does retain the complete set of distinct values, so it can prune exactly in both directions:IN/=when the dictionary is disjoint from the target set, andNOT IN/!=when the dictionary is a subset of the excluded set. This approach stacks with bloom filters: dictionary pruning subsumes bloom-filter pruning where it applies, and is exact (no false positives) where bloom filters are only probabilistic.What changes are included in this PR?
datafusion.execution.parquet.dictionary_filter_on_readconfig option (default
false).DictionaryStatistics(dictionary_filter.rs): an exactPruningStatisticsimpl backed by decoded dictionary values, plusis_fully_dictionary_encoded, which gates use of a chunk'sdictionary on its page encoding stats being exactly
PLAIN_DICTIONARY/RLE_DICTIONARY(a writer that fell back toPLAINpartway through no longer has a complete dictionary).RowGroupAccessPlanFilter::prune_by_dictionaryand aLoadDictionaries/PruneWithDictionariesstage in the Parquetopener's state machine, mirroring the existing bloom-filter stage.
row_groups_pruned_dictionarymetric,.sltcoverage(
parquet_dictionary_pruning.slt, plus the metric added to existingEXPLAIN ANALYZEgolden files), and a criterion benchmark comparingstatistics-only vs. bloom-filter vs. dictionary pruning.
This depends on a new arrow-rs API to decode a Parquet dictionary page: apache/arrow-rs#10420
(apache/arrow-rs#9010).
Temporary
[patch.crates-io]commit in PR exists to cover thatAre these changes tested?
Yes: unit tests in
dictionary_filter.rsmirroringbloom_filter.rs(absent value,
INlist absent, present value,!=sole value,NOT INlist, and a PLAIN-fallback chunk correctly not pruned), new.sltcoverage showingrow_groups_pruned_dictionaryinEXPLAIN ANALYZE, and a criterion benchmark.Are there any user-facing changes?
Yes: a new opt-in config option
dictionary_filter_on_read(defaultfalse, no behavior change unless enabled) and a newrow_groups_pruned_dictionarymetric inEXPLAIN ANALYZE.