Skip to content

perf(table): prune equality deletes by data-file metrics - #1960

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/equality-delete-metrics-pruning
Open

perf(table): prune equality deletes by data-file metrics#1960
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/equality-delete-metrics-pruning

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Port the equality-delete metrics pruning used by Java DeleteFileIndex.
  • After partition and sequence filtering, use equality-field null counts and lower/upper bounds from the data file and delete file.
  • Keep the match when stats are missing, malformed, nested, or otherwise uncertain.
  • Cache typed bounds so the hot loop avoids repeated literal/interface work. ⚡

Java reference: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/DeleteFileIndex.java

Why

With many equality deletes in one partition, most delete files can be ruled out from a data file by its metrics. This reduces the delete files attached to each FileScanTask, which also reduces downstream delete-file reads.

Benchmark

Workload: 10,000 data files, 10,000 equality deletes, one partition, 100 disjoint key-range groups. Each data file overlaps 100 delete files (1%).

                                      ns/op       B/op       attached deletes/data file
baseline                              1.41-1.69s  6.66GB     10,000
metrics pruning                       0.93-1.11s  52.5MB     100

The benchmark was run with go test ./table -run ^$ -bench ^BenchmarkEqualityDeleteIndexMetrics$ -benchtime=1x -count=5 on an Apple M1 Pro.

Checks

  • go test ./...
  • go test -race ./table
  • go vet ./...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant