Skip to content

perf(table): elide identity partition filters - #1966

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/elide-identity-partition-filters
Open

perf(table): elide identity partition filters#1966
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/elide-identity-partition-filters

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What

  • Elide identity partition predicates from local scan task residuals.
  • Mixed filters keep only the remaining row predicate. Example: tenant_id = acme AND amount > 100 becomes amount > 100.
  • Use the task residual for Arrow projection, so AlwaysTrue residuals do not read filter-only columns.
  • Keep unknown partition values and transformed predicates conservative.
  • This PR intentionally supports identity transforms only.

Why

Local scans already use task residuals while reading, but local planning left them nil. This made files re-evaluate predicates already proven by identity partition values.

Benchmark

Planning overhead

Command: go test ./table -run '^$' -bench '^BenchmarkPartitionResidualPlanning$' -benchmem -count=5

Apple M1 Pro:

  • 0.866 to 0.897 ms/op for 4,096 files
  • 425,984 B/op
  • 16,384 allocs/op

This measures residual planning overhead only.

Arrow read path

Command: go test ./table -run '^$' -bench '^BenchmarkArrowScanTaskResidual$' -benchmem -count=5

Apple M1 Pro, 32,768-row Parquet file, projecting payload only:

  • Identity-only filter, original filter: about 2.75 ms/op, 8.98 MB/op, 946 allocs/op
  • Identity-only filter, AlwaysTrue residual: about 1.55 ms/op, 5.55 MB/op, 520 allocs/op
  • Mixed filter, original filter: about 3.52 ms/op, 19.86 MB/op, 1,833 allocs/op
  • Mixed filter, amount > 100 residual: about 3.02 ms/op, 17.33 MB/op, 1,488 allocs/op

The read benchmark compares the original filter path with the task residuals produced by this PR.

Checks

  • go test ./table -count=1
  • go test -race ./table -count=1
  • /Users/hoangvu/go/bin/golangci-lint run ./table/... --timeout=5m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant