Skip to content

perf(table): compute partition residuals for local scan tasks - #1971

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/local-scan-partition-residuals
Open

perf(table): compute partition residuals for local scan tasks#1971
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/local-scan-partition-residuals

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What

  • Compute partition residuals during local planning.
  • Cache one evaluator per partition spec and reuse it for every data file in that spec.
  • Use strict and inclusive projections to return AlwaysTrue, AlwaysFalse, or the remaining row filter for each task.
  • Cover identity, time, truncate, bucket, null, set, boolean, and transformed filter predicates. Unsupported transforms stay conservative.
  • Read only the fields needed by each task residual, so a fully satisfied partition filter does not add filter-only columns to the file read.

Example

For ts >= ... AND ts < ... on day(ts) partitions:

  • Interior partitions get AlwaysTrue.
  • Boundary partitions keep only the needed timestamp predicate.
  • Partitions outside the range get AlwaysFalse.

Benchmark

4,096 files on an Apple M1 Pro:

  • Identity residual evaluation: about 0.67 ms/op, 360 KB/op, 16,384 allocs/op
  • Day-range residual evaluation: about 0.98 ms/op, 449 KB/op, 25,396 allocs/op

Checks

  • go test ./table/... -count=1
  • go test -race ./table -count=1
  • go test ./... -run=^$ -count=1
  • go vet ./...
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run --timeout=10m

🔥

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant