Skip to content

perf(manifest): project scan columns during reads - #1972

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/project-scan-manifest-columns
Open

perf(manifest): project scan columns during reads#1972
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/project-scan-manifest-columns

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Projects manifest data-file reads to the fields needed for scan tasks.
  • Keeps metric maps only when row pruning or positional-delete matching needs them.
  • Drops transient metric maps before local Arrow tasks retain the files.
  • Keeps PlanFiles() on the existing full-metadata path for callers that inspect task stats.
  • Caches projected Avro schemas by writer schema and projection.

Benchmark

Command: go test . -run=^\$ -bench BenchmarkManifestEntryProjection -benchtime=1x -count=1

Apple M1 Pro, 10,000 entries per case:

Stats fields Full bytes Scan bytes Full allocs Scan allocs
10 32.6 MB 15.6 MB 443,928 65,797
100 260.2 MB 100.5 MB 2,257,763 77,930
1,000 2.93 GB 1.45 GB 20,512,582 332,865

Validation

  • go test . ./table -count=1
  • go test ./... -run=^\$ -count=1
  • go test -race ./table -run=TestOpenManifestWithProjectionDropsStatsAfterFiltering -count=1
  • go vet ./...
  • golangci-lint v2.12.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant