Skip to content

perf(table): skip manifests with no live entries - #1964

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/skip-empty-manifests
Open

perf(table): skip manifests with no live entries#1964
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/skip-empty-manifests

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Skip data and delete manifests when both added_files_count and existing_files_count are known to be zero.
  • Keep manifests with unknown counts for safety.
  • This avoids opening manifests that only contain deleted entries.
  • Added scanner regression coverage and a planning benchmark.

Benchmark

Command: go test -run '^$' -bench '^BenchmarkPlanFilesSkipsKnownEmptyManifests$' -benchtime=2s ./table

Apple M1 Pro, 256 manifests, 64 entries per manifest.

Case Before After
All empty 65.0 ms/op, 256 manifest opens/op 0.31 ms/op, 0 manifest opens/op
10% live 65.6 ms/op, 256 manifest opens/op 10.7 ms/op, 26 manifest opens/op

Allocations also dropped from 81.0 MB to 0.32 MB per operation for the all-empty case, and from 81.6 MB to 9.1 MB for the mixed case.

Checks

  • go test ./table -count=1
  • go test -race ./table -count=1
  • golangci-lint run --timeout=10m

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant