Skip to content

perf(table): load deletion-vector Puffin groups lazily - #1968

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/lazy-deletion-vector-puffin-groups
Open

perf(table): load deletion-vector Puffin groups lazily#1968
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/lazy-deletion-vector-puffin-groups

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Index deletion-vector ownership without opening Puffin files during scan setup.
  • Load each Puffin group on first use and cache the decoded bitmaps with single-flight behavior.
  • Keep the existing same-file grouping, range coalescing, and duplicate-DV validation.
  • Start scan workers when the iterator is consumed, so an unread iterator does not load DV files.

Benchmark

Apple M1 Pro. 100 Puffin groups with 10 DV blobs per group.

case Puffin opens/op ns/op B/op
eager all groups 100 2.75 ms 3.41 MB
lazy unread 0 0.23 ms 0.33 MB
lazy first group 1 0.30 ms 0.36 MB
lazy ten groups 10 0.62 ms 0.64 MB
lazy full scan 100 2.71 ms 3.40 MB

Tests

  • go test ./table/... -count=1
  • go test -race ./table/... -count=1
  • go vet ./...
  • golangci-lint run --timeout=10m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant