Skip to content

perf(table): cache snapshot manifests across scans - #1970

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/cache-snapshot-manifests-across-scans
Open

perf(table): cache snapshot manifests across scans#1970
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/cache-snapshot-manifests-across-scans

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Cache decoded manifest lists per table and reuse them across fresh scans.
  • Share concurrent first reads so only one manifest-list read runs for a snapshot.
  • Keep data and delete manifest groups ready for planning paths that need one type.
  • Reset the cache on Refresh so a new metadata snapshot never reuses stale descriptors.
  • Apply the cache to local scans, incremental append scans, table inspection, and AllManifests.

Benchmark

Apple M1 Pro, 1 manifest, 10 repeated scans per benchmark iteration:

  • Manifest-list opens: 10 → 1
  • Time: 4.26 ms → 2.36 ms
  • Memory: 4.74 MB → 2.78 MB
  • Allocations: 37,048 → 23,519

With 10,000 manifests, manifest-list opens still drop from 10 to 1. Total planning time stays close because opening the 10,000 manifest files dominates that workload.

Checks

  • go test ./table/... -count=1
  • go test -race ./table/... -count=1
  • go test ./... -run ^$ -count=1
  • go vet ./...
  • golangci-lint run --timeout=10m

🔥

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant