Skip to content

perf(table): reuse partition path formatting plan - #1958

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/reuse-partition-path-formatting-plan
Open

perf(table): reuse partition path formatting plan#1958
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/reuse-partition-path-formatting-plan

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Why

PartitionToPath runs for every partition path written by the fanout and clustered writers. Each call rebuilt the same schema-dependent formatting state:

  • resolved partition field types
  • transform result types
  • escaped field names
  • output buffer estimate

Only the partition values change between calls.

What changed

  • Reuse the formatting state from the existing partition extraction plan.
  • Use the plan in fanout and clustered writers.
  • Keep value formatting, URL escaping, and public PartitionToPath behavior unchanged.
  • Add parity coverage for escaped names, timestamps, dropped source fields, and unknown transforms.

There are no public API changes.

Benchmark

Apple M1 Pro, 64 partition fields:

Path Time Memory Allocs
PartitionToPath 7.3 us/op 9.62 KB/op 67/op
Reused plan 2.8 us/op 3.10 KB/op 66/op

Tests

  • go test -count=1 -skip '^TestCreateAzureBucketManagedIdentityCredentialCalled$' ./...
  • go test -race -count=1 . ./table
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.8.0 run --timeout=10m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant