Skip to content

perf(table): parallelize non-bulk PurgeFiles deletion - #1973

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parallelize-purge-file-deletion
Open

perf(table): parallelize non-bulk PurgeFiles deletion#1973
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parallelize-purge-file-deletion

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Parallelizes non-bulk PurgeFiles deletes with bounded workers.
  • Keeps BulkRemovableIO as the first choice.
  • Keeps missing-file deletes successful.
  • Collects independent failures in deterministic input order.
  • Stops queued work when the context is canceled.

Benchmark

Simulated 100µs per-file deletion:

  • 1,000 files: 120ms with 1 worker, 30ms with 4 workers, 7.7ms with 16 workers.
  • 10,000 files: 1.22s with 1 worker, 416ms with 4 workers, 76ms with 16 workers.

Tests

  • go test ./table -count=1
  • go test -race ./table -count=1
  • go vet ./...
  • golangci-lint run --timeout=10m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant