Skip to content

perf(rest): fetch remote scan plan tasks concurrently - #1959

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/fetch-remote-scan-plan-tasks-concurrently
Open

perf(rest): fetch remote scan plan tasks concurrently#1959
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/fetch-remote-scan-plan-tasks-concurrently

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Fetch each remote plan-task frontier with up to 8 concurrent REST requests.
  • Keep breadth-first response order by storing each response at its handle index.
  • Keep the existing once-only handle expansion and cycle protection.
  • Add ordering and concurrency tests plus a fake REST latency benchmark.

Benchmark

Command:

go test ./catalog/rest -run '^$' -bench '^BenchmarkCollectScanTasks64Handles$' -benchtime=1x -count=5

Apple M1 Pro, darwin/arm64, 64 plan-task handles, and 10 ms simulated server latency per handle:

Fetch mode Result
Serial baseline (limit 1) 733 to 777 ms/op
8 workers 90 to 93 ms/op

This is about an 8x reduction in the remote fetch portion. The 8-worker result is close to the expected 80 ms network floor, with HTTP and scheduling overhead.

Tests

  • go test ./...
  • go test -race ./catalog/rest -count=1
  • go vet ./catalog/rest
  • golangci-lint run --timeout=10m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant