feat(table): add RewriteManifests clustering

[fallintoplace]

What changed

• add WithRewriteManifestClusterBy for opt-in clustering
• group live data entries by cluster key + partition spec
• roll each key at the configured manifest target size
• keep the default size-only path unchanged
• clean open output manifests when a callback or writer fails

Why

• The Java RewriteManifests.clusterBy API uses the same idea: keep files with the same key together.
• This is useful when the key matches a partition value used by common filters.
• The clustered layout can let scan planning reject unrelated manifests before opening their entries.
• The option is opt-in, so existing rewrites keep the current behavior.

Benchmark

Rewrite throughput was roughly neutral in the benchmark. The important result is the read-side benchmark. It exercises Scan.filterManifestsWithSchema, the manifest-pruning stage used by local PlanFiles.

Command: go test ./table -run=^$ -bench=^BenchmarkManifestPruningModes$ -benchmem -benchtime=1s -count=3

Workload: 512 one-entry input manifests, 32 partition values, and an equality filter on id = 7.

• size-only: 18.6–18.8 µs/op, 17,728 B/op, 580 allocs/op, 64/64 manifests selected
• cluster-by: 10.8–13.5 µs/op, 11,456 B/op, 324 allocs/op, 1/32 manifests selected

That is about 40% lower pruning time, 35% fewer bytes, and 44% fewer allocations for this partition-aligned workload.

Java reference: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/RewriteManifests.java

Tests

• go test ./table/...
• go vet ./table/...
• go test -race ./table -run 'TestRewriteManifests(ClusterBy|CleansOrphansOnInvalidClusterKey)' -count=1

[zeroshade]

The clustering and cleanup paths otherwise look good, and the focused race tests, full table subtree, vet, benchmarks, CI, and synthetic merge all passed. One public-API panic remains in cluster-key validation.

[zeroshade]

[P1] Reject cluster keys that cannot safely round-trip through the writer map

reflect.Type.Comparable() only checks the static type. For example, struct{ Value any }{Value: []int{1}} passes this check and then writers[key] panics with hash of unhashable type: []int. A math.NaN() key also passes, but because it is not equal to itself the inserted writer cannot be retrieved during finalization, causing a nil-pointer panic at line 205. I reproduced both through the exported Transaction.RewriteManifests API; NaN is a realistic value when clustering by an identity float partition. Please validate the value itself (for example, reflect.Value.Comparable() plus a reflexivity check) and add public-API regressions for both cases.