[python] Add Split.to_dict() to expose planned split contents

[XiaoHongbo-Hope]

Purpose

Add Split.to_dict() to the Python bindings so a non-Rust reader (e.g. pypaimon) can rebuild its own split from a Rust-planned split and read the files directly — planning runs in Rust (the serial, driver-side bottleneck), reading stays in the existing reader, no re-planning.

Brief change log

PySplit::to_dict() exposes a planned split as a plain dict — bucket / paths / partition plus per-file metadata (file_path, schema_id, first_row_id, write_cols, …) and deletion files. Planning-only stats omitted.

Tests

test_split_to_dict_exposes_fields, test_split_to_dict_partition_and_reads.

API and Format

Additive — new Split.to_dict() Python method. No storage-format change.

Documentation

Covered by the method docstring.

[JunRuiLee]

Thanks @XiaoHongbo-Hope! A question on necessity first.

The direction of the read effort (#413) is to let pypaimon run its DataFrame read on the Rust core — initially as a basic, opt-in path behind a config flag, not a wholesale replacement, so it can mature alongside the pure-Python path. In that model Rust both plans and reads: PR3 already exposes new_read().read(splits) returning Arrow from the Rust TableRead, and splits stay opaque on the Python side — a transport token, nothing more — so pypaimon never needs to look inside one.

to_dict() exposes the full internal split contents, which is only needed if something rebuilds the split and reads the files outside Rust — i.e. reading in Python, which is the opposite of the direction #413 is moving in.

So: what's the use case for exposing split internals? If it's for a Rust-plans / Python-reads path, I think we should align on that direction first — otherwise it risks pulling us away from the opt-in Rust read path we're building toward.

[XiaoHongbo-Hope]

Thanks @XiaoHongbo-Hope! A question on necessity first.

The direction of the read effort (#413) is to let pypaimon run its DataFrame read on the Rust core — initially as a basic, opt-in path behind a config flag, not a wholesale replacement, so it can mature alongside the pure-Python path. In that model Rust both plans and reads: PR3 already exposes new_read().read(splits) returning Arrow from the Rust TableRead, and splits stay opaque on the Python side — a transport token, nothing more — so pypaimon never needs to look inside one.

to_dict() exposes the full internal split contents, which is only needed if something rebuilds the split and reads the files outside Rust — i.e. reading in Python, which is the opposite of the direction #413 is moving in.

So: what's the use case for exposing split internals? If it's for a Rust-plans / Python-reads path, I think we should align on that direction first — otherwise it risks pulling us away from the opt-in Rust read path we're building toward.

Thanks, Junrui. Got your concern, will discuss with Jingsong and then back to you.

[XiaoHongbo-Hope]

@JunRuiLee Thanks Junrui. My view: the read side can be handled by adding more workers in production, but plan runs on the driver. And It's additive and opt-in — no storage-format change — and it doesn't block #413; once full-Rust read lands, this just becomes an alternate/interim path.

[JingsongLi]

Do you need to use Python? Should we directly let AI see how to design cross language calls? For example, designing a cross language binary format for Split?

[JingsongLi]

Let's create an unified protocol: apache/paimon#8482