feat(fs): add lightweight object-store and S3 filesystem support

[mrdrivingduck]

Background

I started this work because I wanted duckdb-paimon to query Paimon data lakes stored on S3. That required adding S3 filesystem support to paimon-cpp first.

My first implementation used Aws::S3::S3Client. It worked, but the dependency cost was much larger than I expected. Reading a Paimon table only needs a fairly small part of S3: HEAD, paginated LIST, and ranged GET. Pulling in the complete AWS S3 client and its CRT dependency stack felt disproportionate for that job, and it also made the build integration considerably more complicated.

I then looked at how DuckDB's httpfs and Iceberg extensions divide this work. DuckDB keeps the HTTP data path generic and limits the storage-specific code to things such as credentials, request signing, URL construction, and error handling. That seemed like a better direction for paimon-cpp as well.

Based on that, I rewrote the original S3 implementation around a reusable object-store filesystem layer and a small HTTP client abstraction.

Design

The implementation is still being cleaned up, and some names and file boundaries may change before it is ready to merge. But this is the architectural split I expect to keep. I am sharing it now mainly to discuss whether this direction makes sense.

The implementation is organized as follows:

FileSystem
└── ObjectStoreFileSystem
    ├── S3FileSystem
    └── [OSSFileSystem] (WIP)

Shared components
├── ObjectStoreClient
├── HttpClient
├── ObjectStoreInputStream
└── Read-ahead

ObjectStoreFileSystem, ObjectStoreClient, and ObjectStoreInputStream implement the filesystem semantics and reading behavior shared by object stores. S3FileSystem and S3ObjectStoreClient only handle S3-specific URLs, API requests, credentials, signing, and errors.

Object data is transferred by the libcurl-based HttpClient. The AWS C libraries are used only for the credential chain and SigV4 signing, so the implementation does not depend on the complete AWS S3 SDK. All of these interfaces remain internal to paimon-cpp, and the existing public FileSystem interface is unchanged.

OSS experiment

To validate the abstraction against a second provider, I also built a WIP OSS filesystem using Alibaba Cloud OSS C++ SDK v2. It reuses the common filesystem semantics, input stream, range reads, read-ahead, pagination, and error handling; the OSS-specific code mainly adapts credentials, endpoints, requests, and responses.

The OSS implementation is not part of this PR, but it could later replace or coexist with the current Jindo-based path. Jindo is a closed-source binary dependency and is no longer recommended in a number of environments, which makes it less suitable as the project's long-term OSS foundation than the open-source OSS SDK.

Scope

The S3 filesystem in this PR is read-only. It implements the operations needed to read Paimon metadata, manifests, and data files. Write and mutation operations return an explicit unsupported error.

This PR does not add a general-purpose http:// filesystem, and it does not attempt to cover HDFS. HDFS has different filesystem semantics and should remain a separate implementation if it is added later.

Testing

Unit tests cover the shared object-store behavior, reads and read-ahead, error handling, and S3 configuration and signing. I also tested metadata, manifest, and ORC reads against a real S3 Paimon warehouse and verified the integration through duckdb-paimon.

[Copilot]

Pull request overview

This PR adds a new internal object-store filesystem layer and a read-only S3 filesystem implementation, enabling paimon-cpp to read Paimon warehouses stored on S3 with a lightweight HTTP data path (libcurl) and minimal AWS C libraries for credential discovery and SigV4 signing.

Changes:

• Introduce object_store shared components: ObjectStoreFileSystem (read-only semantics + read-ahead) and a libcurl-based HttpClient.
• Add S3FileSystem + factory and an S3ObjectStoreClient that implements HEAD/LIST/Range-GET and request signing via AWS C Auth.
• Extend build/CI to support an opt-in PAIMON_ENABLE_S3 option and build the minimal AWS auth dependency stack.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
third_party/versions.txt	Adds version pins/checksums for minimal AWS C auth dependency set (and s2n).
src/paimon/fs/s3/s3_file_system.h	Declares S3 filesystem options and construction helpers.
src/paimon/fs/s3/s3_file_system.cpp	Implements S3 object-store client (signing + HEAD/LIST/GET range) and S3 filesystem wrapper.
src/paimon/fs/s3/s3_file_system_test.cpp	Adds unit tests for S3 URL construction, signing, option validation, range reads, and list parsing.
src/paimon/fs/s3/s3_file_system_factory.h	Declares S3 filesystem factory for scheme registration.
src/paimon/fs/s3/s3_file_system_factory.cpp	Implements option validation and registers the S3 filesystem factory.
src/paimon/fs/s3/CMakeLists.txt	Builds the S3 filesystem library/tests when PAIMON_ENABLE_S3 is enabled.
src/paimon/fs/object_store/object_store_file_system.h	Defines object-store client interface, read-ahead limiter, and filesystem wrapper.
src/paimon/fs/object_store/object_store_file_system.cpp	Implements read-only object-store filesystem semantics and read-ahead input stream.
src/paimon/fs/object_store/object_store_file_system_test.cpp	Adds unit tests covering object-store semantics, pagination, and read-ahead behavior.
src/paimon/fs/object_store/http_client.h	Defines internal HTTP client interface and Curl implementation.
src/paimon/fs/object_store/http_client.cpp	Implements Curl-based HTTP execution with basic retries and header parsing.
src/paimon/fs/object_store/CMakeLists.txt	Builds the shared object-store components and tests behind PAIMON_ENABLE_S3.
CMakeLists.txt	Adds PAIMON_ENABLE_S3 option and wires S3/object_store subdirectories + CURL discovery.
cmake_modules/ThirdpartyToolchain.cmake	Allows _REVISION= entries and triggers AWS auth build when S3 is enabled.
cmake_modules/BuildAwsAuth.cmake	Adds ExternalProject build for minimal AWS C Auth dependency chain (and s2n on Linux).
cmake_modules/arrow.diff	Adjusts Arrow’s thrift build args (adds -DWITH_OPENSSL=OFF).
ci/scripts/build_paimon.sh	Enables PAIMON_ENABLE_S3=ON in CI build script.
.github/workflows/gcc8_test.yaml	Installs libcurl/OpenSSL dev packages needed for S3/AWS auth build.
.github/workflows/build_and_test.yaml	Installs libcurl/OpenSSL dev packages before building.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

[lucasfang]

Each path under src/paimon/fs should be an independent plugin. I suggest moving object_store to the framework side under src/paimon/common/fs/, so that future OSS FileSystem integrations can also reuse it.

[mrdrivingduck]

Each path under src/paimon/fs should be an independent plugin. I suggest moving object_store to the framework side under src/paimon/common/fs/, so that future OSS FileSystem integrations can also reuse it.

Agreed, done.

[lucasfang]

+1

[mrdrivingduck]

Please use int32_t instead of int

Might be good to have that in AGENTS.md 🤔

[lucasfang]

Please use int32_t instead of int

Might be good to have that in AGENTS.md 🤔

There is docs/code-style.md in AGENTS.md, seems that Agent miss that？

[mrdrivingduck]

There is docs/code-style.md in AGENTS.md, seems that Agent miss that？

Yes, I see it now. Never mind.