Motivation
The C++ TsFile reader currently accesses local files through storage::ReadFile. On POSIX platforms, ReadFile::read() uses pread to read data at a specified offset. The C API and Python reader ultimately share this read path.
This proposal introduces mmap as an optional local-file read backend while retaining the existing pread implementation.
Proposed behavior
Support the following modes:
AUTO: prefer mmap for supported local files and fall back to pread if mapping is unavailable or fails.
MMAP: explicitly use memory-mapped I/O and return a clear error if it cannot be used.
PREAD: preserve the current behavior.
AUTO could be the default mode. The configuration should follow the existing C++ configuration style, for example:
common::set_file_read_backend(FileReadBackend::AUTO);
ReadFile can remain the common entry point and dispatch reads to the selected backend, avoiding changes to the existing metadata, chunk, and page read paths. Equivalent configuration should also be exposed through the C and Python APIs.
Potential benefits
- Reduce system-call overhead for frequent, fine-grained random reads.
- Efficiently locate metadata, chunks, and pages by file offset.
- Provide a foundation for future read-only views over mapped regions to reduce intermediate copies.
- On POSIX systems, close the original file descriptor after a successful mapping, reducing long-lived FD usage when many TsFiles are opened concurrently.
Both pread and mmap use the OS page cache, so the actual performance impact must be validated with benchmarks.
Considerations
mmap should only be used for supported local regular files; other file systems should keep their existing backends.
AUTO must safely fall back to pread.
- Mappings and platform-specific resources must be released when the reader is closed.
- Files must not be truncated while mapped, as this may cause errors such as
SIGBUS.
- Large files, virtual address space limits, empty files, and mapping failures must be handled correctly.
- Windows may use an equivalent mapping implementation or fall back to the existing read path.
- The initial implementation may preserve
ReadFile::read() and copy data from the mapped region. A zero-copy view API can be considered separately.
Acceptance criteria
- The C++ reader can correctly read TsFiles through the
mmap backend.
- Users can select
AUTO, MMAP, or PREAD.
- C and Python APIs can configure or inherit the same backend selection.
- The TsFile format and query results remain unchanged.
- Mapping failures and unsupported environments are handled correctly.
- All mappings, file descriptors, and related resources are released on close.
- Benchmarks compare
mmap and pread for sequential reads, random queries, metadata-intensive reads, and concurrent multi-file workloads.
Motivation
The C++ TsFile reader currently accesses local files through
storage::ReadFile. On POSIX platforms,ReadFile::read()usespreadto read data at a specified offset. The C API and Python reader ultimately share this read path.This proposal introduces
mmapas an optional local-file read backend while retaining the existingpreadimplementation.Proposed behavior
Support the following modes:
AUTO: prefermmapfor supported local files and fall back topreadif mapping is unavailable or fails.MMAP: explicitly use memory-mapped I/O and return a clear error if it cannot be used.PREAD: preserve the current behavior.AUTOcould be the default mode. The configuration should follow the existing C++ configuration style, for example:ReadFilecan remain the common entry point and dispatch reads to the selected backend, avoiding changes to the existing metadata, chunk, and page read paths. Equivalent configuration should also be exposed through the C and Python APIs.Potential benefits
Both
preadandmmapuse the OS page cache, so the actual performance impact must be validated with benchmarks.Considerations
mmapshould only be used for supported local regular files; other file systems should keep their existing backends.AUTOmust safely fall back topread.SIGBUS.ReadFile::read()and copy data from the mapped region. A zero-copy view API can be considered separately.Acceptance criteria
mmapbackend.AUTO,MMAP, orPREAD.mmapandpreadfor sequential reads, random queries, metadata-intensive reads, and concurrent multi-file workloads.