reducing I/O by using buffers and memoryviews

[armijnhemel]

This is something that I haven't fully thought out yet, but which might be useful in certain cases, such as when io is shared between elements (so when no new substreams are created). There are mechanisms in Python to reduce I/O and prevent unnecessary copying of data by using buffers and memoryviews. An article explaining it can be found here:

https://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews

[KOLANICH]

kaitai-io/kaitai_struct#277
kaitai-io/kaitai_struct#65

are related. Mmaps are memory views.

[armijnhemel]

kaitai-io/kaitai_struct#277 kaitai-io/kaitai_struct#65

are related. Mmaps are memory views.

Not according to the documentation:

"Create a memoryview that references object. object must support the buffer protocol. Built-in objects that support the buffer protocol include bytes and bytearray."

https://docs.python.org/3/library/stdtypes.html#memoryview

As far as I can tell mmap does not support the buffer protocol.

[KOLANICH]

@armijnhemel, my bad, you are right. To work with mmaps as with memory views we really have to wrap it into it. Thanks for fixing me.

[KOLANICH]

https://github.com/althonos/iocursor BTW.

[generalmimon]

I investigated how zero-copy substreams as proposed in kaitai-io/kaitai_struct#44 could be implemented in Python. I believe the API shown in kaitai-io/kaitai_struct#44 (comment) is solid and can be used regardless of the implementation details:

diff --git 1/buffered_struct-original.py 2/buffered_struct.py
index 37927c9..2bd3048 100644
--- 1/buffered_struct-original.py
+++ 2/buffered_struct.py
@@ -17,9 +17,8 @@ class BufferedStruct(KaitaiStruct):

     def _read(self):
         self.len_block = self._io.read_u4le()
-        self._raw_block = self._io.read_bytes(self.len_block)
-        _io__raw_block = KaitaiStream(BytesIO(self._raw_block))
-        self.block = BufferedStruct.Block(_io__raw_block, self, self._root)
+        _io_block = self._io.substream(self.len_block)
+        self.block = BufferedStruct.Block(_io_block, self, self._root)


     def _fetch_instances(self):

Of course, the tricky part is how to implement substream() so that memory copying is avoided or at least minimized.

First, we need to find out exactly where the copying is happening in order to understand the situation. Let's analyze the currently generated Python code, assuming that self._io is a KaitaiStream object wrapping a "true" file object, i.e. backed by an actual on-disk file, socket, pipe, etc. By default, the type of such a file object seems to be io.BufferedReader (tested in Python 3.14.2 by looking at the return type of open(..., 'rb')):

    def _read(self):
        self.len_block = self._io.read_u4le()
        self._raw_block = self._io.read_bytes(self.len_block)
        _io__raw_block = KaitaiStream(BytesIO(self._raw_block))
        self.block = BufferedStruct.Block(_io__raw_block, self, self._root)

If we only focus on what is happening in userspace (let's ignore the copy from the OS kernel to userspace), the first copy is triggered by the read_bytes() method. Since io.BufferedIOBase.read() produces bytes objects, data is copied from io.BufferedReader's internal buffer to the newly created bytes object. I don't see how this copy could be avoided in principle. Next, the BytesIO(self._raw_block) part at first glance looks like another candidate to incur a copy, but I found this comment that suggests this is not the case (emphasis mine):

In BytesIO it works in two directions – when you create a BytesIO from a bytes object it does not copy the data, but keep a reference for an existing bytes object, and getvalue() returns the internal bytes object (truncating it if possible).

I tested this and indeed, the BytesIO() constructor does not copy the input bytes object and just keeps a reference to it. This reference is also available via getvalue(), so you can check that it's an identical bytes object to the original:

Python 3.14.2 (tags/v3.14.2:df79316, Dec  5 2025, 17:18:21) [MSC v.1944 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from io import BytesIO
>>> x = b'hello'
>>> hex(id(x))
'0x23bbbb9ddd0'
>>> bio = BytesIO(x)
>>> hex(id(bio.getvalue()))
'0x23bbbb9ddd0'
>>> bio.getvalue() is x
True

(Note that this optimization doesn't seem to be officially documented, so it might be a CPython implementation detail.)

To summarize, the only copying that happens in this scenario is to create the bytes object, which is unavoidable as long as we're using io.BufferedReader (I guess memory-mapped files using mmap could eliminate this copying, but that's a completely different issue with both benefits and drawbacks: kaitai-io/kaitai_struct#277). Therefore, if the resulting substream remains in the form KaitaiStream(BytesIO(...)), it cannot be optimized in terms of the number of copies, since BytesIO inherently requires the substream bytes to be in memory, and the only way how they can transfer from io.BufferedReader to BytesIO is by copying.

Nevertheless, it might not be optimal to load all the underlying bytes of the substream from the file into memory eagerly in advance. Perhaps the file is large (hundreds of MiB or several GiB) and the substream occupies a large part of the file, but the format actually parses only a small part of the substream and the rest is ignored (obviously, what is ignored did not need to be read from the file at all). Or even if the entire substream were ultimately parsed down to the last byte into a huge number of small primitive values (e.g. numbers), it still seems that loading all the bytes of the substream into memory was unnecessary. We're considering buffered I/O here, which already takes care of buffering for better performance. This is actually just another buffer on top of that, which is unnecessary and probably only reduces performance and wastes memory.

Instead of loading the substream eagerly into memory, we could use the same approach as our Ruby implementation of zero-copy substreams does. Our zero-copy substreams in Ruby are virtual: they are represented by the SubIO class, which stores a reference to the parent I/O stream, the start offset of the substream and its size. All reads are delegated to the parent I/O stream, and SubIO acts as its proxy - it translates the relative position in the substream to the corresponding position in the parent I/O stream, reads from the parent I/O stream and then restores its state, and of course enforces the substream boundaries. It should be noted that this requires the parent I/O stream to support seeking: if it doesn't, then we should just fall back to the traditional read_bytes() + KaitaiStream(BytesIO(...)) method (the current Ruby implementation doesn't do that yet, but it should).

For file-based streams, I believe this is probably the best way to implement zero-copy substreams in Python. For in-memory streams (i.e. KaitaiStream object backed by a BytesIO), we also possibly have another option using memoryview, as mentioned earlier in this issue. However, although it's easy to obtain the memoryview object (just call the BytesIO.getbuffer() method and slice it as needed), the question is how to create a KaitaiStream based on it. KaitaiStream expects a file-like object, and Python doesn't seem to offer any built-in way to convert memoryview to it. Obviously, we could solve this by implementing a MemoryViewIO class that serves as an adapter for memoryview to look like a file object. However, that would largely duplicate the general SubIO class and would only be less flexible: SubIO would accept any file-like object as the parent stream, whereas MemoryViewIO would only accept memoryview. And we cannot create memoryview from a file object (except by using mmap), because it doesn't implement the buffer protocol. So I don't think memoryview can really help us with zero-copy substreams.

I guess memoryview might be useful for other things. For example, it might make sense to implement a read_bytes() variant that returns a memoryview object instead of bytes (which is totally reasonable when reading from a BytesIO-backed stream, but it's not a drop-in replacement because memoryview supports only a very limited subset of the operations supported by bytes, see https://pythonspeed.com/articles/bytesio-reduce-memory-usage/#some-limitations-of-memoryview).