Fix PyPy snappy decompression BufferError in CI - #128
Conversation
|
|
||
| dst_offset = dst.tell() - offset | ||
| buf = dst.getbuffer()[dst_offset : dst_offset + length].tobytes() | ||
| dst_view = dst.getbuffer() |
There was a problem hiding this comment.
@copilot why was this not a problem before? Did something change in pypy 3.11?
There was a problem hiding this comment.
Good question — I checked the logs, and the same repo SHA (4d86c1b) passed earlier on PyPy 7.3.21 but failed later on PyPy 7.3.22 (still Python 3.11.15). So this path was relying on behavior that happened to work before: the temporary BytesIO export from getbuffer() was dropped soon enough. In newer PyPy it stays alive across the subsequent dst.write(...) resize check, which raises BufferError. Commit b09ffce fixes that by explicitly releasing the buffer view before writing.
There was a problem hiding this comment.
@copilot I found this PR which is likely the root cause: https://github.com/pypy/pypy/pull/5407/changes
Please confirm and explain
There was a problem hiding this comment.
Confirmed — that PR is very likely the root cause. In our CI, the failing run switched from PyPy 7.3.21 to 7.3.22, and your linked PR adds stricter active-export tracking (_exports) plus _check_exports(...) on bytearray-resizing/mutating paths, raising BufferError while a view is still exported. Our code was taking dst.getbuffer()[...].tobytes() and then immediately doing dst.write(...); with the stricter export checks, that now reliably raises until the view is released. That is exactly why b09ffce fixes it by releasing the buffer view before writing.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #128 +/- ##
=====================================
Coverage 0.00% 0.00%
=====================================
Files 29 29
Lines 2103 2106 +3
=====================================
- Misses 2103 2106 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
|
||
| dst_offset = dst.tell() - offset | ||
| buf = dst.getbuffer()[dst_offset : dst_offset + length].tobytes() | ||
| dst_view = dst.getbuffer() |
There was a problem hiding this comment.
Maybe add a comment to the new try/finally to explain why this was added, and perhaps a reference to this PR or PyPy?
There was a problem hiding this comment.
I will add a reference to the python documentation,
where is is explicitly stated that
"Many objects take special actions when a view is held on them (for example, a bytearray would temporarily forbid resizing)"
https://docs.python.org/3/library/stdtypes.html#memoryview:~:text=release
So I think calling release is not a workaround around some PyPy quirk, but necessary in general; we were just "lucky" before.
f13bfce to
32020fe
Compare
Agent-Logs-Url: https://github.com/fox-it/dissect.util/sessions/d2b460a0-c74f-4589-a8cc-fe9bf778cbc3 Co-authored-by: twiggler <12800443+twiggler@users.noreply.github.com>
Agent-Logs-Url: https://github.com/fox-it/dissect.util/sessions/d2b460a0-c74f-4589-a8cc-fe9bf778cbc3 Co-authored-by: twiggler <12800443+twiggler@users.noreply.github.com>
32020fe to
a6b4532
Compare
(found by monorepo ci)
Summary
ci / test (ubuntu-latest, pypy3.11)(run26149297732, job76912461018)dissect.util.compression.snappy.decompressby releasing the temporaryBytesIObuffer view before writing back to the same streamBufferError: Existing exports of data: object cannot be re-sizedRoot cause
PyPy keeps exported
BytesIObuffers alive longer than CPython in this code path. The previous implementation read copy data viadst.getbuffer()[...].tobytes()and then resizeddstwithdst.write(...)in the same loop. On PyPy 7.3.22 this triggersBufferErrorbecause the buffer export is still active when resizing. The offending PR is https://github.com/pypy/pypy/pull/5407/changes.