Skip to content

Fix PyPy snappy decompression BufferError in CI - #128

Merged
twiggler merged 3 commits into
mainfrom
copilot/fix-ci-test-ubuntu-latest-pypy3-11
May 21, 2026
Merged

Fix PyPy snappy decompression BufferError in CI#128
twiggler merged 3 commits into
mainfrom
copilot/fix-ci-test-ubuntu-latest-pypy3-11

Conversation

Copilot AI commented May 20, 2026

Copy link
Copy Markdown
Contributor

(found by monorepo ci)

Summary

  • investigate failing GitHub Actions job ci / test (ubuntu-latest, pypy3.11) (run 26149297732, job 76912461018)
  • fix dissect.util.compression.snappy.decompress by releasing the temporary BytesIO buffer view before writing back to the same stream
  • preserve existing decompression logic while avoiding PyPy's BufferError: Existing exports of data: object cannot be re-sized

Root cause

PyPy keeps exported BytesIO buffers alive longer than CPython in this code path. The previous implementation read copy data via dst.getbuffer()[...].tobytes() and then resized dst with dst.write(...) in the same loop. On PyPy 7.3.22 this triggers BufferError because the buffer export is still active when resizing. The offending PR is https://github.com/pypy/pypy/pull/5407/changes.


dst_offset = dst.tell() - offset
buf = dst.getbuffer()[dst_offset : dst_offset + length].tobytes()
dst_view = dst.getbuffer()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot why was this not a problem before? Did something change in pypy 3.11?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question — I checked the logs, and the same repo SHA (4d86c1b) passed earlier on PyPy 7.3.21 but failed later on PyPy 7.3.22 (still Python 3.11.15). So this path was relying on behavior that happened to work before: the temporary BytesIO export from getbuffer() was dropped soon enough. In newer PyPy it stays alive across the subsequent dst.write(...) resize check, which raises BufferError. Commit b09ffce fixes that by explicitly releasing the buffer view before writing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot I found this PR which is likely the root cause: https://github.com/pypy/pypy/pull/5407/changes

Please confirm and explain

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — that PR is very likely the root cause. In our CI, the failing run switched from PyPy 7.3.21 to 7.3.22, and your linked PR adds stricter active-export tracking (_exports) plus _check_exports(...) on bytearray-resizing/mutating paths, raising BufferError while a view is still exported. Our code was taking dst.getbuffer()[...].tobytes() and then immediately doing dst.write(...); with the stricter export checks, that now reliably raises until the view is released. That is exactly why b09ffce fixes it by releasing the buffer view before writing.

Copilot stopped work on behalf of twiggler due to an error May 20, 2026 09:16
Copilot AI requested a review from twiggler May 20, 2026 09:16
@codecov

codecov Bot commented May 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (dde4491) to head (a6b4532).

Files with missing lines Patch % Lines
dissect/util/compression/snappy.py 0.00% 4 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##            main    #128   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         29      29           
  Lines       2103    2106    +3     
=====================================
- Misses      2103    2106    +3     
Flag Coverage Δ
unittests 0.00% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented May 20, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 33 untouched benchmarks


Comparing copilot/fix-ci-test-ubuntu-latest-pypy3-11 (a6b4532) with main (dde4491)

Open in CodSpeed

@twiggler
twiggler marked this pull request as ready for review May 20, 2026 11:34
@twiggler
twiggler requested a review from Miauwkeru May 20, 2026 11:34

dst_offset = dst.tell() - offset
buf = dst.getbuffer()[dst_offset : dst_offset + length].tobytes()
dst_view = dst.getbuffer()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment to the new try/finally to explain why this was added, and perhaps a reference to this PR or PyPy?

@twiggler twiggler May 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a reference to the python documentation,
where is is explicitly stated that

"Many objects take special actions when a view is held on them (for example, a bytearray would temporarily forbid resizing)"

https://docs.python.org/3/library/stdtypes.html#memoryview:~:text=release

So I think calling release is not a workaround around some PyPy quirk, but necessary in general; we were just "lucky" before.

@twiggler
twiggler force-pushed the copilot/fix-ci-test-ubuntu-latest-pypy3-11 branch from f13bfce to 32020fe Compare May 21, 2026 09:58
@twiggler
twiggler force-pushed the copilot/fix-ci-test-ubuntu-latest-pypy3-11 branch from 32020fe to a6b4532 Compare May 21, 2026 14:53
@twiggler
twiggler merged commit fd59c1c into main May 21, 2026
33 of 34 checks passed
@twiggler
twiggler deleted the copilot/fix-ci-test-ubuntu-latest-pypy3-11 branch May 21, 2026 16:07
twiggler pushed a commit to twiggler/dissect-monorepo-test that referenced this pull request May 26, 2026
twiggler pushed a commit to twiggler/dissect-monorepo-test that referenced this pull request Jun 4, 2026
twiggler pushed a commit to twiggler/dissect-monorepo-test that referenced this pull request Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants