Extract nested zip files regardless of conform file filter - #112
Conversation
23b9027 to
49f4e3f
Compare
|
Rebased onto master to resolve the conflict (both sides touched the same import block in Also added zip bomb guards while I was in here: this PR makes nested zips always extract regardless of the file filter, which removed what was previously (accidentally) acting as a partial size limiter. Added a per-entry declared-size cap (checked from zip metadata before extracting, so no need to decompress first) and a nesting-depth cap, both raising |
|
Added one more optimization: This falls back to the original full-recursion behavior (safe default) whenever completeness can't be proven - no filter set (caller wants everything), or a filter entry that only matched via the directory-enclosure branch of Also dropped un-recursed New test: |
Summary
Commit 82076c5 ("Recursively check zip files") added support for a source zip that contains a nested zip containing the actual data, which is the scenario in #35. That fix works when a source has no conform
filetag, but breaks when one is set (a common case for disambiguating the target shapefile from sibling files in the zip).The bug:
ZipDecompressTask._extract_zipapplies thefilenamesallow-list (derived from the conformfiletag) to every entry in the zip being extracted, including at the outer/top level. A nested zip likenested.zipnever matches that allow-list (which contains the shapefile's.shp/.shx/.dbf/.prjnames, not the zip's own name), so it gets skipped entirely and the recursive-unzip logic that looks for nested zips never finds anything to recurse into. The requested file is silently never extracted.Fix: entries ending in
.zipare now always extracted, regardless of the filenames filter, since they may contain the requested file. The filter is re-applied normally when that nested zip is itself extracted, so irrelevant sibling files (docs, license, lookup tables) are still correctly skipped.Test plan
TestZipDecompressTaskinopenaddr/tests/conform.py, covering:filefilter naming the shapefile (previously failed, now passes)DecompressionErrorpython3 test.py) in the project'sbatch-machine-test:latestdocker image: 120 tests, all passing.Fixes #35