Skip to content

Extract nested zip files regardless of conform file filter - #112

Merged
iandees merged 3 commits into
masterfrom
verify-issue-35-nested-zip
Aug 22, 2026
Merged

Extract nested zip files regardless of conform file filter#112
iandees merged 3 commits into
masterfrom
verify-issue-35-nested-zip

Conversation

@iandees

@iandees iandees commented Aug 22, 2026

Copy link
Copy Markdown
Member

Summary

Commit 82076c5 ("Recursively check zip files") added support for a source zip that contains a nested zip containing the actual data, which is the scenario in #35. That fix works when a source has no conform file tag, but breaks when one is set (a common case for disambiguating the target shapefile from sibling files in the zip).

The bug: ZipDecompressTask._extract_zip applies the filenames allow-list (derived from the conform file tag) to every entry in the zip being extracted, including at the outer/top level. A nested zip like nested.zip never matches that allow-list (which contains the shapefile's .shp/.shx/.dbf/.prj names, not the zip's own name), so it gets skipped entirely and the recursive-unzip logic that looks for nested zips never finds anything to recurse into. The requested file is silently never extracted.

Fix: entries ending in .zip are now always extracted, regardless of the filenames filter, since they may contain the requested file. The filter is re-applied normally when that nested zip is itself extracted, so irrelevant sibling files (docs, license, lookup tables) are still correctly skipped.

Test plan

  • Added TestZipDecompressTask in openaddr/tests/conform.py, covering:
    • a zip-of-a-zip-containing-a-shapefile with no file filter (already worked)
    • the same structure with a conform file filter naming the shapefile (previously failed, now passes)
    • multiple nested zips at the same level still raise DecompressionError
  • Ran the new tests and the full suite (python3 test.py) in the project's batch-machine-test:latest docker image: 120 tests, all passing.

Fixes #35

@iandees
iandees force-pushed the verify-issue-35-nested-zip branch from 23b9027 to 49f4e3f Compare August 22, 2026 12:54
@iandees

iandees commented Aug 22, 2026

Copy link
Copy Markdown
Member Author

Rebased onto master to resolve the conflict (both sides touched the same import block in openaddr/tests/conform.py; merged both).

Also added zip bomb guards while I was in here: this PR makes nested zips always extract regardless of the file filter, which removed what was previously (accidentally) acting as a partial size limiter. Added a per-entry declared-size cap (checked from zip metadata before extracting, so no need to decompress first) and a nesting-depth cap, both raising DecompressionError loudly. Covered by two new tests; full suite is at 73 passing.

@iandees

iandees commented Aug 22, 2026

Copy link
Copy Markdown
Member Author

Added one more optimization: decompress() now stops recursing into further nested zips as soon as every filename the file filter asked for has actually been extracted (tracked by exact name match, returned from _extract_zip). If the target file is already satisfied by the outer zip, an unrelated sibling nested zip is never opened at all - smaller attack surface, less wasted work.

This falls back to the original full-recursion behavior (safe default) whenever completeness can't be proven - no filter set (caller wants everything), or a filter entry that only matched via the directory-enclosure branch of is_in() rather than an exact filename. So the #35 fix (finding a filtered target file that's actually inside a nested zip) still works - covered by the existing test_single_nested_zip_is_extracted_with_file_filter test, still green.

Also dropped un-recursed .zip files from the final output file list, since a zip that was correctly left unopened isn't a usable source file.

New test: test_skips_nested_zip_once_filtered_file_is_already_found - outer zip contains the target shapefile directly plus an unrelated nested zip with a decoy file; asserts the decoy is never extracted. Full suite: 74 tests in unittest.conform, 123 in the full test.py runner, all green.

@iandees
iandees merged commit 6358ad5 into master Aug 22, 2026
1 check passed
@iandees
iandees deleted the verify-issue-35-nested-zip branch August 22, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support zipped shapefiles within source zip

1 participant