Problem
export_public_data opens its archive as ZipFile(zip_file_name, "w") with no compression argument. Python defaults to ZIP_STORED, so every member is written raw and the archive on disk is approximately 30 GB.
The published artifact is roughly 2 GB because the ZIP is gzipped by hand before upload. Practically, this means the export cannot run on the worker as written since it does not have 30 GB of free space.
Proposed behavior
The archive is written compressed. The external gzip step is retired and the published artifact is a plain .zip.
Acceptance criteria
- The archive is written with DEFLATE compression; a generated archive is under 3 GB for the current corpus.
- Peak local disk use during a run stays under 4 GB.
- Individual members can be read from the archive without decompressing the whole file.
deployment/docs/zenodo-release.md no longer references a gzip step, and its stated archive size matches what the script now produces.
src/mavedb/scripts/resources/CHANGELOG.md records the format change and names the extension change from .zip.gz to .zip.
- The archive's member list and contents are unchanged from the previous release for equivalent database state.
Problem
export_public_dataopens its archive asZipFile(zip_file_name, "w")with nocompressionargument. Python defaults toZIP_STORED, so every member is written raw and the archive on disk is approximately 30 GB.The published artifact is roughly 2 GB because the ZIP is gzipped by hand before upload. Practically, this means the export cannot run on the worker as written since it does not have 30 GB of free space.
Proposed behavior
The archive is written compressed. The external gzip step is retired and the published artifact is a plain
.zip.Acceptance criteria
deployment/docs/zenodo-release.mdno longer references a gzip step, and its stated archive size matches what the script now produces.src/mavedb/scripts/resources/CHANGELOG.mdrecords the format change and names the extension change from.zip.gzto.zip.