Skip to content

fix(export): compress the public data archive at write time #851

Description

@bencap

Problem

export_public_data opens its archive as ZipFile(zip_file_name, "w") with no compression argument. Python defaults to ZIP_STORED, so every member is written raw and the archive on disk is approximately 30 GB.

The published artifact is roughly 2 GB because the ZIP is gzipped by hand before upload. Practically, this means the export cannot run on the worker as written since it does not have 30 GB of free space.

Proposed behavior

The archive is written compressed. The external gzip step is retired and the published artifact is a plain .zip.

Acceptance criteria

  • The archive is written with DEFLATE compression; a generated archive is under 3 GB for the current corpus.
  • Peak local disk use during a run stays under 4 GB.
  • Individual members can be read from the archive without decompressing the whole file.
  • deployment/docs/zenodo-release.md no longer references a gzip step, and its stated archive size matches what the script now produces.
  • src/mavedb/scripts/resources/CHANGELOG.md records the format change and names the extension change from .zip.gz to .zip.
  • The archive's member list and contents are unchanged from the previous release for equivalent database state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions