zenodo-dl

Download publications and metadata from Zenodo for offline archives.

For each publication version, zenodo-dl saves:

• Every file in the record, each checked against the MD5 checksum Zenodo publishes for it
• Zenodo's own record (zenodo.json), verbatim
• Four sidecar metadata files: metadata.md, metadata.yaml, metadata.json, metadata.bib

Any version can be archived, not just the latest.

Only records whose resource type is publication (articles, preprints, reports, theses, and so on) are archived. Zenodo also hosts datasets and software, often many gigabytes, and zenodo-dl refuses those. Only open-access records are archived. Restricted, embargoed, and closed records have no downloadable files.

Zenodo's terms

From Zenodo's terms of use: access to Zenodo's content is open to all, for non-military purposes only. Users of content shall respect applicable license conditions (zenodo-dl records each record's license). Zenodo metadata may be reused under CC0 unless specified otherwise. Download and use of content does not transfer any intellectual property rights.

Zenodo limits guest API use to 30 search requests a minute. zenodo-dl's default of one request every 3 seconds stays under that.

Installation

gem install zenodo-dl

CLI usage

zenodo-dl <ZENODO_ID_OR_URL> [<ZENODO_ID_OR_URL>...]

Accepted input forms:

Form	Example
Record ID	884117
DOI	10.5281/zenodo.884117, doi:10.5281/zenodo.884117
DOI URL	https://doi.org/10.5281/zenodo.884117
Record URL	https://zenodo.org/records/884117, https://zenodo.org/record/884117
File URL	https://zenodo.org/records/884117/files/<name>
API URL	https://zenodo.org/api/records/884117

A version's record ID archives that version. A concept record ID (the ID for all versions) archives the latest version.

Flags

Flag	Description
-i FILE, --input FILE	Read IDs/URLs from FILE, one per line (- for stdin, blanks and # skipped)
-p PATH, --path PATH	Root download directory
--rate-limit SECONDS	Seconds between HTTP requests (0 disables throttling)
-v, --verbose	Print step lines and per-request URL/byte logs to stdout
-q, --quiet	Print nothing to stdout. Errors still go to stderr.
--version	Print the gem version and exit
-h, --help	Print help and exit

-v and -q are mutually exclusive.

Environment variables

Variable	Effect
ZENODO_DOWNLOAD_PATH	Root download directory (default: $HOME/Downloads/Zenodo_Papers)
ZENODO_RATE_LIMIT	Seconds between HTTP requests (default: 3, 0 disables)

Precedence: CLI flag, then ENV var, then default.

Errors and exit status

A target that fails (unrecognized ID, no such record, not a publication, not open access, checksum mismatch, HTTP error, network failure) is reported on stderr as <target>: <message>, and the remaining targets still download. Exit status is 0 when every target succeeds and 1 when any fails.

Output layout

$ZENODO_DOWNLOAD_PATH/                  # default: $HOME/Downloads/Zenodo_Papers
  YYYY/MM/DD/<type>/<concept-id>-<slug>/
    <file>, <file>, ...                 # every file in the record
    zenodo.json                         # Zenodo's record, verbatim
    metadata.md                         # YAML frontmatter + Markdown body
    metadata.yaml
    metadata.json
    metadata.bib                        # synthesized @misc with the DOI

YYYY/MM/DD is the publication date. <type> is the publication type (preprint, article, report, and so on). The folder is keyed on the concept ID, so every version of a publication lands in the same folder. <slug> is derived from the title.

A publication with only v1 archived is kept flat, as above. When it has more than one version, each version gets its own v<N>/ folder with the same contents. Archiving a second version of a flat publication first moves the existing files into v<N>/. A publication whose latest version is v2 or later starts out in v<N>/ folders.

Each version downloads into a sibling .partial folder and is renamed into place only when every file matched its checksum. Re-running skips versions already archived.

Library usage

require 'zenodo/downloader'

identifier = Zenodo::Downloader::Identifier.new 'https://doi.org/10.5281/zenodo.884117'
client     = Zenodo::Downloader::Client.new               # 3-second rate limit by default
path       = Zenodo::Downloader::Archive.new(identifier, root: '/tmp/papers', client: client).run

Development

script/setup    # install dependencies
script/test     # run specs and rubocop
script/console  # interactive prompt

Specs run offline against recorded fixtures in spec/fixtures/http/. To check the record fixtures against the live Zenodo API, run:

ZENODO_LIVE=1 script/test

License

MIT. See LICENSE.md.

Code of Conduct

This project follows the Contributor Covenant 3.0. See CODE_OF_CONDUCT.md.