Skip to content

feat(pelican): add GET /pelican/read for inline object contents - #265

Merged
rbardaji merged 1 commit into
mainfrom
feature/262-pelican-read
Aug 30, 2026
Merged

feat(pelican): add GET /pelican/read for inline object contents#265
rbardaji merged 1 commit into
mainfrom
feature/262-pelican-read

Conversation

@rbardaji

Copy link
Copy Markdown
Collaborator

Part of #262. The subscribe half of that issue follows separately and will close it.

Why

The Pelican integration could list objects (/browse, /info) and hand back a file to save (/download), but there was no way to get an object's contents inline. A caller wanting to work with the data had to write it to disk and read it back. /pelican/read returns the contents in the response body, so the object referenced by an event can be piped straight into the caller's own code.

The route

GET /pelican/read?path=<path>&federation=<name>
{
  "success": true,
  "path": "/ospool/uc-shared/public/osg-training/testing/nested/actual_dir/test.txt",
  "size": 29,
  "encoding": "utf-8",
  "content": "Testing symlinks and Pelican\n"
}

Three decisions worth flagging for review:

Binary is encoded, not refused. Pelican namespaces hold binary payloads, so a body that is not valid UTF-8 comes back base64-encoded and the encoding field says which of the two it is, rather than the request failing.

The read is capped by PELICAN_MAX_READ_BYTES, 10 MiB by default. The contents travel in the response body, so an uncapped read would put an arbitrarily large object into the API's memory. The cap bounds the read itself — the service reads through an open handle and asks for one byte past the limit — so an oversized object is never pulled in just to be rejected. A missing, non-numeric or non-positive value falls back to the default, which matters because settings are declared with extra: "allow" and a typo would otherwise pass silently.

Failures are distinguishable from the status code: 404 when the object is not in the federation, 413 when it is past the cap, 502 when the federation cannot be reached.

Verification

Exercised against the live OSDF federation, not only against mocks:

Case Result
No token 401
29-byte text object 200, encoding: "utf-8", contents returned
364 MB .sif 413, message points at /download
Non-existent path 404, detail names the cache that answered

The 413 returned in seconds on the 364 MB object, which confirms the cap bounds the read rather than rejecting after the fact.

17 new cases in tests/test_pelican_read.py cover the service (text, binary, at the cap, past the cap, missing object, federation failure carrying the exception type, and that the read is capped while reading), the PELICAN_MAX_READ_BYTES resolution including the malformed-value fallbacks, and the route's status-code mapping plus its inherited authorization.

Local gate: black and flake8 clean, 1290 tests pass. The three failures in tests/repositories/test_catalog_settings.py and tests/test_publi_env.py predate this branch — they reproduce on a clean main and come from a local .env being picked up by the settings tests, which does not exist in CI.

Backwards compatibility

Purely additive. PELICAN_MAX_READ_BYTES is optional and documented in example.env and docs/configuration.md, so existing deployments need no change. The new route sits behind the authorization added in #261 and is only mounted when PELICAN_ENABLED is set.

The integration could list objects and hand back a file to save, but
there was no way to get an object's contents inline, so a caller had to
write to disk and read back just to work with the data. /pelican/read
returns the contents in the response body instead.

Text comes back as text and non-UTF-8 payloads are base64-encoded rather
than refused, with an encoding field saying which. The read is capped by
PELICAN_MAX_READ_BYTES and the cap bounds the read itself, so an
oversized object is never pulled into memory just to be rejected.

Part of #262.
@rbardaji
rbardaji merged commit 4d32923 into main Aug 30, 2026
2 checks passed
@rbardaji
rbardaji deleted the feature/262-pelican-read branch August 30, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant