Publish only multi-layer Docker images, stop pushing ORAS SIFs - #3
Merged
Conversation
Both `just build --push` and the CI workflow pushed the Docker image and then
`apptainer push`ed a SIF to the *same* repo and tag. A tag resolves to exactly
one manifest, so the ORAS push silently replaced the multi-layer Docker
manifest with a single-blob SIF artifact.
The effect is visible in the registry today:
rc-foundry:0.2.0-weights sylabs SIF config, 1 layer x 9.68 GB
rc-foundry:0.1.12-weights docker config, 11 layers x 9.30 GB (ORAS push
did not land here, so the Docker manifest survived)
A single 9.68 GB blob cannot be fetched in parallel and, more importantly,
cannot be resumed - a dropped connection restarts the whole transfer. Pulling
that image over a flaky link failed repeatedly with HTTP/2 PROTOCOL_ERROR and
then, over HTTP/1.1, `unexpected EOF`.
Changes:
- justfile: drop the ORAS push block and the two `apptainer registry login
oras://ghcr.io` calls in `push-all` and `login`. The local SIF build is kept,
since a local .sif is still useful for testing, but it is no longer published.
- CI: remove the Setup Apptainer / ORAS login / SIF build / ORAS push steps.
The SIF build goes too - nothing consumes it now, and it cost a multi-GB
build on every run.
Consumers should pull docker://ghcr.io/<org>/<container>:<version> and let
Apptainer or Singularity convert it locally.
Note for downstream: nf-binder-design currently references
oras://ghcr.io/.../rc-foundry:0.2.0-weights and will need to move to docker://
once that tag is next rebuilt.
Claude-Session: https://claude.ai/code/session_015xPe8vN1g2M5dZgpuv35z5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
just build --pushand the CI workflow both push the Docker image and thenapptainer pusha SIF to the same repo and tag. A tag resolves to exactly one manifest, so the ORAS push silently replaces the multi-layer Docker manifest with a single-blob SIF artifact.The effect is visible in the registry right now:
rc-foundry:0.2.0-weightsrc-foundry:0.1.12-weightsrc-foundry:0.1.11The multi-layer image is being built — it just gets clobbered.
A single 9.68 GB blob can't be fetched in parallel and, more importantly, can't be resumed: a dropped connection restarts the whole transfer. Pulling this image over a flaky link failed repeatedly, first with HTTP/2
PROTOCOL_ERROR, then (forcing HTTP/1.1 viaGODEBUG=http2client=0) withunexpected EOF. Nextflow's "increaseapptainer.pullTimeout" hint is a red herring — these are dropped connections, not timeouts.SIF has no layers by design:
apptainer buildflattens every Docker layer into one squashfs, and ORAS stores that file as a single blob. So anyapptainer pushproduces exactly one layer, regardless of the Dockerfile.Changes
apptainer registry login oras://ghcr.iocalls inpush-allandlogin. The local SIF build is kept, since a local.sifis still useful for testing; it's just no longer published.Verification
just --listparses;datestampandapptainer_imgare both still referenced.oras/apptainerreferences remain outside explanatory comments.Downstream note
nf-binder-designreferencesoras://ghcr.io/.../rc-foundry:0.2.0-weightsin its rfd3 modules. Once that tag is next rebuilt it will be a Docker manifest, and those modules need to move todocker://.A follow-up PR splits the rc-foundry weight downloads into multiple layers and adds
rfd3na. It's deliberately held until this merges, so the rebuild it triggers runs under the fixed workflow and publishes a proper multi-layer image.