Skip to content

Publish only multi-layer Docker images, stop pushing ORAS SIFs - #3

Merged
pansapiens merged 1 commit into
mainfrom
fix/no-oras-push-publish-docker-only
Sep 9, 2026
Merged

Publish only multi-layer Docker images, stop pushing ORAS SIFs#3
pansapiens merged 1 commit into
mainfrom
fix/no-oras-push-publish-docker-only

Conversation

@pansapiens

@pansapiens pansapiens commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

just build --push and the CI workflow both push the Docker image and then apptainer push a SIF to the same repo and tag. A tag resolves to exactly one manifest, so the ORAS push silently replaces the multi-layer Docker manifest with a single-blob SIF artifact.

The effect is visible in the registry right now:

tag config layers
rc-foundry:0.2.0-weights sylabs SIF 1 × 9.68 GB — ORAS push won
rc-foundry:0.1.12-weights docker 11 layers, 9.30 GB — ORAS push didn't land, Docker manifest survived
rc-foundry:0.1.11 OCI index (2 manifests) multi-platform, so the justfile skips the apptainer build

The multi-layer image is being built — it just gets clobbered.

A single 9.68 GB blob can't be fetched in parallel and, more importantly, can't be resumed: a dropped connection restarts the whole transfer. Pulling this image over a flaky link failed repeatedly, first with HTTP/2 PROTOCOL_ERROR, then (forcing HTTP/1.1 via GODEBUG=http2client=0) with unexpected EOF. Nextflow's "increase apptainer.pullTimeout" hint is a red herring — these are dropped connections, not timeouts.

SIF has no layers by design: apptainer build flattens every Docker layer into one squashfs, and ORAS stores that file as a single blob. So any apptainer push produces exactly one layer, regardless of the Dockerfile.

Changes

  • justfile — drop the ORAS push block and the two now-pointless apptainer registry login oras://ghcr.io calls in push-all and login. The local SIF build is kept, since a local .sif is still useful for testing; it's just no longer published.
  • CI — remove the Setup Apptainer, ORAS login, Build Apptainer image and Push via ORAS steps. The SIF build goes too: nothing consumes it now, and it cost a multi-GB build on every run.

Verification

  • just --list parses; datestamp and apptainer_img are both still referenced.
  • Workflow YAML validates; the job is now six Docker-only steps.
  • No oras/apptainer references remain outside explanatory comments.

Downstream note

nf-binder-design references oras://ghcr.io/.../rc-foundry:0.2.0-weights in its rfd3 modules. Once that tag is next rebuilt it will be a Docker manifest, and those modules need to move to docker://.

A follow-up PR splits the rc-foundry weight downloads into multiple layers and adds rfd3na. It's deliberately held until this merges, so the rebuild it triggers runs under the fixed workflow and publishes a proper multi-layer image.

Both `just build --push` and the CI workflow pushed the Docker image and then
`apptainer push`ed a SIF to the *same* repo and tag. A tag resolves to exactly
one manifest, so the ORAS push silently replaced the multi-layer Docker
manifest with a single-blob SIF artifact.

The effect is visible in the registry today:

  rc-foundry:0.2.0-weights   sylabs SIF config, 1 layer  x 9.68 GB
  rc-foundry:0.1.12-weights  docker config,    11 layers x 9.30 GB (ORAS push
                             did not land here, so the Docker manifest survived)

A single 9.68 GB blob cannot be fetched in parallel and, more importantly,
cannot be resumed - a dropped connection restarts the whole transfer. Pulling
that image over a flaky link failed repeatedly with HTTP/2 PROTOCOL_ERROR and
then, over HTTP/1.1, `unexpected EOF`.

Changes:

- justfile: drop the ORAS push block and the two `apptainer registry login
  oras://ghcr.io` calls in `push-all` and `login`. The local SIF build is kept,
  since a local .sif is still useful for testing, but it is no longer published.
- CI: remove the Setup Apptainer / ORAS login / SIF build / ORAS push steps.
  The SIF build goes too - nothing consumes it now, and it cost a multi-GB
  build on every run.

Consumers should pull docker://ghcr.io/<org>/<container>:<version> and let
Apptainer or Singularity convert it locally.

Note for downstream: nf-binder-design currently references
oras://ghcr.io/.../rc-foundry:0.2.0-weights and will need to move to docker://
once that tag is next rebuilt.

Claude-Session: https://claude.ai/code/session_015xPe8vN1g2M5dZgpuv35z5
@pansapiens
pansapiens merged commit 3380b31 into main Sep 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant