Perlmutter spike: login-node findings (layer 7)

[EiffL]

Results of the login-node half of the layer-7 spike (CLAUDE.md "Pending the one-time Perlmutter spike"), run on login37, 2026-08-21. The allocation half (sbatch materialize, multi-node) is still pending — list at the bottom.

Host: SLES 15-SP6, kernel 6.4.0-cray_shasta, x86_64 · podman-hpc 1.2.2 wrapping podman 5.8.2, no docker · uv self-installed in ~/.local/bin · git-annex wheel 10.20260717 · $SCRATCH = Lustre, $HOME = GPFS.

Spike questions answered

Landlock is not in the SLES boot LSM list

/sys/kernel/security/lsm → lockdown,capability,selinux,bpf. Direct landlock_create_ruleset syscall returns ENOTSUP. The handled branch works exactly as designed: lc run executes, prints the downgrade note, lc status shows sandbox: none, runs record fs: open. So on Perlmutter direct mode is unenforced and containerized mode is the only real boundary — worth saying in the user docs' NERSC material.

One message nit: the note's parenthetical says "kernel < 5.13, blocked by seccomp, or unsupported arch" — none of which is the cause here (kernel 6.4; the LSM simply isn't in the boot lsm= list). A fourth clause would make the note truthful on the machine most likely to show it. Compute nodes may boot a different image; re-check there.

podman-hpc migrate: bare id accepted, re-run is a no-op

podman-hpc migrate <bare-image-id> exits 0 in 0.55 s, and a second invocation is the same 0.55 s no-op. runtime_for_run's unconditional migrate is validated — no guard needed.

--module site mounts vs fs: declared: honest by default, one leak vector

All 13 site modules (/etc/podman_hpc/modules.d/) are opt-in, each gated on a CLI flag or an env var: MOUNT_HOME, MOUNT_SCRATCH, MOUNT_CFS, MOUNT_JUPYTER, ENABLE_GPU, ENABLE_MPICH_SS, ENABLE_NCCL*, ENABLE_OPENMPI_*, ENABLE_CUDA_MPICH_SS, ENABLE_CVMFS. A bare podman-hpc run injects no site mounts (verified: /proc/mounts inside the container shows nothing beyond standard plumbing), and nothing in /etc/profile.d sets the gate variables. So fs: declared is honest out of the box.

The leak vector is real, though: MOUNT_SCRATCH=1 podman-hpc run … silently bind-mounts $SCRATCH into the container (verified) — and lc passes the driver's ambient environment to the runtime wrapper (the container env allowlist doesn't cover the wrapper's own env). A user with export MOUNT_HOME=1 in their dotfiles gets $HOME inside every recipe container while the manifest attests fs: declared. This is the recorded "one item that could add a flag", now with a concrete shape. Tension to weigh before deciding: ENABLE_GPU/ENABLE_MPICH_SS are the legitimate site mechanism for GPU/MPI recipes under podman-hpc, so a blanket scrub (the UV_* treatment) would close the honesty hole and break GPU recipes with it. Options seem to be: scrub MOUNT_* only; attest the gate variables that were set; or a declaration surface for modules. Not decided here.

git-annex on Lustre vs GPFS: throughput fine, small-file add is the cost

200 MB file + 500×16 KB files, template .gitattributes, annex.thin=true add — the exact operations lc pays:

operation	$SCRATCH (Lustre)	$HOME (GPFS)
git add 200 MB (filter=annex, thin)	1.9 s	2.1 s
git add 500 × 16 KB	25.4 s (~50 ms/file)	8.6 s
commit	2.2 s	1.7 s
git annex get 200 MB (same-fs origin)	4.5 s	3.1 s
git annex get 500 small	1.0 s	0.6 s

Thin hard links behave normally on Lustre (link count 2, object store not doubled) — the "does Lustre make the hard link behave differently" question is answered: no. The real cost is per-file metadata latency on add (wall 25 s vs ~8 s CPU): an output of many small files pays ~50 ms each on $SCRATCH. One output = one file stays cheap; sharded outputs (thousands of files) are where this bites. Nothing to design around yet.

nidXXXXXX resolution — partial (login-node) answer

nid200001 / nid001000 resolve via site DNS from the login node (to .chn.perlmutter addresses). Whether a peer compute node resolves and can reach SLURMD_NODENAME's address (vs needing --interface hsn0) still needs the allocation.

Bugs found

1. tests/test_container_smoke.py::_inspect_id KeyError: 'podman-hpc' — the helper's argv dict has only podman/docker keys, but the suite parametrizes over podman-hpc. Fails test_build_commits…[podman-hpc] and test_materialize_end_to_end…[podman-hpc] at their final assert. Everything before the helper passed — see below. podman-hpc wraps podman, so ["podman-hpc", "image", "inspect", …] would work; the fix is treating the podman family positively, as container._PODMAN_FAMILY does. This bug is only reachable on a host with podman-hpc installed — which is exactly why the spike says running the suite here is the spike.

2. Bare podman cannot run containers on a login node — podman run --userns=keep-id fails at container creation: creating an ID-mapped copy of layer … chown …/.local/share/containers/storage/…: operation not permitted, with podman's own warning Network file system detected as backing store ($HOME is GPFS). Build works; run does not. This is precisely why NERSC ships podman-hpc, and detection order (podman-hpc → podman) means real lc usage never hits it. Consequence for the suite: test_the_probe_and_its_boundary[podman] and test_materialize_end_to_end…[podman] fail on this host, so LC_CONTAINER_TESTS_REQUIRED=1 cannot go fully green on a NERSC login node. Worth considering a skip (with reason) for the bare-podman parametrization when its graph root sits on a network filesystem — podman's warning string shows it can be probed.

3. Minor, template/docs: the scaffolded pyproject.toml pins [tool.uv] required-version = ">=0.12"; NERSC users self-install uv (per docs/user/install.md), and a stale one (0.11.8 here) fails lc init with uv's own clear uv self update message. Works as designed, just recording that the failure mode is real on this site (fixed here by updating; uv is now 0.12.5).

What works end-to-end on a login node

• lc init on $SCRATCH: 1.2 s, converges clean on Lustre.
• Login guard: lc materialize refuses (exit 1) with the exact salloc/sbatch remedies; the rerun entry point (python -m lightcone.engine.worker …) refuses too (exit 2). lc status / lc materialize --check / lc run all work here, as documented.
• lc run (direct mode): 0.5 s round trip on $SCRATCH, correct downgrade note.
• lc build via podman-hpc: real project on $SCRATCH, [tool.lightcone.image] with apt-install — build, docker-archive save, annex commit (225 MB, thin: link count 2) in 11 s warm. Detection picked podman-hpc over podman correctly with NERSC_HOST set, no fixtures.
• Containerized lc run: executes in-image, ~9 s including the in-container env converge. lc status header: sandbox: podman-hpc (fs: declared, network: allowed).
• Smoke suite under podman-hpc: test_the_probe_and_its_boundary[podman-hpc] passed; test_a_rerun_on_a_clone_fetches_the_archive_and_reproduces[podman-hpc] passed — a real datalad rerun on a bytes-free clone fetched the archive through the annex and reproduced, on the site runtime. The e2e materialize ran a real Dask cluster in-image and wrote a manifest asserting mechanism: podman-hpc, fs: declared before dying in the helper bug above.
• Full suite baseline: 564 passed, 27 skipped, 4 failed in 7:42 — the 4 failures are exactly the two bugs above (2× the _inspect_id helper under podman-hpc, 2× bare podman's network-fs store). Everything else is green on this host, including the venue suite's real worker-process path and the crate smoke against the real validator; the sandbox enforcement suite skips as designed (no mechanism, LC_SANDBOX_TESTS_REQUIRED unset).

Still needs the allocation

• srun --overlap / --cpus-per-task behavior inside salloc/sbatch steps
• SLURMD_NODENAME reachability from peer nodes (else --interface hsn0)
• SLURM_CPUS_ON_NODE on a CPU node (128 vs 256 hyperthreads)
• cold-Lustre distributed import vs the 120 s worker wait (warm import: 0.9 s)
• one real lc materialize through sbatch, then multi-node containerized materialize (the podman-hpc shared-store claim in anger)
• compute-node checks of the two host facts above: the boot LSM list, and whether any prolog sets MOUNT_* there

Residue left on the machine: uv upgraded 0.11.8→0.12.5 (~/.local/bin), and the smoke images in the per-user podman-hpc stores (/images/$UID_hpc, $SCRATCH/storage — ~235 MB squash). All probe repos/projects were deleted.

[EiffL]

Allocation half — run 2026-08-21, 2-node interactive allocation (nid[004167-004168], milan CPU nodes, QOS interactive, whole nodes)

This completes the spike: every pending question now has a measured answer. Both materialize modes work multi-node. Two new real findings (the uv cache lock and the squash-store wedge), one wart (teardown), and a handful of recorded host facts.

The headline: multi-node materialize works, in both modes

• Direct mode: 4 universes × 1 output, recipes recording hostname — lc materialize inside the allocation made all 4 in 2m03s, with u1/u3 executing on nid004167 and u2/u4 on nid004168 (proven by the outputs' own bytes and the scheduler's worker logs). Tree clean afterwards, one [DATALAD RUNCMD] commit per output, all four manifests record the same pre-run git_sha (the HEAD discipline holding), mechanism: none, fs: open honestly.
• Containerized mode: declaring [tool.lightcone.image] put all 4 outputs behind (correct — environment moved, nothing stale), and lc materialize --refresh remade them in-image on both nodes in 2m02s. Manifests: mechanism: podman-hpc, fs: declared, image id pinned, arch: amd64. The podman-hpc shared-store claim is real: one driver-side migrate, both nodes ran the image.
• Compute nodes can lc build (1m23s with a cold node-local store — outbound network works). Login-node builds remain the documented path but aren't the only one.

New finding 1: uv's cache lock fails on $HOME from compute nodes

lc init on $SCRATCH failed on the compute node: Could not acquire lock for ~/.cache/uv … os error 524 (ENOTSUPP — flock unsupported on the home filesystem from compute nodes; login nodes are fine, which is why the login half never saw this). This is precisely the spec's "site registry supplies UV_CACHE_DIR on Perlmutter" row, now confirmed empirically — and lc has no site registry yet, so today every compute-node run needs export UV_CACHE_DIR=$SCRATCH/… before any uv-touching verb. UV_CACHE_DIR is in _UV_KEPT, so it flows through the ambient scrub; with it set, everything worked. Until the registry lands, this wants a line in docs/user/cluster.md — it is the first wall every real Perlmutter user will hit. (Side benefit of scratch: cache and project share a filesystem, so uv's hardlink sharing works.)

New finding 2: rebuilding the same tag wedges the podman-hpc shared store

The first containerized materialize failed with Error: configure storage: read-only image store assigns the same name to multiple images — every podman-hpc invocation for the project failed, not just the run. Cause: the login-node build (this morning's #192 work) had squash-migrated lc-env-b34dd13718ade3ec into $SCRATCH/storage; the compute-node rebuild of the identical declaration produced the same deterministic tag with a different image id (builds aren't bit-reproducible), and migrate pushed a second same-named image into the R/O store, which podman refuses to configure. Remedy: podman-hpc rmsqi <tag> clears the stale copy; after that everything ran. This is the multi-node runtime's sharp edge: rebuilds under an unchanged tag are normal lc behavior, so runtime_for_run's podman-hpc branch probably wants to detect/replace a stale same-tag squash around its migrate step (or at minimum map this error string to a refusal naming rmsqi). Filing-worthy on its own.

Spike questions, measured

• srun --overlap --ntasks=N --ntasks-per-node=1 --cpus-per-task=<cpus> inside an interactive salloc: works exactly as the venue composes it — both nodes launch, step affinity 0-255, no interference with the allocation's interactive step.
• SLURM_CPUS_ON_NODE on a CPU node: 256 (hyperthreads, not the 128 physical cores). Workers get --nthreads=256.
• nidXXXXXX resolution from peer nodes: works — site DNS resolves SLURMD_NODENAME to the .chn address from peers, and workers connected to the scheduler bound there. No --interface hsn0 needed.
• Cold-Lustre/GPFS distributed import: 21.6 s on an untouched node (tool env on $HOME/GPFS) — comfortably inside the 120 s worker wait. (Warm re-import is still ~10 s; GPFS from compute nodes is just slow.)
• podman-hpc migrate: already answered on the login side (bare id accepted, 0.55 s no-op re-run) — and the duplicate-tag wedge above is the migrate story's missing half.
• Landlock on compute nodes: absent, same boot LSM list as login (lockdown,capability,selinux,bpf). Direct-mode runs on Perlmutter are unenforced everywhere; the downgrade note printed per task and the manifests recorded fs: open — the handled branch, verified end-to-end in real manifests.
• Site module env-gates on compute nodes: none set (MOUNT_*/ENABLE_* all absent in a fresh salloc shell), matching login. fs: declared stayed honest in the real containerized run.

Wart: teardown is not the designed silent retire

Both clean runs ended with srun: forcing job termination, srun: error: nidXXXXXX: task N: Killed, and distributed's Removing worker … caused the cluster to lose already computed task(s) warnings. The runs were fully successful — every output made and committed — so this is cosmetic, but the layer-7 design says retirement should let workers exit 0 and srun end silently, and on real SLURM it escalates instead. Worth a look at whether --no-nanny workers actually exit on retire within the escalation window here.

Site footgun, recorded

NERSC's salloc environment exports OMP_NUM_THREADS=2 (from the 1-cpu-per-task request shape), and srun propagates the ambient value into steps (verified: OMP_NUM_THREADS=77 srun … → step sees 77). Recipes therefore inherit OMP_NUM_THREADS=2 on a 256-thread node — not lc's doing, but BLAS-heavy recipes will silently underuse the node. A docs/user/cluster.md note (or a venue decision about the variable) may be warranted.

Not run, deliberately

A separate sbatch --wrap 'lc materialize' job — the detection ladder it exercises (SLURM_JOB_ID set → slurm_client) is identical to what both salloc runs just validated, and it would burn a second allocation. Left for the first real batch use.

Residue: probe project deleted; $SCRATCH/uv-cache (36 KB) and the squashed lc-env image in $SCRATCH/storage left in place.

[EiffL]

Implemented and verified on Perlmutter (2-node interactive allocation, nid[004167-004168])

Everything below was built against the findings above and then verified by running it on the machine, not by reasoning about it. Full suite: 568 passed, 0 failed on a compute node.

What changed

1. The site registry supplies uv's plumbing (venue.site_env, consumed by project.child_env). The _Site row gained a scratch field; on a known center, unset/empty UV_CACHE_DIR and UV_PYTHON_INSTALL_DIR are filled from $SCRATCH/.lightcone/. Per variable, ambient always wins, silent by design (a default doing its job is not a warning). Deterministic in the ambient environment, so every node of an allocation derives the same answer with nothing handed down. This is spec §4's "the site registry supplies UV_CACHE_DIR on Perlmutter" row, finally made real — and _UV_KEPT already kept both variables for exactly this story.

2. MOUNT_* is scrubbed; ENABLE_* is kept (project._mount_scrubbed). One predicate in the one env composer covers both wrapper-spawn paths (project._run for build/load/migrate/sync, boundary.run for recipe exec). uv_scrub_warning → scrub_warning, now naming both families.

3. A stale squashed image is healed before the load (container._heal_squash). Probes podman-hpc images --format '{{.Id}} {{.ReadOnly}}' localhost/<tag>; a read-only row whose id differs from the current one gets rmsqi'd. An unrecognized probe outcome is left for migrate's own loud path, so the heal cannot break a healthy run. Before the load, not just the migrate — a wedged store fails the load too, on a node that has not loaded yet.

4. The driver releases each future as its result is consumed (_Dask.completed). The cause of the teardown noise: a key still client-held at teardown makes retire_workers replicate it onto peers that are also retiring, so the run stalls past the reap grace and srun gets killed.

5. Landlock downgrade note names the LSM list first — Perlmutter is kernel 6.4 with Landlock simply absent from the boot list, so the old text listed three causes none of which applied.

One bug found by the verification, fixed

container.sync bind-mounts uv's cache directory, but in containerized mode uv never runs on the host, so nothing creates it — podman then dies with a raw statfs ... no such file or directory about a path the user never named. Latent before; my cache supply made it reachable. sync now creates the directory. (The test fixture's stubbed /home/user/.cache/uv moved to tmp_path, since it is now really created.)

A second filesystem finding, and why it grew the fix

The suite flaked on compute nodes — four test_materialize tests failing per run, a different four each time, each passing in isolation. The real error, once captured:

error: Failed to bytecode-compile Python file in: .venv/lib/python3.13/site-packages
  Caused by: Python startup timed out (60s)

uv's own 60 s ceiling, blown by interpreter startup. Measured on the same compute node, 5 runs each:

interpreter store	python -c pass
$SCRATCH (Lustre)	0.22, 0.06, 0.05, 0.05, 0.05 s
$HOME (GPFS)	9.21, 1.22, 1.38, 1.08, 0.41 s

So home is 20-180x slower and, under load, crosses uv's timeout — a hard failure mid-run, not a slowdown. That is the same disease as the cache lock, with a different symptom, which is why the supply covers both variables rather than only UV_CACHE_DIR. (An earlier copy-based measurement of mine was invalid — the uv store's version directory is a symlink farm, so cp -a duplicated links and both sides still read from home. The table above is from a real second install on scratch.)

The proof, and the reason this matters beyond tidiness — same suite, same node, only the interpreter store moved:

run	result	wall
interpreter on $HOME	9 failed, 568 passed	42:16
interpreter on $SCRATCH	568 passed, 0 failed	13:45

Three times faster and the flake is gone. Note the suite itself does not benefit automatically: conftest scrubs NERSC_HOST suite-wide (deliberately — it must exercise the non-site path), so developers running the suite on a NERSC compute node should export UV_PYTHON_INSTALL_DIR=$SCRATCH/.lightcone/uv-python or expect the flake. Real users get it from child_env.

Live verification

Check	Result
lc init on a compute node, empty environment	works — the flock/524 wall is gone; cache and interpreters land on $SCRATCH/.lightcone/
Project venv interpreter startup	0.04-0.05 s (was 0.4-9.2 s)
Direct 4-universe multi-node materialize	4 outputs, u1/u3 on nid004167 and u2/u4 on nid004168, clean tree, zero teardown noise
Containerized multi-node materialize	in-image on both nodes, mechanism: podman-hpc, fs: declared, image id pinned
Squash heal, manufactured wedge	built a genuine duplicate-name wedge (same tag, different id, migrated) that failed every podman operation; lc materialize --refresh healed and ran clean
Squash heal, natural case	fired on its own during a later rebuild — the case users actually hit
MOUNT_HOME=1 lc run ls $HOME	"No such file or directory" + the warning naming the drop. Mutation-checked: bare podman-hpc with the same variable does mount it
Fresh containerized first run, nothing exported	lc init → lc materialize → committed result in 61 s, clean tree
Full suite (smoke deselected)	568 passed, 0 failed, 13:45
ruff + mypy	clean

Docs landed with the code: a "NERSC (Perlmutter) specifics" section in the cluster guide (cache/interpreter supply, the Landlock reality, the MOUNT_/ENABLE_ split, the OMP_NUM_THREADS trap), the duplicate-squash entry in troubleshooting, and four API pages. CLAUDE.md carries the decisions.

Still open (deliberately out of scope here)

• The smoke suite's _inspect_id KeyError: 'podman-hpc' and the bare-podman-on-network-fs failures — both from the first half of this issue, unfixed, so LC_CONTAINER_TESTS_REQUIRED=1 still cannot go green on a NERSC host.
• OMP_NUM_THREADS is documented, not touched.
• A separate sbatch materialize was skipped by decision: it exercises the identical detection path both salloc runs just validated.
• --ignore=tests/test_container_smoke.py silently does nothing in this repo (testpaths wins); --deselect is the spelling that works. Worth knowing before trusting an "except smoke" run.