WIP PE-9376: ship a default DHCP networkd config for physical NICs in the SLE Micro 5.5 image

[kirankn8]

Problem

A host installed from the SLE Micro 5.5 installer boots to multi-user with no network: all physical NICs down and unmanaged, /etc/systemd/network/ empty, systemd-networkd active, NetworkManager and wicked inactive, systemd-networkd-wait-online.service failing and sshd failing in a loop. No IP, no registration, console only.

Root cause

Confirmed against the real images, not inferred:

1. No CanvOS image ever ships a .network for physical NICs. kairos-init writes none at build time for any flavour. Verified directly: /etc/systemd/network/ is empty in kairos-slem:5.4-core-amd64-generic-v0.17.1, and /usr/lib/systemd/network/ in registry.suse.com/suse/sle-micro/5.5:latest holds only 99-default.link plus container/VM/wifi units — nothing matching physical ethernet. This matches the reported state exactly.

2. The only thing that ever creates 20-dhcp.network is a runtime yip stage, /system/oem/05_network.yaml (rootfs.before and initramfs). Every flavour depends on it — it is a single point of failure at boot.

3. SLE Micro is the only flavour where that failure is fatal. kairos-init pkg/stages/steps_init.go:513-521 (OnlyMicroRegex) does systemctl disable NetworkManager, and :523-534 enables systemd-networkd/systemd-resolved/sshd. Verified: NetworkManager.service is present in multi-user.target.wants in the SLE Micro base image and absent after kairos-init. On RHEL, kairos-init re-enables NetworkManager when the binary is present (steps_init.go:596), so a missed stage is survivable there. On SLE Micro nothing is left to bring the NICs up.

4. slem/5.5/Dockerfile does not compensate. Unlike rhel-core-images/Dockerfile.rhel9:87-89 and ubuntu-kairos-base/Dockerfile:248-249, it had no networking or unit-enablement step at all.

Ruled out

The ticket suggested the generated config might reference the wrong NIC names. It does not, and neither of the obvious suspects holds:

• only_os regex mismatch — ruled out. yip matches the regex against PRETTY_NAME, not NAME (yip/pkg/plugins/if_os.go:31 → sysinfo.OS.Name; zcalusic/sysinfo/os.go:50). SLE Micro's PRETTY_NAME is "SUSE Linux Enterprise Micro for Rancher 5.5", which matches SUSE.*. Verified empirically by running kairos-agent run-stage initramfs with each flavour's PRETTY_NAME: Ubuntu 22.04/24.04, RHEL 8/9, openSUSE Leap 15.6 and SLE Micro for Rancher 5.4/5.5 all produce both files. Only a NAME-style "SLE Micro 5.5" fails to match.
• 05_network.yaml missing or altered in the slem image — ruled out. Present in kairos-slem:5.4, byte-identical to the Ubuntu image's copy (2980 bytes, diff clean). Nothing in Earthfile or slem/5.5/Dockerfile removes it (the rm -f /system/oem/*_elemental-* glob does not match it).
• Units not enabled — ruled out. systemd-networkd, systemd-resolved and sshd are all enabled in the built slem base.

What I could not determine from the artifacts available to me is which runtime failure mode stopped the stage from taking effect on that specific host — I had no access to the box and cannot build the image (see Verification). Candidates include the /etc/systemd persistent bind (immucore rsyncs with -u, so a stale state copy pins forever) and OpMountBind being skipped when OpOverlayMount fails. This change makes that question moot: it removes the dependency on the runtime stage entirely.

The change

Ship the same DHCP default the yip stage would write, baked into the image:

• slem/5.5/overlay/usr/lib/systemd/network/20-dhcp.network — [Match] Name=en*, [Network] DHCP=yes
• slem/5.5/overlay/usr/lib/systemd/network/20-dhcp-legacy.network — [Match] Name=eth*
• slem/5.5/Dockerfile: COPY overlay/ / after kairos-init (same idiom as rhel-core-images), plus an assertion that the networkd binary exists and the three units are enabled — kairos-init enables networkd for SLE Micro without the test -f guard the RHEL branch has, and the binary only arrives via an openSUSE Leap OSS repo whose signing key this Dockerfile already documents as expired. Better to fail the build than ship an image with no network stack.

Content is byte-identical to what upstream writes, so behaviour matches every other flavour.

Both layers

slem/5.5/Dockerfile produces BASE_IMAGE, consumed by Earthfile:940 +base-image, which is the ancestor of both +iso-image (installer/live squashfs, Earthfile:1234) and +provider-image (installed root, Earthfile:733). /usr is read-only image content in both. Verified in the built image: /usr/lib/systemd/network/20-dhcp.network and 20-dhcp-legacy.network present with the expected content; /etc/systemd/network/ still empty (we clobber nothing).

Why /usr/lib and not /etc

/usr is never overlaid, bind-mounted or made writable by immucore — the only /usr paths it touches are /usr/local (the COS_PERSISTENT mountpoint) and /usr/local/.state. So the file is byte-for-byte the image's, on every boot, in live and installed mode alike.

/etc/systemd is different: it is in PERSISTENT_STATE_PATHS (00_rootfs.yaml), and immucore rsyncs it into /usr/local/.state/etc-systemd.bind with rsync -aquAX (immucore/internal/utils/mounts.go:95-106). The -u means a state copy that is newer on the receiver is never overwritten, and there is no --delete, so an /etc-based default would be pinned on first boot and could never be corrected or retracted by a later image.

How precedence is guaranteed

systemd-networkd merges /etc, /run and /usr/lib and applies the first matching .network in lexicographic order by filename. Stylus writes, all into /etc/systemd/network/:

Artifact	Name	stylus
bond .netdev	05-<bond>.netdev	pkg/network/network_networkd.go:664
per-NIC / bond member / VLAN	10-<iface>.network	network_networkd.go:493, :714, :363
DHCP file re-pointed at a Palette-selected NIC	10-<selected>.network	network_networkd.go:782
day-2 VLAN-on-bridge member	15-<member>.network	bridge_service.go:749
its own catch-all	20-dhcp{,-legacy}.network	domain/constants.go:368-369

Every file Stylus writes sorts before 20-, so user-data always wins. This is checked by test, not asserted: a Stylus-style /etc/systemd/network/10-enp2s0.network with a static address takes enp2s0 and the address is applied, while unlisted NICs keep DHCP.

The 20-dhcp names are deliberate rather than arbitrary — a higher number would have broken Stylus. network_networkd.go:762-783 renames 20-dhcp.network by that exact name when Palette selects a management NIC, and cmd/init-upgrade/main.go:252-292 reads /etc/systemd/network/20-dhcp{,-legacy}.network off disk to pin it across an A/B upgrade. A 99- file would be invisible to both. Same-name masking also means that if the runtime stage does write /etc/systemd/network/20-dhcp.network, it takes precedence over ours cleanly, with no duplicate-match conflict — verified by test.

User-data path checked: user-data.template:30-52 (stylus.site.network.interfaces.<iface>.{type,ipAddress,gateway,nameserver}), applied by stylus as a yip initramfs stage (pkg/service-handler/provider/event_handler.go:248) or synchronously at stylus.init during registration (:327-359). With no network: block stylus writes nothing at all (network_networkd.go:150-160) and the OS default is what applies — which is precisely the contract user-data.template:40 documents: "If omitted all interfaces will default to dhcp".

Multi-NIC behaviour (3-NIC host)

Name=en* matches all three, so enp1s0/enp2s0/enp3s0 are each managed and each runs an independent DHCP client — the files never fight over "which one is management", because they express no opinion about it. Management selection stays where it already lives: stylus.site.network.managementInterface.name binds the local UI, and Palette's per-host Nic.NicName drives selectedInterfaceName; both act via 10- files that outrank this default. A NIC with no DHCP server simply gets no lease, exactly as on every other flavour. Verified: three NICs, three distinct subnets, three leases, wait-online returns 0.

Behaviour is unchanged for lo and for container/VM links (vz-*, ve-*, host0, vt-*) — they stay unmanaged, checked by test.

Tests

New test/test-slem-network-defaults.sh — runs the shipped files against a real systemd-networkd on a three-NIC host, with veth pairs whose peers are served by the same networkd acting as a DHCP server, so leases are real. 17 checks:

• the image ships both files
• the defect reproduces with the files removed: all enp* report unmanaged
• with them: all three configured, each holding a lease from its own subnet, systemd-networkd-wait-online returns 0
• en* → 20-dhcp.network, eth* → 20-dhcp-legacy.network, lo and vz-* untouched
• Stylus 10-<iface>.network static config wins and its address is applied; unlisted NICs keep DHCP
• an /etc file of the same name masks the /usr/lib default and its settings are the ones applied

All 17 pass.

Cortex review

Breadth — coupled_files(spectrocloud/CanvOS, slem/5.5/Dockerfile), 26 changes, population 539. Every partner disposed of:

Partner	p(also changes)	support	Disposition
rhel-core-images/Dockerfile.rhel8	0.80	20	skipped — co-changes come from repo-wide kairos-init/base bumps, not networking. RHEL keeps NetworkManager enabled when the binary is present (kairos-init steps_init.go:596), so a missed yip stage is not fatal there.
rhel-fips/Dockerfile.rhel8	0.80	20	skipped — same RHEL reasoning.
ubuntu-fips/20.04/Dockerfile (sibling_variant)	0.80	20	skipped — verified the runtime stage covers PRETTY_NAME="Ubuntu 22.04/24.04"; flavour not reported broken.
rhel-core-images/Dockerfile.rhel9	0.52	13	skipped — same RHEL reasoning; verified stage covers RHEL 8/9 PRETTY_NAME.
rhel-fips/Dockerfile.rhel9	0.40	10	skipped — same.
rhel-core-images/Dockerfile.rhel9.sat	0.40	10	skipped — same; .sat variants do not even COPY the overlay.
rhel-core-images/Dockerfile.rhel8.sat	0.40	10	skipped — same.
ubuntu-fips/22.04/Dockerfile.ubuntu22.04-fips	0.28	7	skipped — Ubuntu, covered by the stage.
rhel-stig/Dockerfile.rhel9	0.24	6	skipped — RHEL.
rhel-stig/Dockerfile.rhel9-fips	0.24	6	skipped — RHEL.
slem/5.4/README.md	0.12	3	skipped — STALE (836d); no user-facing behaviour change to document for 5.4.
slem/5.5/build.sh	0.12	3	skipped — STALE (836d); it is a thin docker build wrapper, unaffected.
ubuntu-fips/24.04/Dockerfile.ubuntu24.04-fips	0.12	3	skipped — Ubuntu, covered by the stage.
ubuntu-kairos-base/Dockerfile	0.08	2	skipped — informative: false, STALE 476d.
rhel-fips/dracut.conf	0.08	2	skipped — informative: false.

coupled_directories(slem/5.5) adds rhel-core-images 0.807/20, rhel-fips 0.768/20, ubuntu-fips/20.04 0.742/20 (same dispositions) and slem/5.4 0.618/4 — see the flag below.

Flagged, deliberately not changed: the CI-published slem base is slem-5.4, and it is built from registry.suse.com/suse/sle-micro-rancher/5.4 through dockerfiles/kairosify/Dockerfile.kairosify (.github/workflows/base-images.yaml:124-134) — not through slem/5.4/Dockerfile or slem/5.5/Dockerfile, neither of which is referenced by Earthfile or CI at all. slem/5.5 is a hand-run path (slem/5.5/build.sh), which is what the reported installer was built from, so the fix is in the right place for this ticket. But I confirmed kairos-slem:5.4 carries the identical exposure (empty /etc/systemd/network, NetworkManager not enabled). Extending the fix to the kairosify builder would touch the file that builds every flavour and I have no way to verify it here — calling that out for a maintainer decision rather than changing it blind.

Depth — upstream_repos(spectrocloud/CanvOS): "Nothing validated upstream. 1 discovery-tier edge withheld. This repository declares no internal dependencies." coupling_chain(upstream): no chains, "there was nothing to walk" — not evidence of absence.

So Cortex offers no validated upstream edge. By code reading the defect does originate upstream, in kairos-io/kairos-init: the SLE Micro branch turns off the working network stack and enables an unconfigured one, and no version through main fixes it (v0.17.3 changes only sshd hardening/CI/deps; main is byte-identical on values.go:201-206, packagemaps.go:552 and steps_init.go:488-534, and the repo now carries an archive notice pending migration into kairos-io/kairos). kairos-io is outside Cortex's spectrocloud coverage, so its silence here says nothing — the upstream conclusion is from reading the code, not from Cortex.

The upstream-shaped fix would be to write the .network at build time in the OnlyMicroRegex branch, or to guard the networkd enable with test -f the way the RHEL branch does. That is out of scope for this repo and the repo is being archived. Shipping the config in our own OS image is not merely a symptom patch: an OS image is the right place for its own default network configuration, and doing it here removes the runtime single point of failure for the flavour that cannot survive it.

Verification — read this before merging

This PR is WIP. GATE 3 (build the SLE Micro installer and boot it in a multi-NIC VM) was NOT done.

Blocked: slem/5.5/Dockerfile stage 1 registers against SCC and requires a SUSE_REGCODE build secret (slem/5.5/build.sh:15-35). No registration code is reachable from this environment — Keeper reports persistent login is not usable non-interactively. Without it the image cannot be built, so no ISO and no VM boot.

Verified

• Real systemd-networkd, three NICs, real DHCP leases. All 17 checks in test/test-slem-network-defaults.sh pass, including the before/after (unmanaged → configured), wait-online returning 0, and both precedence cases. Run on a native arm64 Kairos base image with this overlay applied — not on the SLE Micro image (see below).
• Both new Dockerfile instructions on genuine SLE Micro userspace. Built COPY overlay/ / + the assertion FROM us-docker.pkg.dev/palette-images/edge/kairos-slem:5.4-core-amd64-generic-v0.17.1: COPY lands both files at /usr/lib/systemd/network/ with the expected content, /etc/systemd/network/ stays empty, and systemctl is-enabled systemd-networkd systemd-resolved sshd returns enabled ×3. The same assertion correctly fails against the raw pre-kairos-init sle-micro/5.5 base, so it is not vacuous.
• Root-cause claims, each against the real images as described above.

Not verified

• No VM boot. No ip -br link / networkctl list / systemctl status sshd on a booted SLE Micro host. The acceptance criteria are argued from the test above, not observed on the target OS.
• The full slem/5.5 build has never been run — the SCC stage, kairos-init, and my two instructions in sequence. Only the last two were exercised, on a 5.4 base.
• The behaviour test ran on an arm64 Ubuntu-based Kairos image, not SLE Micro. SLE Micro is amd64-only here and its systemd-networkd cannot start under emulation on this arm64 host (Could not create manager: Protocol not supported), so the test image was a native stand-in with the same overlay. systemd majors match (249), and the files are plain networkd config with no distro-specific syntax, but this is a stand-in. On an amd64 runner the same script pointed at the real image would be faithful — that is the remaining gate.
• docker build --check lint did not run (transient docker.io egress failure resolving the # syntax= image, unrelated to this change).
• UKI / trusted-boot path not assessed. immucore's UKI DAG is a separate code path; /usr/lib should behave the same but I did not verify it.

To take this out of WIP

On a host with a SUSE_REGCODE and an amd64 Docker daemon:

cd slem/5.5 && ./build.sh <REGCODE> && cd -
./test/test-slem-network-defaults.sh slem-kairos:5.5      # now faithful: real image, native arch
./earthly.sh +iso --BASE_IMAGE=slem-kairos:5.5 ...        # then boot with 3 NICs and check:
#   ip -br link ; networkctl list ; ls -la /etc/systemd/network/
#   systemctl is-active systemd-networkd
#   systemctl status systemd-networkd-wait-online.service ; systemctl status sshd ; ip -br addr

🤖 Generated with Claude Code