ephemeral: Do container namespace setup in bcvk instead of bwrap - #329
ephemeral: Do container namespace setup in bcvk instead of bwrap#329yeetypete wants to merge 5 commits into
Conversation
Requiring bubblewrap in the target image is not obvious to users, and several distributions' images do not carry it. bwrap was providing an unshare, a handful of bind mounts, a pivot_root and a PID namespace. Podman already gives the container a PID namespace with bcvk as its init, so what remains is a short piece of rustix. The entrypoint script moves under /run, because `podman exec` joins the namespace bcvk now runs in, where /var/lib/bcvk is not visible. Closes: bootc-dev#7 Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
The script assembled /run/tmproot, read the image's systemd version and exec'd bcvk. Doing this in Rust allows us to stop depending on bash in a target image. Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
…lation readme. Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
bcvk is bind-mounted into the container it starts and runs there before the entrypoint switches to the host's /usr. A dynamic build must load against the image's glibc, which fails when the image ships an older one than the build host, such as centos-bootc:stream9. A static build has no such dependency. Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
|
@cgwalters thoughts? I see in #7 you mention trying out a different approach. |
|
The tradeoff here is "link bcvk statically". Perhaps we could just have a little helper binary that's statically linked? Though I guess mechanically the "bcvk is a single binary" thing would need to be solved by actually bundling the inner statically linked binary as an ELF segment in the outer binary or so...this came up in #259 too. There's some corner cases here, like I am not totally sure this will work if we're doing e.g. cross-arch qemu emulation, as we'd be mixing a native arch binary into a foreign arch root. But...yeah on the positive side it's really nice to move what's currently in shell script into Rust. |
There was a problem hiding this comment.
Pull request overview
Moves ephemeral container namespace setup from bubblewrap/bash into bcvk, addressing AppArmor compatibility and reducing target-image dependencies.
Changes:
- Adds Rust-based hybrid-root and namespace initialization.
- Reworks container launch and status monitoring around the static bcvk executable.
- Removes the shell/bubblewrap entrypoint.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Builds and installs a statically linked binary. |
docs/src/installation.md |
Updates target-image requirements. |
crates/kit/src/sandbox.rs |
Implements namespace and root setup. |
crates/kit/src/run_ephemeral.rs |
Launches bcvk directly as container entrypoint. |
crates/kit/src/run_ephemeral_ssh.rs |
Uses bcvk for status monitoring. |
crates/kit/src/main.rs |
Runs sandbox setup before Tokio initialization. |
crates/kit/scripts/entrypoint.sh |
Removes the former bubblewrap entrypoint. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| fn init_tmproot() -> Result<()> { | ||
| let root = Path::new(TMPROOT); |
| RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target $(CARGO_BUILD_TARGET) | ||
| install -D -m755 target/$(CARGO_BUILD_TARGET)/release/bcvk target/release/bcvk |
| For `bcvk ephemeral` operations, the bootc container images you run must contain: | ||
| - systemctl (systemd) | ||
| - objcopy (binutils) | ||
| - bwrap (bubblewrap) | ||
| - ssh, ssh-keygen (openssh-clients) |
| /// Assemble the hybrid root and make it this process's root. | ||
| /// | ||
| /// Only the VM supervisor calls this. Anything else arrives later through | ||
| /// `podman exec`, which joins the supervisor's namespaces and so is already in | ||
| /// the hybrid root. |
|
|
||
| all: bin manpages | ||
|
|
||
| # bcvk is bind-mounted and ran inside the container it starts. We build it statically |
Yes would be nice if this can be made to support qemu emulation too as i'd also like to be able to do this with bcvk in the future. Let me investigate a bit more how hard it would be to make this work more generally. |
systemd-sysusers was executed from the image's PATH before the root change, which required the target image to ship it and to be executable on the host architecture. The host's systemd-sysusers cannot simply be called by path before the pivot either, as its loader and libc would then be resolved from the image. Run it after pivot_root instead, where / is the hybrid root: the host's binary, its libc and its sysusers.d then produce the /etc/passwd that the host's ssh-keygen consumes. systemctl --version remains the only image binary executed, as it has to report the image's systemd. Signed-off-by: Peter Siegel <psiegel2000@icloud.com> Assisted-by: AI
Closes: #7
Target images currently have to ship bubblewrap and bash.
entrypoint.shbuilds the hybrid root and then execs bwrap, which unshares, binds the API filesystems, callspivot_rootand becomes PID 1.bcvk can do this itself: podman already provides the PID namespace with bcvk as its init, so the rest is replaced by a lightweight namespace setup in the container entrypoint. The dependency on both bubblewrap and bash go away with removing
entrypoint.sh.bcvk now runs in the image's userspace so it needs to be linked statically to work on older distros like stream9 with an older libc. Thoughts on this change?
This also fixes running Ubuntu 26.04 hosts. This was actually the original reason I looked into removing bubblewrap. On 26.04 the bwrap-userns-restrict AppArmor profile denies capabilities to bwrap's children. virtiofsd exits at startup with "can't apply the child capabilities" and bcvk polls for SSH until it times out. This might also be related to #306.
Tested by running all integration tests (
just test-integration) on an ubuntu 26.04 host. Note thattest_to_base_disk_integration_with_listfails with:FAIL [ 0.326s] integration-tests::integration-tests test_to_base_disk_integration_with_list stdout ─── running 1 test Testing to-base-disk integration with base-disks list Initial base disk count: 12 to-base-disk output: Created base disk: /home/psiegel/.local/share/libvirt/images/bootc-base-edda9015d6f14aa4.qcow2 Final base-disks list: ┌───────────────────────────────────┬──────────┬──────┬──────────────────┬──────────────────────────────────────────────────────────┐ │ NAME ┆ SIZE ┆ REFS ┆ CREATED ┆ IMAGE DIGEST │ ╞═══════════════════════════════════╪══════════╪══════╪══════════════════╪══════════════════════════════════════════════════════════╡ │ bootc-base-7b23c3ceb66bf50c.qcow2 ┆ 1.69 GiB ┆ 0 ┆ 2026-08-23 01:12 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-edda9015d6f14aa4.qcow2 ┆ 1.55 GiB ┆ 0 ┆ 2026-08-23 01:20 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-92414e97c3abaecc.qcow2 ┆ 1.55 GiB ┆ 0 ┆ 2026-08-23 01:22 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-487758b3396b22bb.qcow2 ┆ 1.69 GiB ┆ 0 ┆ 2026-08-23 01:10 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-7ac29c1405a4366c.qcow2 ┆ 1.69 GiB ┆ 0 ┆ 2026-08-23 01:08 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ └───────────────────────────────────┴──────────┴──────┴──────────────────┴──────────────────────────────────────────────────────────┘ Found 5 base disks test test_to_base_disk_integration_with_list ... FAILED failures: ---- test_to_base_disk_integration_with_list ---- test panicked: Base disk count should increase after creation failures: test_to_base_disk_integration_with_list test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 89 filtered out; finished in 0.32s stderr ─── thread '<unnamed>' (3280798) panicked at crates/integration-tests/src/tests/libvirt_to_base_disk.rs:205:9: Base disk count should increase after creation note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceBut this looks unrelated.
Also tested together with https://github.com/yeetypete/bootc-ubuntu where this fixes a previous hang due to the AppArmor issue.
Assisted-by: AI