Skip to content

ephemeral: Do container namespace setup in bcvk instead of bwrap - #329

Draft
yeetypete wants to merge 5 commits into
bootc-dev:mainfrom
yeetypete:feat/drop-bwrap
Draft

ephemeral: Do container namespace setup in bcvk instead of bwrap#329
yeetypete wants to merge 5 commits into
bootc-dev:mainfrom
yeetypete:feat/drop-bwrap

Conversation

@yeetypete

@yeetypete yeetypete commented Aug 23, 2026

Copy link
Copy Markdown

Closes: #7

Target images currently have to ship bubblewrap and bash. entrypoint.sh builds the hybrid root and then execs bwrap, which unshares, binds the API filesystems, calls pivot_root and becomes PID 1.

bcvk can do this itself: podman already provides the PID namespace with bcvk as its init, so the rest is replaced by a lightweight namespace setup in the container entrypoint. The dependency on both bubblewrap and bash go away with removing entrypoint.sh.

bcvk now runs in the image's userspace so it needs to be linked statically to work on older distros like stream9 with an older libc. Thoughts on this change?

This also fixes running Ubuntu 26.04 hosts. This was actually the original reason I looked into removing bubblewrap. On 26.04 the bwrap-userns-restrict AppArmor profile denies capabilities to bwrap's children. virtiofsd exits at startup with "can't apply the child capabilities" and bcvk polls for SSH until it times out. This might also be related to #306.

Tested by running all integration tests (just test-integration) on an ubuntu 26.04 host. Note that test_to_base_disk_integration_with_list fails with:

FAIL [   0.326s] integration-tests::integration-tests test_to_base_disk_integration_with_list
  stdout ───

    running 1 test
    Testing to-base-disk integration with base-disks list
    Initial base disk count: 12
    to-base-disk output: Created base disk: /home/psiegel/.local/share/libvirt/images/bootc-base-edda9015d6f14aa4.qcow2
    Final base-disks list:
    ┌───────────────────────────────────┬──────────┬──────┬──────────────────┬──────────────────────────────────────────────────────────┐
    │ NAME                              ┆ SIZE     ┆ REFS ┆ CREATED          ┆ IMAGE DIGEST                                             │
    ╞═══════════════════════════════════╪══════════╪══════╪══════════════════╪══════════════════════════════════════════════════════════╡
    │ bootc-base-7b23c3ceb66bf50c.qcow2 ┆ 1.69 GiB ┆ 0    ┆ 2026-08-23 01:12 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │
    ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    │ bootc-base-edda9015d6f14aa4.qcow2 ┆ 1.55 GiB ┆ 0    ┆ 2026-08-23 01:20 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │
    ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    │ bootc-base-92414e97c3abaecc.qcow2 ┆ 1.55 GiB ┆ 0    ┆ 2026-08-23 01:22 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │
    ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    │ bootc-base-487758b3396b22bb.qcow2 ┆ 1.69 GiB ┆ 0    ┆ 2026-08-23 01:10 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │
    ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    │ bootc-base-7ac29c1405a4366c.qcow2 ┆ 1.69 GiB ┆ 0    ┆ 2026-08-23 01:08 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │
    └───────────────────────────────────┴──────────┴──────┴──────────────────┴──────────────────────────────────────────────────────────┘

    Found 5 base disks
    test test_to_base_disk_integration_with_list ... FAILED

    failures:

    ---- test_to_base_disk_integration_with_list ----
    test panicked: Base disk count should increase after creation

    failures:
        test_to_base_disk_integration_with_list

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 89 filtered out; finished in 0.32s

  stderr ───

    thread '<unnamed>' (3280798) panicked at crates/integration-tests/src/tests/libvirt_to_base_disk.rs:205:9:
    Base disk count should increase after creation
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

But this looks unrelated.

Also tested together with https://github.com/yeetypete/bootc-ubuntu where this fixes a previous hang due to the AppArmor issue.

Assisted-by: AI

Requiring bubblewrap in the target image is not obvious to users, and
several distributions' images do not carry it. bwrap was providing an
unshare, a handful of bind mounts, a pivot_root and a PID namespace.
Podman already gives the container a PID namespace with bcvk as its
init, so what remains is a short piece of rustix.

The entrypoint script moves under /run, because `podman exec` joins the
namespace bcvk now runs in, where /var/lib/bcvk is not visible.

Closes: bootc-dev#7
Assisted-by: AI
Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
The script assembled /run/tmproot, read the image's systemd version and
exec'd bcvk. Doing this in Rust allows us to stop depending on bash in a
target image.

Assisted-by: AI
Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
…lation readme.

Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
bcvk is bind-mounted into the container it starts and runs there before
the entrypoint switches to the host's /usr. A dynamic build must load
against the image's glibc, which fails when the image ships an older one
than the build host, such as centos-bootc:stream9. A static build has no
such dependency.

Assisted-by: AI
Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
@yeetypete

Copy link
Copy Markdown
Author

@cgwalters thoughts? I see in #7 you mention trying out a different approach.

@cgwalters

Copy link
Copy Markdown
Collaborator

The tradeoff here is "link bcvk statically". Perhaps we could just have a little helper binary that's statically linked? Though I guess mechanically the "bcvk is a single binary" thing would need to be solved by actually bundling the inner statically linked binary as an ELF segment in the outer binary or so...this came up in #259 too.

There's some corner cases here, like I am not totally sure this will work if we're doing e.g. cross-arch qemu emulation, as we'd be mixing a native arch binary into a foreign arch root.

But...yeah on the positive side it's really nice to move what's currently in shell script into Rust.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Moves ephemeral container namespace setup from bubblewrap/bash into bcvk, addressing AppArmor compatibility and reducing target-image dependencies.

Changes:

  • Adds Rust-based hybrid-root and namespace initialization.
  • Reworks container launch and status monitoring around the static bcvk executable.
  • Removes the shell/bubblewrap entrypoint.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
Makefile Builds and installs a statically linked binary.
docs/src/installation.md Updates target-image requirements.
crates/kit/src/sandbox.rs Implements namespace and root setup.
crates/kit/src/run_ephemeral.rs Launches bcvk directly as container entrypoint.
crates/kit/src/run_ephemeral_ssh.rs Uses bcvk for status monitoring.
crates/kit/src/main.rs Runs sandbox setup before Tokio initialization.
crates/kit/scripts/entrypoint.sh Removes the former bubblewrap entrypoint.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread crates/kit/src/sandbox.rs
Comment on lines +49 to +50
fn init_tmproot() -> Result<()> {
let root = Path::new(TMPROOT);
Comment thread Makefile
Comment on lines +17 to +18
RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target $(CARGO_BUILD_TARGET)
install -D -m755 target/$(CARGO_BUILD_TARGET)/release/bcvk target/release/bcvk
Comment thread docs/src/installation.md
Comment on lines 35 to 37
For `bcvk ephemeral` operations, the bootc container images you run must contain:
- systemctl (systemd)
- objcopy (binutils)
- bwrap (bubblewrap)
- ssh, ssh-keygen (openssh-clients)
Comment thread crates/kit/src/sandbox.rs
Comment on lines +30 to +34
/// Assemble the hybrid root and make it this process's root.
///
/// Only the VM supervisor calls this. Anything else arrives later through
/// `podman exec`, which joins the supervisor's namespaces and so is already in
/// the hybrid root.
Comment thread Makefile

all: bin manpages

# bcvk is bind-mounted and ran inside the container it starts. We build it statically
@yeetypete

Copy link
Copy Markdown
Author

Though I guess mechanically the "bcvk is a single binary" thing would need to be solved by actually bundling the inner statically linked binary as an ELF segment in the outer binary or so...this came up in #259 too.

There's some corner cases here, like I am not totally sure this will work if we're doing e.g. cross-arch qemu emulation, as we'd be mixing a native arch binary into a foreign arch root.

Yes would be nice if this can be made to support qemu emulation too as i'd also like to be able to do this with bcvk in the future. Let me investigate a bit more how hard it would be to make this work more generally.

systemd-sysusers was executed from the image's PATH before the root
change, which required the target image to ship it and to be executable
on the host architecture. The host's systemd-sysusers cannot simply be
called by path before the pivot either, as its loader and libc would
then be resolved from the image.

Run it after pivot_root instead, where / is the hybrid root: the host's
binary, its libc and its sysusers.d then produce the /etc/passwd that
the host's ssh-keygen consumes. systemctl --version remains the only
image binary executed, as it has to report the image's systemd.

Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
Assisted-by: AI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ephemeral: switch to systemd-in-container

3 participants