Skip to content

feat: Add Podman support as drop-in replacement for Docker - #360

Open
ZPascal wants to merge 3 commits into
cloudfoundry:mainfrom
ZPascal:add-podman-support
Open

feat: Add Podman support as drop-in replacement for Docker #360
ZPascal wants to merge 3 commits into
cloudfoundry:mainfrom
ZPascal:add-podman-support

Conversation

@ZPascal

@ZPascal ZPascal commented May 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Podman runtime support — scripts/detect-runtime.sh auto-detects Docker or Podman (explicit CONTAINER_RUNTIME override takes precedence). All scripts (create-kind.sh, delete-kind.sh, init.sh) use the detected runtime; no Docker
    aliasing required.
  • Podman on macOS/Windows — scripts/setup-podman-vm.sh creates and configures a rootful Podman machine (4 CPU, 8 GB, 60 GB) with NFS kernel modules and inotify limits applied inside the VM.
  • Cilium disabled under rootless Podman — SKIP_CILIUM=true is set automatically when Podman is detected. This skips the Cilium Helm release and suppresses CiliumClusterwideNetworkPolicy CRDs
    (assets/base-chart/templates/network-policy.yaml); https://github.com/kubernetes-sigs/kindnet provides pod networking instead. assets/values/cilium-podman.yaml documents a degraded-mode Cilium config for opt-in use.
  • KinD config for Podman — kind-podman.yaml adds BPF filesystem mounts on all nodes (required by kindnet on some kernels).
  • Remove CAPI udp-forwarder sidecar — the udp-forwarder sidecar from cloud_controller_ng was removed; loggregator endpoint is now configured directly.
  • CF API readiness polling — make login now polls /v2/info for up to 10 minutes before calling cf login, fixing a race where make up returned before the cloud-controller pod was healthy.
  • CI: Podman smoke job — kind-smoke.yaml now dispatches both Docker and Podman smoke runs via a shared reusable workflow (smoke-run.yaml), eliminating ~230 lines of duplication.

ARM / Apple Silicon limitations

The CF application workload layer is amd64-only:

  • cflinuxfs4 rootfs and all pre-compiled buildpack archives are x86-64 images.
  • Docker Desktop on Apple Silicon: enable Rosetta emulation (Settings → General → Use Rosetta for x86_64/amd64 emulation). Well-tested path.
  • Podman on Apple Silicon: Podman machine runs under QEMU/Rosetta. Functional but slower than Docker Desktop with Rosetta.
  • Linux ARM64: not supported — cflinuxfs4 and buildpacks are amd64-only; app staging will fail without kernel-level x86 emulation (binfmt_misc + QEMU).

CF platform components (gorouter, diego, CAPI, etc.) build and run natively on ARM; only app workloads are restricted to amd64.

Rootless Podman limitations

  • No Cilium: CAP_NET_ADMIN in the host user namespace is required by Cilium 1.18.x and is unavailable in rootless containers. kindnet is used instead; Cilium network policies are not enforced.
  • No make build: docker buildx bake is not supported with Podman. Use podman build directly with releases/*/Dockerfile.
  • Linux prerequisites: fs.inotify.max_user_instances=512 and net.ipv4.ip_unprivileged_port_start=80 must be set before make up.

@ZPascal
ZPascal force-pushed the add-podman-support branch 2 times, most recently from e11a2de to 362eb75 Compare May 6, 2026 14:02
@linux-foundation-easycla

linux-foundation-easycla Bot commented May 7, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

@lukaszgryglicki

lukaszgryglicki commented May 7, 2026

Copy link
Copy Markdown

/easycla

@ZPascal
ZPascal force-pushed the add-podman-support branch from 4bc003f to 61afe52 Compare May 10, 2026 15:32
@ZPascal
ZPascal force-pushed the add-podman-support branch 5 times, most recently from feef7d2 to 0648cc2 Compare May 25, 2026 13:38
@ZPascal
ZPascal force-pushed the add-podman-support branch 4 times, most recently from 22c8334 to 162984c Compare May 27, 2026 17:02
@ZPascal ZPascal changed the title Add podman support feat: Add Podman support as drop-in replacement for Docker May 27, 2026
@ZPascal
ZPascal force-pushed the add-podman-support branch 2 times, most recently from 3c80079 to bb577c4 Compare May 27, 2026 21:47
@ZPascal
ZPascal force-pushed the add-podman-support branch 6 times, most recently from fd8cb71 to eb77fe2 Compare June 14, 2026 19:13
@ZPascal
ZPascal marked this pull request as ready for review June 14, 2026 19:30
@ZPascal
ZPascal requested a review from a team as a code owner June 14, 2026 19:30
c0d1ngm0nk3y
c0d1ngm0nk3y reviewed Jun 15, 2026
View reviewed changes
Comment thread Makefile Outdated
@ZPascal
ZPascal force-pushed the add-podman-support branch 2 times, most recently from 41a67f7 to 1041b76 Compare June 15, 2026 09:56
@ZPascal
ZPascal requested a review from c0d1ngm0nk3y June 15, 2026 14:13
@ZPascal
ZPascal force-pushed the add-podman-support branch from 1041b76 to 43c32e8 Compare June 16, 2026 06:46
@ZPascal
ZPascal force-pushed the add-podman-support branch from 43c32e8 to cf675d9 Compare June 16, 2026 06:51
@pbusko

pbusko commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Hi @ZPascal,

The rootless Podman is incompatible with the CNI, therefore it is incompatible with kind-deployment (it breaks policy-agent). However, the support for rootfull Podman as an alternative to Docker is valuable and would be a nice addition to the project.

@ZPascal
ZPascal force-pushed the add-podman-support branch from cf675d9 to c192589 Compare June 24, 2026 09:12
@ZPascal
ZPascal marked this pull request as draft June 24, 2026 13:49
@ZPascal
ZPascal force-pushed the add-podman-support branch from 2eb7302 to abcb0ed Compare July 2, 2026 11:57
@ZPascal

ZPascal commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Hi @ZPascal,

The rootless Podman is incompatible with the CNI, therefore it is incompatible with kind-deployment (it breaks policy-agent). However, the support for rootfull Podman as an alternative to Docker is valuable and would be a nice addition to the project.

I've adapted the PR and removed the rootless support.

@ZPascal
ZPascal marked this pull request as ready for review July 2, 2026 11:57
@ZPascal
ZPascal force-pushed the add-podman-support branch 2 times, most recently from 7d111d9 to d077578 Compare July 15, 2026 05:50
pbusko
pbusko requested changes Jul 16, 2026
View reviewed changes

@pbusko pbusko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately this PR still requires rebasing due to the usage of the releases folder, which no longer exists, and there are still rootless Podman artifacts, as well as Cilium replacement with kindnet

Comment thread .github/workflows/kind-cats.yaml
--privileged \
--pid host \
-v /lib/modules:/lib/modules:ro \
alpine sh -c "sysctl -w fs.inotify.max_user_instances=512 && modprobe nfs && modprobe nfsd"

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still needed for rootful Podman?

@ZPascal ZPascal Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed. This CI job runs on Docker (ubuntu-latest) and is unchanged from before this PR. The step loads the NFS kernel modules and raises fs.inotify.max_user_instances which Docker requires for the test run.

Comment thread assets/base-chart/values.yaml Outdated
Comment on lines +5 to +8
# Set to false when Cilium is not installed (e.g. rootless Podman CI).
# CiliumClusterwideNetworkPolicy CRDs won't exist without Cilium.
ciliumNetworkPolicies:
enabled: true

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as it was discussed, Cilium replacement is not possible at this stage of the project

@ZPascal ZPascal Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Removed all SKIP_CILIUM conditionality. Podman now uses Cilium via CILIUM_EXTRA_VALUES pointing to assets/values/cilium-podman.yaml (reduced-privilege Cilium config: no eBPF kube-proxy replacement, VXLAN tunnel, BPF masquerade disabled). kind-podman.yaml updated to disableDefaultCNI: true + kubeProxyMode: none so Cilium manages networking. CiliumClusterwideNetworkPolicy resources are always deployed.

Comment thread assets/base-chart/templates/network-policy.yaml Outdated
@@ -1,3 +1,4 @@
{{- if .Values.ciliumNetworkPolicies.enabled }}

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/cloudfoundry/kind-deployment/pull/360/changes#r3593341067

Comment thread assets/values/cilium-podman.yaml Outdated
@@ -0,0 +1,68 @@
# This file is NOT used by default. When SKIP_CILIUM=true (the default under

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/cloudfoundry/kind-deployment/pull/360/changes#r3593341067

Comment thread releases/nfs-volume/helm/templates/nfsv3driver.yaml Outdated
Comment thread helmfile.yaml.gotmpl Outdated
- name: tlsKey
file: {{ .Values.certsDir }}/all-in-one.key
- name: ciliumNetworkPolicies.enabled
value: {{ ne (env "SKIP_CILIUM") "true" }}

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/cloudfoundry/kind-deployment/pull/360/changes#r3593341067

Comment thread kind-podman.yaml Outdated
# at startup which requires CAP_NET_ADMIN in the *host* (initial) user namespace –
# a privilege that rootless Podman containers structurally cannot have.
# kindnet provides full pod-to-pod connectivity without BPF privileges.
disableDefaultCNI: false

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/cloudfoundry/kind-deployment/pull/360/changes#r3593341067

Comment thread Makefile Outdated
install:
@ ./scripts/install.sh
@ . ./scripts/detect-runtime.sh; \
if [ "$$IS_PODMAN" = "true" ]; then export SKIP_CILIUM="true"; fi; \

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/cloudfoundry/kind-deployment/pull/360/changes#r3593341067

Comment thread README.md Outdated
Comment on lines +58 to +60
### Linux (rootless Podman)

Rootless Podman is fully supported on Linux. The following kernel settings must be applied before running `make up` — either manually or via your system's sysctl configuration:

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rootless Podman is incompatible

@ZPascal ZPascal Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've adjusted it and the README now only documents rootful Podman.

Comment thread README.md Outdated
Comment on lines +73 to +76
### Limitations

- **CNI:** Cilium is skipped under rootless Podman (Linux CI / rootless desktop) because Cilium 1.18.x requires `CAP_NET_ADMIN` in the host user namespace, which rootless containers cannot provide. [kindnet](https://github.com/aojea/kindnet) is used instead, providing full pod-to-pod connectivity without eBPF privileges. Cilium network policies are therefore not enforced in this mode.
- **Image builds:** `make build` (which uses `docker buildx bake`) is not supported with Podman. Use `podman build` directly with the Dockerfiles in `releases/` for local image development.

@pbusko pbusko Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/cloudfoundry/kind-deployment/pull/360/changes#r3593384975

ZPascal added 2 commits August 18, 2026 10:39
@ZPascal
feat: Add Podman support as drop-in replacement for Docker
597acdb 
Signed-off-by: Pascal Zimmermann <pascal.zimmermann01@sap.com>
@ZPascal
feat: Remove the Podman rootless mode
f45c240 
Signed-off-by: ZPascal <pascal.zimmermann01@sap.com>
@ZPascal
ZPascal force-pushed the add-podman-support branch from d077578 to f45c240 Compare August 18, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pbusko pbusko pbusko requested changes
@c0d1ngm0nk3y c0d1ngm0nk3y Awaiting requested review from c0d1ngm0nk3y

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ZPascal @lukaszgryglicki @pbusko @c0d1ngm0nk3y