chapter zero

Kubernetes The Hard Way starts at lab 01. This is chapter zero.

Hightower's tutorial opens by asking for four Debian 12 machines and then says nothing more about where they come from. This is that part: a fast cloud-image VM workflow for local virsh/libvirt, and the k8s lab stands built on top of it.

1. Keep one immutable cloud image per distro in images/.
2. Start each VM from a small qcow2 overlay.
3. Inject the host user's SSH public key through cloud-init.

This avoids reinstalling Debian and avoids cloning identical SSH host keys into every VM.

Requirements

Everything runs against the host's system libvirt (qemu:///system), so this is a Linux host with KVM. You need these commands on the host:

Command	Debian/Ubuntu package	Arch package	Used for
libvirtd + KVM	libvirt-daemon-system qemu-system-x86	libvirt qemu-desktop	the hypervisor
virsh	libvirt-clients	libvirt	domains, networks, DHCP leases
virt-install	virtinst	virt-install	creating domains
qemu-img	qemu-utils	qemu-img	qcow2 overlay disks
xorriso	xorriso	libisoburn	the NoCloud seed ISO
curl or wget	curl / wget	curl / wget	fetch-base downloads
ssh	openssh-client	openssh	logging into guests
osinfo-query (optional)	libosinfo-bin	libosinfo	resolve the osinfo id; degrades gracefully if absent
dhcp_release (optional)	dnsmasq-base	dnsmasq	drop DHCP leases on teardown, so an address can be reused straight away

Then make sure the daemon is running and the default network is up:

sudo systemctl enable --now libvirtd
sudo virsh net-start default && sudo virsh net-autostart default

Add your user to the libvirt (and kvm) group, or run as root, so the scripts can reach qemu:///system. Guests install qemu-guest-agent themselves through cloud-init — vm-ip asks it for the guest's address.

First run

cd ~/devel/chapter-zero
bin/fetch-base debian
bin/vm-up cka1
bin/vm-ip cka1
ssh debian@$(bin/vm-ip cka1)

Commands

bin/fetch-base [DISTRO]
bin/vm-up NAME [DISK_SIZE]
bin/vm-ip NAME
bin/vm-console NAME
bin/vm-rm NAME

Layout

lib/distros.sh    cloud image registry: image, user, admin group, seed, video
lib/net.sh        dhcp-host reservations — static IPs on a libvirt network
lib/vm.sh         vm_create(): validation, seed ISO, overlay, virt-install
profiles/         one file per cloud-config flavour
bin/              thin wrappers that pick a profile and call vm_create

Every VM takes the same path through vm_create; a profile only supplies the cloud-config. That is what keeps the seed-device and video quirks below in one place instead of drifting between scripts.

LIBVIRT_DEFAULT_URI defaults to qemu:///system inside the scripts. Without it a non-interactive ssh host 'bin/vm-ip x' reads the shell's environment, not the profile that usually exports it, and answers from qemu:///session — which reports an empty host with no VMs on it.

Distros

VM_DISTRO picks the base image, the login account and the osinfo id together. Defaults to debian, so existing usage is unchanged.

VM_DISTRO	Image	User	Admin group	Seed	Video	Min RAM
debian (default)	Debian 12 genericcloud	debian	sudo	virtio disk	none	1024
ubuntu	Ubuntu 24.04 LTS cloudimg	ubuntu	sudo	virtio disk	none	1024
rocky	Rocky Linux 10 GenericCloud	rocky	wheel	cdrom	vga	2048

bin/fetch-base rocky
VM_DISTRO=rocky bin/vm-up rocky1
ssh rocky@$(bin/vm-ip rocky1)

The registry lives in lib/distros.sh. Point a distro at another release with DEBIAN_CLOUD_URL / UBUNTU_CLOUD_URL / ROCKY_CLOUD_URL — note that the local image filename comes from the registry, so change both together when switching releases.

The osinfo id is resolved against the host osinfo-db and falls back to the closest known id, because an unknown id aborts virt-install and the db lags new releases.

The last three columns are not cosmetic — each is a distro that refuses to boot otherwise:

• Seed device. Debian and Ubuntu genericcloud kernels have no cdrom driver, so their seed goes on a virtio disk. Rocky documents a cdrom and its kernel has the driver.
• Video. Rocky's GRUB hangs before it emits a single character when the guest has no video device, which is exactly what --graphics none produces. Symptom: a VM burning 100% of a core, empty serial console, no DHCP lease. Debian and Ubuntu boot fine headless.
• Admin group. RHEL rebuilds use wheel; cloud-init fails to create the user at all if the group does not exist, leaving a VM you cannot log into.

CPU model

VM_CPU defaults to host-passthrough. The previous --cpu host let libvirt pick a "safe" custom model, which on this host resolved to Denverton — x86-64-v2, no AVX2. Debian and Ubuntu do not care; Rocky 10 and RHEL 10 require x86-64-v3 and will not run on it. Passthrough also matches what virt-manager creates by default.

Defaults:

• distro: debian (seed ISO is attached as a virtio disk: the genericcloud cloud kernel has no cdrom driver, so a cdrom seed is invisible to cloud-init and the VM never gets an IP)
• image and user: from the distro registry, see Distros
• SSH key: ~/.ssh/id_ed25519.pub
• network: libvirt default
• RAM: 2048 MiB
• vCPU: 2
• disk overlay size: 20G

Override with environment variables:

VM_MEMORY=4096 VM_VCPUS=4 VM_NET=default VM_SSH_KEY=~/.ssh/id_rsa.pub bin/vm-up node1 30G
VM_DISTRO=ubuntu bin/vm-up node2

vm_create aborts when host MemAvailable is below the VM's RAM plus 512 MiB; VM_FORCE=1 overrides.

Profiles

VM_PROFILE picks the cloud-config. A profile is a file in profiles/ defining profile_user_data(), which prints cloud-config on stdout and may read $vm_hostname, $vm_user, $vm_admin_group, $vm_ssh_key_line and $vm_ip. Setting PROFILE_PKG_FAMILY makes it refuse distros it cannot provision, up front rather than halfway through cloud-init.

VM_PROFILE	What the VM boots with
base (default)	admin user + SSH key + guest agent
k8s-node	base plus containerd, k8s sysctls, kubelet/kubeadm/kubectl (apt only)
kthw	bare Debian for Kubernetes The Hard Way — see below (apt only)

Hostnames

VM_HOSTNAME sets the guest hostname independently of the libvirt domain name, which defaults to it. Domain names are global to the host, so a tutorial that wants to talk to a machine called server cannot own that name outright:

VM_HOSTNAME=server bin/vm-up kthw-server    # domain kthw-server, guest server

The hostname is validated as RFC 1123 ([a-z0-9-], no underscores).

k8s stand

vm-up stays generic; k8s-specific provisioning lives in separate scripts. The node profile installs containerd and kubeadm from apt, so it accepts debian and ubuntu and refuses other distros up front.

bin/k8s-cluster-up NAME [WORKERS]    # cp + N workers (default 2), static IPs, ansible inventory
bin/vm-up-k8s-node NAME IP [DISK]    # single k8s-ready node with a static IP
bin/k8s-cluster-rm NAME

• Static IPs: dnsmasq dhcp-host reservations in the libvirt network (MAC is derived from the last IP octet). vm-rm removes the reservation with the VM.
• Node profile (cloud-init): containerd with SystemdCgroup, overlay + br_netfilter, k8s sysctls, swap off, kubelet/kubeadm/kubectl from pkgs.k8s.io, version-pinned and held. kubeadm init/join is left to ansible or manual — inventory is written to clusters/NAME/inventory.ini.
• Defaults: cp1 = .101, workers = .111+. Env knobs: K8S_VERSION (1.35), K8S_SUBNET_PREFIX (192.168.200), K8S_CP_OCTET, K8S_W_OCTET_BASE.
• RAM: scripts abort when host MemAvailable is too low (VM_FORCE=1 overrides). On a 16 GiB host cp + 2 workers @ 2048 MiB ≈ 6 GiB — stop other big VMs first.

Kubernetes The Hard Way stand

bin/kthw-stand-up [NAME]    # jumpbox + server + 2 nodes, KTHW lab 01 spec
bin/kthw-stand-rm [NAME]

Kelsey Hightower's tutorial opens by asking for four Debian 12 machines and then says nothing more about where they come from. This is that part, and only that part.

Machine	Guest hostname	IP	RAM	Disk
NAME-jumpbox	jumpbox	.120	512	10G
NAME-server	server	.121	2048	20G
NAME-node-0	node-0	.122	2048	20G
NAME-node-1	node-1	.123	2048	20G

The table is the tutorial's spec, and it is what the stand defaults to. On a tight host it goes lower: 768 MiB and a 10G disk carry server and the nodes through all thirteen labs — measured by walking the tutorial end to end on exactly that. 512 MiB does not. That puts the whole stand at 2.75 GiB against 6.5 at the defaults:

KTHW_NODE_MEMORY=768 KTHW_NODE_DISK=10G bin/kthw-stand-up

lib/distros.sh sets DISTRO_MIN_MEMORY=1024 for Debian, so each machine prints a Warning: ... wants at least 1024 MiB. It is a warning, not a refusal — expected here, and the stand comes up anyway.

NAME defaults to kthw and prefixes the libvirt domain names only — inside, the machines are called what the tutorial calls them. The addresses avoid the kubeadm stand's .101/.111+, and pod subnets 10.200.N.0/24 do not collide with 192.168.200.0/24. Knobs: KTHW_NODES, KTHW_DOMAIN, KTHW_SUBNET_PREFIX, KTHW_POD_SUBNET_PREFIX, KTHW_JUMPBOX_OCTET, KTHW_SERVER_OCTET, KTHW_NODE_OCTET_BASE, KTHW_*_MEMORY, KTHW_*_DISK, KTHW_VCPUS.

The script prints machines.txt in lab 03's format and writes it to clusters/NAME/machines.txt. It does not copy it to the jumpbox: that is a lab 03 step, and so are root SSH, the FQDNs and the shared /etc/hosts. The stand hands over machines built to lab 01 and stops.

Two things the kthw profile does do, because neither is an exercise:

• manage_etc_hosts: false. cloud-init rewrites /etc/hosts from a template on every boot. Lab 03 has you put an entry per machine in that file; under the base profile they survive until the first reboot and then vanish, and the cluster comes apart by name with nothing obvious to blame. The jumpbox gets the same treatment — lab 03 writes to its /etc/hosts too.
• net.ipv4.ip_forward=1. Lab 11 routes pod subnets between nodes. br_netfilter and the bridge-nf sysctls are kubeadm's requirements, not this tutorial's, so the profile leaves them out.

KTHW_SEED_ROOT=1 seeds root SSH for anyone skipping the first step of lab 03. It writes PermitRootLogin prohibit-password as an sshd_config.d drop-in rather than the tutorial's yes — login is by key either way.