docs: Kubernetes quota-enforcement design (RFC) - #444
Open
powderluv wants to merge 8 commits into
Open
Conversation
…ative CNI Adds SPUR-native Kubernetes support: SPUR provisions and owns a Kubernetes (k0s) cluster across its nodes. `spur k8s up` assigns roles + mesh IPs + pod CIDRs, installs a pinned k0s on each bare-metal node, brings up the control plane, mints join tokens, joins the workers, and (for a native-routing CNI) generates the k0s config + programs the WireGuard full mesh — all driven by the existing spurctld/spurd control plane. This is the inverse of `[kubernetes]` (SPUR running *inside* an existing k8s): here SPUR is the cluster's lifecycle owner. k0s runs under a spurd-owned systemd unit (never a job), so it survives spurd restarts. Core pieces: - spur-core: `[cluster]` / `[cluster.arc]` config; shared k0s types (role, phase, join-token, cluster state); the pinned k0s version + `k0s_controller_config_yaml` generator. - spur-proto: SlurmController cluster lifecycle RPCs (Up/Down/Status/Kubeconfig) and SlurmAgent component RPCs (Start/Stop/GetStatus, CreateK0sJoinToken, GetAdminKubeconfig, ApplyMesh). - spurctld: leader-gated k0s reconcile loop — AddressPool IPAM (mesh IP + pod /24 per node), deterministic control-plane selection, agent fan-out, and durable role/token/CIDR/phase state in the raft state machine (survives failover). - spurd: RPC-driven `K0sAgent` that owns the k0s systemd unit (write token/config, render/reconcile/heal the unit, re-adopt a running unit on restart), installs a pinned k0s if missing, mints tokens / reads the admin kubeconfig on the control plane, applies the WireGuard mesh, and writes the GPU CDI spec on join. - spur-cli: `spur k8s up|down|status|kubeconfig|install-k0s` + `spur net mesh`. Highlights: - k0s packaging: downloads a pinned (or `--latest`) k0s release, SHA-256 verified, atomic install — bare-metal nodes need no manual k0s prestaging. - Mesh-native CNI: `[cluster] cni = "calico"` generates a k0s config advertising the API on the control-plane mesh IP with Calico in bird mode, and sets each worker's kubelet `--node-ip` to its mesh IP, so pods route over the WireGuard mesh (native routing, no overlay). Default `kuberouter` is unchanged. - Mesh reconcile: the controller pushes the authoritative full-mesh membership (ApplyMesh), level-triggered so node-local drift, failed pushes, and failover self-heal; the agent prunes departed peers. No hardcoded infrastructure IPs (test/example addresses use RFC 5737 TEST-NET). Full workspace tests pass. HW-validated on a 5-node cluster: `spur k8s up` → 4 workers Ready on the pinned k0s; token-mint join; ApplyMesh reconcile; Calico bird BGP full-mesh established over the WireGuard mesh; `spur k8s down --reset` tears every node down (k0s reset, mesh preserved); full up→down→up lifecycle. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- write_secret_file: create the join-token file 0600 from the start (was write-then-chmod, leaving a brief umask-dependent world-readable window). - K0sAgent::start: propagate create_dir_all errors instead of ignoring them. - ClusterSupervisor is now Clone; K0sAgent::stop/status/supervise snapshot the supervisor out from under the `active` mutex and do the (seconds-long) systemctl /k0s IO WITHOUT holding the lock, so they no longer serialize concurrent RPCs. - build_mesh_membership: sort members numerically by IPv4 (a string sort put ".10" before ".2", causing spurious per-tick membership diffs). - live_node_statuses: report the agent's real `enabled` bit, not a hard-coded true. - spawn_apply_mesh: bound connect + RPC with AGENT_TIMEOUT so a hung agent can't leak accumulating detached tasks (it fires every reconcile tick). - cluster_up: `--install-arc` is not implemented — warn + say so in the response instead of silently ignoring it. - is_valid_cidr: IPv4-only (the provisioning paths are all Ipv4Addr; an IPv6 CIDR would validate here and then fail at runtime). - mesh endpoint: don't derive the WireGuard endpoint from Node.address (which is the mesh IP when WG is up, not the underlay) — leave it empty so apply_mesh preserves the tunnel `spur net join` established; mesh_peers_for omits an empty endpoint from `wg set`. Full workspace tests pass. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Admin + user guide for the SPUR-owned k0s cluster: configuring `[cluster]`, installing k0s, `spur k8s up/down/status/kubeconfig/install-k0s`, the kuberouter-vs-calico CNI choice, teardown, and (for users) getting a kubeconfig + running GPU workloads. Added to the deployment toctree. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…orrectness gaps Resolves the review on #432: remove config/state that reads as load-bearing but has no consumer, and fix several correctness gaps in the k0s lifecycle. Drops: - Join-token metadata raft state (K0sJoinTokenRecord, K0sClusterState.join_tokens, the K0sJoinTokenCreate/Revoke WAL variants + apply arms, and the create/revoke mutators). The real join path (agent CreateK0sJoinToken -> plaintext token) is unchanged; the metadata layer returns with a real consumer. - [cluster.arc] config block, --install-arc CLI flag, and the proto install_arc field (reserved 2). ARC is not installed by spur/k0s. - Unread config fields flannel_iface, gpu_worker_selector, device_plugin (the worker --node-ip already pins the mesh IP). Fixes: - Wire cni_mtu into the generated Calico config (spec.network.calico.mtu); it was dead and left pods at MTU 1500 over the ~1450 mesh (fragmentation). - Refresh the WireGuard mesh key on every heartbeat (HeartbeatRequest.wg_pubkey) so a key that appears/changes after registration reaches the controller and the node joins ApplyMesh without a spurd restart. - Cache the minted worker join token per node so it is minted once, not every reconcile tick (token churn / join race). - Surface k0s reset failures: k0s_reset returns Result through stop/stop_untracked, the agent reports stopped=false + message, and the controller inspects that in-band response so `down --reset` no longer reports a false success. - Reject a control-plane change once roles are assigned (cluster_up returns FailedPrecondition) instead of leaving an inconsistent controller/worker split. - Give the k0s unit a finite StartLimitIntervalSec/StartLimitBurst so a hard crash-loop backs off; the reconcile supervisor still re-enables it. Verified: cargo check --workspace + full test suite green; deployed and validated the up -> down --reset -> up lifecycle on a 5-node cluster (mtu emitted, CP change rejected, StartLimitBurst set, mesh reformed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ass)
k0s bundles no storage, so a plain SPUR-managed cluster has no StorageClass and
any PersistentVolumeClaim stays Pending. Ship local-path-provisioner as the
cluster's default StorageClass so PVC workloads bind out of the box.
- New [cluster] storage_provisioner ("local-path" default, or "none") +
local_path_dir (the on-node PV directory; point at a big disk for data-heavy
PVCs). local_path_dir is validated (absolute, no quotes/backslashes/
whitespace) since it is interpolated verbatim into the ConfigMap JSON.
- Vendor the local-path v0.0.31 manifest (templated data dir + default-class
annotation) and add k0s_local_path_manifest() to render it.
- The control-plane spurd agent writes the rendered manifest into k0s's
manifest-deployer directory on component start; k0s applies + reconciles it,
so there is no in-cluster kube client.
Verified: unit tests + a 5-node cluster (down/up regenerates the manifest with
the configured path, k0s applies it, the default StorageClass appears).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A spurd restart deregisters on SIGTERM, which removes the node (and its k0s role / mesh IP / pod CIDR) from the controller; re-registration then creates a fresh, un-roled node. The reconcile loop only ran provisioning in the Provisioning phase, so once the cluster was Ready a re-added node was never re-assigned and stayed out of the WireGuard mesh until a manual `spur k8s up`. Run the same assignment + converge reconcile in the Ready phase as in Provisioning (extracted into a testable reconcile_phase). It is idempotent — assigned + active nodes are skipped — so a converged cluster does no extra work beyond the per-node status probes, and a re-added node self-heals into the cluster + mesh on the next tick. Guard converge's set_k0s_phase(Ready) with a phase check so it does not churn a WAL write + log line every tick while Ready. Verified: ready_phase_reconcile_assigns_unroled_node (fails if the Ready-phase change is reverted) + provision_reassigns_a_readded_node; and on a 5-node cluster a worker whose spurd is stopped + recreated rejoins in ~2 ticks with no `spur k8s up`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI builds with -D warnings; the new provision_reassigns_a_readded_node test imported K0sRole but only compares Option<K0sRole> values, so the unused import failed the build/clippy/test jobs (a plain `cargo test` only warns). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A design/RFC for enforcing SPUR's resource model (accounts, QoS, fair-share, partitions, GPU limits) on a SPUR-managed k0s cluster while the cluster stays a normal Kubernetes deployment to users. Covers the reconciler architecture, the SPUR->native-k8s concept mapping, a layered plan + milestones, a pluggable Kueue/native fairness backend, the M4 mapping grounded in the actual scheduler code, alternatives considered, and open questions. Not implemented yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
powderluv
requested review from
sajmera-pensando,
sgopinath1,
shiv-tyagi and
yansun1996
as code owners
July 14, 2026 16:30
powderluv
force-pushed
the
users/powderluv/m8-native-k8s
branch
2 times, most recently
from
July 17, 2026 13:32
308fd0b to
f2d7b3b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design / RFC for the quota-enforcement layer on top of the SPUR-managed k0s cluster (#432): make SPUR enforce its resource model (accounts, QoS, fair-share, partitions, GPU limits) on k8s workloads while the cluster stays a normal Kubernetes deployment to users — vanilla
kubectl, no proprietary API in the pod-creation hot path.Docs-only: adds
docs/developer/k8s-quota-enforcement.rst. Stacked on #432 (Layer 0 = the SPUR-managed cluster this builds on). Opening for review before we implement M1.Covers
FairnessBackend— Kueue (default/recommended) + a nativeschedulingGatesfallback, with tests for both.schedulingGates.Decisions (locked)
Related
The M4 analysis surfaced three SPUR-side scheduler cleanups worth doing regardless of k8s, already tracked: #439 (GPU not enforced / not tracked for fair-share), #440 (base-0 priority collapse), #441 (dead QoS/account config fields).
🤖 Generated with Claude Code