feat(spurctld): scope k0s cluster to a subset of nodes

[yansun1996]

Summary

spur k8s up provisioned the managed k0s cluster over the controller's entire registered node inventory — the reconcile loop assigned one control plane plus a worker role to every other agent node, with no way to target a subset. On a large shared scheduler that means k8s up tries to build a cluster spanning all nodes, and there was no clean way to keep a node out (setting a node's local [cluster].enabled=false didn't help: the controller still assigned it a role, the node refused to start k0s, and the cluster stalled in Provisioning).

This adds node-selection scoping to spur k8s up:

• --nodes <hostlist> (e.g. gpu[01-08])
• --partition <name>
• --selector key=val (repeatable; a node matches when all pairs match its labels)

The three are combined as a union and resolved (fail-closed) to a concrete member set that is persisted in the Raft-replicated cluster state. The reconcile loop assigns k0s roles only to member nodes; un-scoped nodes get no role and stay schedulable for the Spur batch scheduler. That also gives a clean per-node opt-out — a node simply left out of the scope is never claimed, so it can't stall bring-up.

Empty selection enrolls the whole inventory, so existing single-cluster usage is unchanged.

Approach

• Scope resolution is a pure, unit-tested function (union of hostlist / partition members / label selector), fail-closed: every named or expanded node must be registered, and an empty match is rejected.
• The resolved member set is persisted in the k0s cluster state and consulted by the reconcile loop's node filter — the single chokepoint that already drives role assignment — so scoping is enforced consistently on every tick, including after a controller restart.
• Control-plane candidates are drawn only from the in-scope members, so a --control-plane-node(s) outside the selected scope is rejected rather than silently relocated.
• On an already-assigned cluster, changing the scope is rejected (tear down with --reset first), mirroring the existing control-plane-change guard. k8s down --reset clears the scope; a bare re-up preserves it.
• spur k8s status now shows the member scope.

Persistence / compatibility

• The new member-scope field on the persisted cluster state and its WAL operation are #[serde(default)], and a frozen pre-feature payload test guards replay so an upgraded controller cannot crash on an old Raft log entry.
• The new proto fields are append-only (no renumbering); empty values preserve prior behavior.

Known limitations / follow-ups

• The scope is resolved once at k8s up time into a fixed member set; it is not re-evaluated as inventory changes (a node added later that would match the selector does not auto-join, and a member that deregisters is simply skipped). Changing the membership requires k8s down --reset then up. This is intentional (membership is frozen for the cluster's lifetime), and documented in the CLI/proto help.
• Two concurrent k8s up calls on a fresh cluster can race (last-writer-wins on the recorded scope/control-plane set). This is a pre-existing property of the handler (it already applied to the control-plane set) and is unchanged by this PR; a follow-up can serialize the handler.

Testing

• Unit tests: union resolution (hostlist expand/sort, partition membership, selector AND, cross-surface union/dedup) and its fail-closed rejections; control-plane-outside-scope rejection; membership is_member semantics; the reconcile filter (out-of-scope node stays un-roled); WAL round-trip + frozen pre-feature replay fixture; handler-level tests for scoped up, CP-outside-scope rejection, bare-re-up scope preservation, and scope-change rejection; CLI flag parsing. Behavioral tests were confirmed to fail against the unfixed code.
• Validated end-to-end on an isolated 2-node deployment: --nodes, --partition, and --selector each scoped membership correctly; a job targeting a scoped (k8s-reserved) node pended with the reserved reason while a job on an out-of-scope node ran to completion; control-plane-outside-scope, unknown-node, and scope-change-on-assigned-cluster were all rejected with the expected errors; down --reset cleared the scope and a bare re-up preserved it.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.93671% with 24 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #577      +/-   ##
==========================================
+ Coverage   75.83%   75.96%   +0.13%     
==========================================
  Files         166      166              
  Lines       64696    65158     +462     
==========================================
+ Hits        49058    49496     +438     
- Misses      15638    15662      +24     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds first-class node scoping for the managed k0s cluster lifecycle (spur k8s up) so that provisioning/role assignment is limited to an explicitly selected subset of the registered inventory, while persisting that resolved membership in Raft state for consistent reconciliation across restarts.

Changes:

• Add --nodes, --partition, and repeatable --selector key=val to spur k8s up, and expose resolved membership via spur k8s status.
• Persist member_nodes in the replicated k0s cluster state/WAL and filter role assignment to in-scope nodes.
• Extend proto surface area and add unit + e2e tests covering scoping, rejection cases, and backward-compatible WAL replay.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/native_host/e2e/test_k8s_scheduling.py	Adds an e2e assertion that k8s up --nodes <node> records and reports a single-node membership scope.
tests/native_host/e2e/cluster.py	Adds a helper to parse the members: line from spur k8s status.
proto/slurm.proto	Appends request fields for scoping inputs and a status field for the resolved member set.
crates/spurctld/src/server.rs	Resolves/records membership scope on cluster_up, enforces scope immutability once assigned, and returns member nodes in status.
crates/spurctld/src/cluster.rs	Extends set_k0s_phase to include persisted member_nodes and clears scope on reset.
crates/spurctld/src/cluster_k8s.rs	Filters provisioning assignments by state.is_member, adds scope resolution, and enforces pinned-CP validity within candidate scope.
crates/spur-core/src/wal.rs	Adds member_nodes to the WAL op with #[serde(default)] and includes a frozen pre-feature payload replay test.
crates/spur-core/src/k0s.rs	Adds persisted member_nodes plus is_member() semantics (empty = whole inventory).
crates/spur-cli/src/k8s.rs	Adds CLI flags, selector parsing, and prints membership scope in k8s status.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

crates/spurctld/src/server.rs:2237

• The assigned-cluster guard compares member_nodes vectors directly, but an empty member_nodes is a sentinel meaning “whole inventory”. That makes a semantically no-op request like --nodes <all nodes> (or a selector/partition that expands to all nodes) look different from the recorded empty scope and get rejected as a scope change.

            if scope_requested && member_nodes != state.member_nodes {
                return Err(Status::failed_precondition(
                    "cluster membership is already assigned; tear the cluster down \
                     (spur k8s down --reset) before changing the node scope",
                ));

crates/spurctld/src/cluster_k8s.rs:132

• provision_assignments filters nodes with state.is_member, which does a linear scan of member_nodes for every node. For large clusters or large scopes this becomes O(N*M) work per reconcile tick. Since member_nodes is a fixed list, build a HashSet once and do O(1) membership checks (and skip filtering entirely when the scope is empty = whole inventory).

    let mut nodes = cluster.get_nodes();
    nodes.retain(|n| state.is_member(&n.name));