feat(quota): Kubernetes quota-enforcement — M1 (tenancy + hard quotas + scoped kubeconfig)

[yansun1996]

What

Implements M1 of the Kubernetes quota-enforcement design (#444 RFC): the tenancy + hard-quota layer that projects SPUR accounts onto native Kubernetes objects, plus per-user scoped kubeconfigs — so a cluster looks like a normal Kubernetes deployment to its users while SPUR is the invisible policy plane.

Picks up the work from #471 (which was stacked on #432). Rebased cleanly onto main — the diff here is exactly the 4 quota commits, no #432 dependency, plus one review-fix commit (below).

End-to-end flow

sacctmgr add account name=physics grptres=cpu=64,mem=131072,gres/gpu=8
  → the operator's quota reconciler (opt-in --enable-quota) projects it to:
      namespace spur-acct-physics
      + ResourceQuota (requests.cpu, requests.memory, requests.amd.com/gpu)
      + LimitRange (default requests so unset-request pods still count)
      + Role + RoleBinding (the account's members)

spur k8s kubeconfig --user alice
  → spurctld resolves alice → physics → its namespace; the control-plane agent
    mints a ServiceAccount + bound token there → a namespace-scoped kubeconfig
    (no cluster-admin credential ever handed out)

Increments (each independently reviewable)

Commit	Increment
projection module	Pure account → Namespace/ResourceQuota/LimitRange/RBAC mapping (spur-k8s/quota.rs), unit-tested
persist grp_tres	Account-level TRES allocation through DB (column + migration) + proto + gRPC + sacctmgr
reconciler	quota_controller in the operator: lists accounts+members over gRPC, server-side-applies with force (drift-correcting), opt-in --enable-quota
kubeconfig --user	Shared naming in spur-core, scoped kubeconfig minted by the control-plane agent, CLI flag
review fixes	mem units, DNS name cap, fail-closed paths (see below)

Design decision

Per the RFC, SPUR persists resource caps on QoS, not accounts. M1 wires account-level allocations (grp_tres on the Account) so each account/namespace has a definite ResourceQuota — the cleanest tenancy model, and it gives the GPU cluster the GPU quota it lacks today.

Review fixes applied

A round of review surfaced several correctness/safety issues, all fixed here:

• Memory units: TRES mem is base-10 MB, but the ResourceQuota emitted Mi (mebibytes) → ~4.86% over-allocation. Now emits M.
• DNS name length: the 63-char cap was applied to the label before the spur-acct-/spur-user- prefix, so long names produced invalid (>63-char) namespace/SA names. The cap now includes the prefix.
• Fail closed on malformed allocation: a non-empty but unparseable grp_tres used to silently leave the namespace uncapped; the reconciler now skips such an account.
• Fail closed on cold cache: the scoped-kubeconfig path now refuses when associations aren't loaded (the association cache resolves fail-open, which could have minted an unscoped token).
• LimitRange: dropped the default limit (kept the default request) — a forced small limit would reject ordinary pods that omit limits; the ResourceQuota is what bounds usage.
• ResourceQuota caps requests only: dropped the limits.cpu/limits.memory hard keys — with no default limit those keys would reject any pod that omits limits at admission. The cap is enforced on requests.* (matching the requests-only LimitRange).
• Account grp_tres validated on create: create_account now runs validate_tres (as add/modify user already did), so an invalid allocation is rejected with invalid_argument instead of being stored and later skipped by the reconciler.
• Reconciler user lookup batched: replaced the per-account ListUsers (N+1) with a single ListUsers grouped client-side by account.

Known limitations (M1 scope)

• No GC of deleted accounts: deleting a SPUR account leaves its namespace/quota/RBAC in place (objects are labelled app.kubernetes.io/managed-by=spur-quota, so a prune pass is a natural follow-up).
• Token lifetime / revocation: scoped tokens are long-lived and not revoked on member removal (RBAC subjects do shrink on the next reconcile). Shortening TTL + SA-deletion on removal is follow-up work.
• Name collisions: the DNS sanitizer is not injective (e.g. physics_lab and physics.lab collapse to the same namespace). Documented with a test; a hash-suffix disambiguation is a follow-up.
• sacctmgr modify account is a full-resend upsert, so it clears fields not restated (a pre-existing behavior across all account fields, not specific to grp_tres). Out of scope here; worth a dedicated fix.
• The whole client gRPC surface is unauthenticated today; this PR adds an RPC that mints bearer tokens, so caller-identity enforcement is important future work.

Testing

cargo build / clippy -D warnings / fmt green across the workspace (excl. spur-ffi). spur-k8s (146) + spur-core (339) unit tests pass, including the new mem-unit, prefixed-name-length, sanitizer-collision, and malformed-grp_tres tests. DB round-trip test is #[ignore] (runs in CI's Postgres).

Also validated end-to-end on an isolated 2-node deployment: sacctmgr add account ... grptres=... persists through gRPC → Postgres and reads back; the grp_tres migration applies cleanly (ADD COLUMN IF NOT EXISTS); and k8s kubeconfig --user fails closed for unknown users / unresolved accounts and never falls back to the admin kubeconfig. (Cluster-side SA/token minting still needs a live k0s control plane — that path is integration-level.)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.47934% with 227 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #475      +/-   ##
==========================================
- Coverage   72.94%   72.83%   -0.11%     
==========================================
  Files         157      160       +3     
  Lines       53701    54299     +598     
==========================================
+ Hits        39171    39547     +376     
- Misses      14530    14752     +222     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR implements M1 of Kubernetes quota-enforcement in Spur by persisting account-level resource allocations (grp_tres), projecting accounts into Kubernetes tenancy/quota/RBAC objects via an operator reconciler, and adding a kubeconfig --user flow that mints namespace-scoped ServiceAccount tokens via the control-plane agent.

Changes:

• Extend proto + controller/agent RPCs to support per-user (scoped) kubeconfig minting, while keeping the admin kubeconfig path.
• Persist grp_tres on accounts end-to-end (CLI → gRPC → DB → list/read back) to drive per-account ResourceQuota projection.
• Add quota projection + reconciler logic in spur-k8s, and shared DNS-safe naming in spur-core so reconciler and kubeconfig minting agree.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
proto/slurm.proto	Adds request fields for scoped kubeconfig minting and grp_tres on accounts.
crates/spurd/src/cluster.rs	Implements SA creation + token mint + kubeconfig templating on control-plane node.
crates/spurd/src/agent_server.rs	Extends get_admin_kubeconfig RPC to optionally mint scoped kubeconfigs.
crates/spurctld/src/server.rs	Adds ClusterKubeconfigRequest.user handling and fail-closed association-cache gating.
crates/spurctld/src/cluster_k8s.rs	Adds client helper to request scoped kubeconfig from control-plane agent.
crates/spurctld/src/accounting/grpc.rs	Wires grp_tres through CreateAccount/ListAccounts gRPC.
crates/spurctld/src/accounting/db.rs	Adds grp_tres column + upsert/list plumbing and a DB round-trip test.
crates/spur-k8s/src/quota.rs	Pure mapping from account allocation/users → Namespace/ResourceQuota/LimitRange/RBAC.
crates/spur-k8s/src/quota_controller.rs	Implements reconciler loop + SSA apply with force and fail-closed grp_tres parsing.
crates/spur-k8s/src/main.rs	Adds --enable-quota flag and spawns quota reconciler when enabled.
crates/spur-k8s/src/lib.rs	Exposes quota modules.
crates/spur-core/src/quota_names.rs	Shared DNS-safe naming for account namespaces and per-user ServiceAccounts.
crates/spur-core/src/lib.rs	Exports the new quota_names module.
crates/spur-cli/src/sacctmgr.rs	Adds grptres=... support for account add/modify.
crates/spur-cli/src/k8s.rs	Adds spur k8s kubeconfig --user <name> option to request scoped kubeconfig.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

crates/spurd/src/agent_server.rs:1752

• When req.user is set but namespace/service_account are empty, the call correctly fails in user_kubeconfig, but the RPC maps it to Status::internal. This is a client-input error; returning invalid_argument here makes the contract clearer and avoids treating caller bugs as server faults.

        let result = if req.user.is_empty() {
            self.k0s.admin_kubeconfig().await
        } else {
            self.k0s
                .user_kubeconfig(&req.user, &req.namespace, &req.service_account)

[shiv-tyagi]

Verified core (non-K8s) spur is not broken: build is green and 394 tests pass on the PR head. Proto fields are append-only, the grp_tres migration is idempotent and runs at startup before any read, create_account validation accepts empty, and both kubeconfig paths preserve the exact old behavior for an empty user. The quota reconciler is opt-in behind --enable-quota.

Four comments below: one RBAC/security question on secrets, an RPC-naming clarity point, a missing test on the scoped-kubeconfig resolution, and a unit-consistency nit. None are blocking.

[yansun1996]

Rebased onto latest main.

One commit from this branch (persist account grp_tres allocation — M1 increment 2) has been dropped: that feature already shipped to main as a separate merge, so its changes to the proto/db/CLI grp_tres plumbing were purely redundant with what's already there.

Two small follow-up commits were needed to make the rebase build clean:

• a rustfmt pass (formatting drift from auto-merging one file during the rebase, no functional change)
• a one-line fix to pass the new user filter field on ListUsersRequest, which main gained after this branch diverged

Build, clippy, and the spur-k8s/spurctld/spur-core/spur-cli test suites all pass on the rebased branch.