feat(quota): Kubernetes quota-enforcement — M1 (tenancy + hard quotas + scoped kubeconfig) by powderluv · Pull Request #471 · ROCm/spur

powderluv · 2026-07-17T23:20:23Z

What

Implements M1 of the Kubernetes quota-enforcement design (#444 RFC): the tenancy + hard-quota layer that projects SPUR accounts onto native Kubernetes objects, plus per-user scoped kubeconfigs — so a cluster is a completely normal Kubernetes deployment to its users while SPUR is the invisible policy plane.

Draft because the cluster end-to-end test on crsuse2 is still pending (see Status below). All code is build- + unit-test-verified, including spurd on Linux.

Stacked on #432 — the base branch is users/powderluv/m8-native-k8s, so this diff is just the 4 quota commits. Re-target to main once #432 merges.

End-to-end flow this lands

sacctmgr add account physics grptres=cpu=64,mem=131072,gres/gpu=8
  → the operator's quota reconciler (opt-in --enable-quota) projects it to:
      namespace spur-acct-physics
      + ResourceQuota (requests/limits.cpu, .memory, requests.amd.com/gpu)
      + LimitRange (default requests so unset-request pods still count)
      + Role + RoleBinding (the account's members)

spur k8s kubeconfig --user alice
  → spurctld resolves alice → physics → its namespace, the control-plane agent
    mints a ServiceAccount + bound token there → a namespace-scoped kubeconfig
    (no cluster-admin credential ever handed out)

Increments (each independently reviewable)

Commit	Increment
projection module	Pure `account → Namespace/ResourceQuota/LimitRange/RBAC` mapping (`spur-k8s/quota.rs`), fully unit-tested
persist `grp_tres`	Account-level TRES allocation through DB (column + migration) + proto + gRPC + `sacctmgr`
reconciler	`quota_controller` in the operator: lists accounts+members over gRPC, server-side-applies with `force` (drift-correcting), opt-in `--enable-quota`
`kubeconfig --user`	Shared naming in `spur-core`, scoped kubeconfig minted by the control-plane agent (`k0s kubectl create token`), CLI flag

Design decision

Per the RFC, SPUR persists resource caps on QoS, not accounts. M1 wires account-level allocations (grp_tres on the Account) so each account/namespace has a definite ResourceQuota — the cleanest tenancy model, and it gives the GPU cluster the GPU quota it lacks today.

Status / not in this PR

Opt-in: the reconciler only runs with --enable-quota (off by default) — zero effect on existing deployments.
Not yet cluster-tested: the SA/token minting + the reconcile apply path are integration-level and pending a crsuse2 run. The pure logic (TRES→ResourceQuota mapping, DNS-safe naming, CA/server parse, kubeconfig template, build_account_quota) is unit-tested.
Later milestones (M2 partitions+priority, M3 usage feedback, M4 Kueue/native fairness, M5 polish) are separate.

Testing

Build + tests + fmt + clippy -D warnings green across every crate — spur-core/spurctld/spur-cli/spur-k8s on macOS and spurd on Linux (cargo test -p spurd, clippy clean). DB round-trip tests are #[ignore] (run in CI's Postgres).

🤖 Generated with Claude Code

…a/LimitRange/RBAC) Pure, unit-tested mapping from a SPUR account allocation (grp_tres) to the native k8s objects that enforce it. 6 tests green. Foundation for the quota policy reconciler; DB/proto persistence + controller + `kubeconfig --user` follow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Wire an account-level resource allocation (grp_tres: cpu/mem/amd.com/gpu) through the accounting stack so the quota reconciler can project it to a per-account ResourceQuota. SPUR previously persisted TRES caps only on QoS, not accounts. - proto: CreateAccountRequest + AccountInfo gain a `grp_tres` TRES string. - db: accounts table gets a grp_tres column (+ ADD COLUMN migration for existing DBs); upsert_account/list_accounts/AccountRecord carry it; round-trip test. - grpc: create_account persists grp_tres; list_accounts returns it. - cli: `sacctmgr add/modify account grptres=cpu=16,mem=32768,gres/gpu=8`. Verified: build + spurctld/spur-cli/spur-k8s tests + fmt + clippy -D warnings (the DB round-trip test is #[ignore], runs in CI's Postgres). Also folds a rustfmt/doc-lint cleanup of increment 1's quota.rs + the CP-fix formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Makes the quota layer live: an opt-in reconcile loop that projects every SPUR account into its native k8s objects and drift-corrects them. - quota_controller::run: connects a SlurmAccountingClient to spurctld, then every 30s lists accounts (with grp_tres) + members and server-side-applies (force, so an admin hand-edit is reverted) the Namespace + ResourceQuota + LimitRange + Role + RoleBinding from crate::quota. - build_account_quota: pure AccountInfo + members -> AccountQuota (unit-tested). - apply<K>: generic SSA helper (injects apiVersion/kind, which k8s-openapi types don't serialize but server-side apply requires). - Wired into main.rs behind --enable-quota (default off; opt-in policy plane), spawned with the same run_with_retry backoff as the other controllers. Verified: build + spur-k8s tests (146) + fmt + clippy -D warnings. The apply/RPC path is integration-level (needs a cluster); the mapping + build logic are unit-tested. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…onfig (M1 increment 4) A SPUR user gets a namespace-scoped credential into their account's namespace, without an admin ever handing out the cluster-admin kubeconfig. - spur-core::quota_names: the per-account namespace + per-user ServiceAccount naming, shared so the operator (which creates them) and spurctld (which mints the kubeconfig) agree. Moved out of spur-k8s::quota. - proto: ClusterKubeconfigRequest.user; GetAdminKubeconfigRequest gains user/namespace/service_account (reuses the existing RPC — no new trait method). - spurd: K0sAgent::user_kubeconfig ensures the SA exists + mints a bound token via `k0s kubectl create token`, then templates a scoped kubeconfig from the admin cluster CA/server (pure parse + template helpers, unit-tested). - spurctld: cluster_kubeconfig resolves the user's account (association cache) -> namespace + SA and forwards to the control-plane agent. - cli: `spur k8s kubeconfig --user <u>`. Verified (no cluster run, per request): build + tests + fmt + clippy -D warnings across all crates incl. spurd on Linux (shark-a). The SA/token minting is integration-level; the naming, CA/server parse, and kubeconfig template are unit-tested. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

yansun1996 · 2026-07-21T05:58:13Z

cherry-picked to PR #475 to keep working on this feature

yansun1996 · 2026-07-28T05:40:32Z

close this PR since the commits has been cherry-picked to PR #475 , and the commits have been merged.

powderluv and others added 4 commits July 17, 2026 13:45

yansun1996 mentioned this pull request Jul 20, 2026

feat(quota): Kubernetes quota-enforcement — M1 (tenancy + hard quotas + scoped kubeconfig) #475

Merged

yansun1996 closed this Jul 28, 2026

Navigation Menu

Uh oh!

feat(quota): Kubernetes quota-enforcement — M1 (tenancy + hard quotas + scoped kubeconfig) - #471

feat(quota): Kubernetes quota-enforcement — M1 (tenancy + hard quotas + scoped kubeconfig)#471
powderluv wants to merge 4 commits into
users/powderluv/m8-native-k8sfrom
users/powderluv/k8s-quota-m1

powderluv commented Jul 17, 2026

Uh oh!

yansun1996 commented Jul 21, 2026

Uh oh!

yansun1996 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

powderluv commented Jul 17, 2026

What

End-to-end flow this lands

Increments (each independently reviewable)

Design decision

Status / not in this PR

Testing

Uh oh!

yansun1996 commented Jul 21, 2026

Uh oh!

yansun1996 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants