Support NVIDIADriver reconciliation and driver upgrades without a ClusterPolicy by karthikvetrivel · Pull Request #2572 · NVIDIA/gpu-operator

karthikvetrivel · 2026-06-23T15:51:30Z

Stacked on #2571 (kv-gpuclusterconfig-crd). This PR's diff is just the two commits below. GitHub will retarget it onto main once #2571 lands.

Summary

This PR lets the NVIDIADriver controller reconcile without a ClusterPolicy, so a GPUCluster (the DRA stack) can use operator-managed drivers standalone.

The driver host root is now sourced via the info catalog instead of the ClusterPolicy CR. The driver renderer consumes a new InfoTypeHostRoot catalog entry rather than reading the ClusterPolicy directly, which decouples it from the gpuv1 types. When both CRs exist, the ClusterPolicy's hostPaths.rootFS takes precedence. Otherwise the GPUCluster's value is used.
Enabling DRA on a cluster requires that all driver management in that cluster is done through NVIDIADriver CRs. A ClusterPolicy may coexist with a GPUCluster CR, but only if it delegates driver management by setting driver.useNvidiaDriverCRD=true. A ClusterPolicy that owns driver daemonsets directly (useNvidiaDriverCRD=false) alongside a GPUCluster CR is an invalid configuration. There is no supported topology where ClusterPolicy-owned drivers and NVIDIADriver-owned drivers run in the same cluster.

Testing

There is unit coverage for the host-root sourcing and the gate transitions, plus two live validations on a single-node Tesla T4 cluster (Kubernetes 1.34, containerd 2.2).

Standalone end-to-end: with no ClusterPolicy, a GPUCluster plus a default NVIDIADriver rolled out the driver DaemonSet. The DRA kubelet-plugin came up against the operator-installed driver root (devRoot=/run/nvidia/driver) and published its DeviceClasses and ResourceSlice. A pod with a DRA claim ran nvidia-smi against the operator-installed driver.
Gate matrix, observed via .status.state transitions on a cluster converted from a released operator install. A ClusterPolicy that is not delegating with no GPUCluster produced disabled. A GPUCluster alone produced ready. With neither CR, the NVIDIADriver went notReady with the expected message. The ClusterPolicy plus GPUCluster coexistence path is covered by the unit tests.
All transitions were watch-driven. Deleting or creating a GPUCluster flipped the NVIDIADriver state without manual intervention, and a single GPUCluster creation transitioned two NVIDIADriver instances at once.

coveralls · 2026-07-01T15:31:22Z

coverage: 35.959% (+1.4%) from 34.591% — kv-nvidiadriver-standalone into main

tariq1890 · 2026-07-24T18:43:09Z

+		hostRoot = gpuCluster.Spec.HostPaths.RootFS
+	default:
+		err := fmt.Errorf("no ClusterPolicy or GPUCluster object found in the cluster")
+		logger.Error(err, "failed to get a cluster-wide configuration object")


Suggested change

logger.Error(err, "failed to get a cluster-wide configuration object")

logger.Error(err, "failed to retrieve hostPaths information. No ClusterPolicy or GPUCluster object found in the cluster")

Done, applied in the squashed commit.

tariq1890 · 2026-07-24T18:53:13Z

 		return reconcile.Result{}, err
 	}

+	if clusterPolicy == nil {


We can merge two if blocks into one with this change

Suggested change

if clusterPolicy == nil {

if clusterPolicy == nil || clusterPolicy.Spec.Driver.UseNvidiaDriverCRDType() {

Actually, isn't it possible now to have both ClusterPolicy-backed drivers and NVIDIADriver-backed drivers (we can continue to debate whether this is actually desired or not in a separate thread)? IIUC the upgrade-controller, as currently implemented, does not account for that use case -- it only calls reconcileNVIDIADriverUpgrades() OR reconcileClusterPolicyDriverUpgrades.

The problematic scenario -- ClusterPolicy and GPUCluster exist; clusterpolicy.spec.driver.useNvidiaDriverCRD=false; NVIDIADriver CRs exist which target DRA nodes. In this case, the upgrade-controller will not be able to manage the upgrades for both the ClusterPolicy-owned driver daemonsets and the NVIDIADriver-owned driver daemonsets.

Good catch. You are right that the upgrade controller does not handle that.

As we discussed offline, we decided not to support it. To summaize, enabling DRA now requires that all driver management in the cluster is done through NVIDIADriver CRs. A ClusterPolicy with useNvidiaDriverCRD=false alongside a GPUCluster is treated as an invalid configuration.

tariq1890 · 2026-07-24T18:58:34Z

 		}
+		hostRoot = clusterPolicy.Spec.HostPaths.RootFS
+	case gpuCluster != nil:
+		hostRoot = gpuCluster.Spec.HostPaths.RootFS


Correct me if I am wrong, but in the scenario where both ClusterPolicy and GPUCluster exist, will the .spec.hostPaths.rootFS from GPUCluster have the higher order of precedence?

ClusterPolicy takes precedence today. Now, I believe this combination is only valid when the ClusterPolicy sets useNvidiaDriverCRD=true, and in that case ClusterPolicy's hostPaths.rootFS is used.

I see that this hostRoot value gets used here

- name: host-root hostPath: path: {{ .HostRoot | default "/" }}

Thinking about this more, it doesn't make much sense to make this global value and we are better off getting this value from the NVIDIADriver CR. For this, we would need to add a new field called hostRoot to the NVIDIADriver CR. This is the behaviour I envision in the following scenarios:

If the GPUCluster resource is present (whether by itself or along with ClusterPolicy, then we get the hostRoot from the NVIDIADriver CR, else we fallback to the GPUCluster hostRoot. Here we will never get the hostRoot from ClusterPolicy. This way we ensure zero dependency on ClusterPolicy in a GPUCluster flow

If only ClusterPolicy is present, then we either get the hostRoot from the NVIDIADriver CR (the newly added field), else we fallback to the ClusterPolicy.Spec.HostPaths.RootFS

This can be done in a follow-up PR

Note that when we add hostRoot to the NVIDIADriver CR, it'll be an additive change and it will be an optional field. If this is empty, it will be defaulted to "/" as we currently do today

Note, the host root is needed for many of our daemonsets, not just the driver. So we still need this field to be present in the GPUCluster CR. What is the benefit in having users configure this field in multiple places (GPUCluster and NVIDIADriver CRs)?

Yes, the rest of the HostPaths fields are used correctly in operands whether they come from GPUCluster or ClusterPolicy. For the driver daemonset however, it's just the .HostPaths.RootFS field that is relevant

I'm in favor of moving hostRoot to NVIDIADriver and removing it entirely from GPUCluster.

In the legacy ClusterPolicy path, we can get the hostRoot from either NVIDIADriver CR (if specified) or from ClusterPolicy.Spec.HostPaths.RootFS when useNvidiaDriverCRD=true, and from ClusterPolicy.Spec.HostPaths.RootFS only when useNvidiaDriverCRD=false.

tariq1890 · 2026-07-24T19:01:35Z

Thanks @karthikvetrivel! Can you also squash your commit history here?

…sterPolicy Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

karthikvetrivel requested review from cdesiniotis, rahulait, rajathagasthya, shivamerla and tariq1890 as code owners June 23, 2026 15:51

karthikvetrivel marked this pull request as draft June 23, 2026 15:54

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch 3 times, most recently from 8ffcfc3 to 479562c Compare June 24, 2026 19:05

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from 5691bbc to e5dcecd Compare July 1, 2026 15:24

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 479562c to 0e75d4c Compare July 1, 2026 15:24

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 6 times, most recently from 3b7cc55 to 4ad2c9d Compare July 8, 2026 18:48

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 8 times, most recently from 1a06b01 to b101993 Compare July 16, 2026 16:01

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 0e75d4c to 7f38f41 Compare July 16, 2026 16:11

karthikvetrivel marked this pull request as ready for review July 16, 2026 16:48

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from b101993 to 7f26b3c Compare July 20, 2026 18:31

karthikvetrivel mentioned this pull request Jul 20, 2026

Add GPUCluster CRD and controller for DRA-based stack #2571

Merged

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 2 times, most recently from c4a3751 to 3c9eb31 Compare July 21, 2026 19:56

karthikvetrivel changed the title ~~Support NVIDIADriver reconciliation without a ClusterPolicy~~ Support NVIDIADriver reconciliation and driver upgrades without a ClusterPolicy Jul 21, 2026

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 3 times, most recently from afefc58 to 3be4304 Compare July 22, 2026 18:23

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 7f38f41 to fdb7c6e Compare July 22, 2026 18:54

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 11 times, most recently from d5ef70f to a063aa9 Compare July 23, 2026 20:00

Base automatically changed from kv-gpuclusterconfig-crd to main July 23, 2026 23:05

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from fdb7c6e to a9629e2 Compare July 24, 2026 16:19

tariq1890 reviewed Jul 24, 2026

View reviewed changes

rahulait assigned karthikvetrivel Jul 27, 2026

rahulait added this to the v26.7 milestone Jul 27, 2026

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch 2 times, most recently from 9530322 to f9e8163 Compare July 27, 2026 16:30

Support NVIDIADriver reconciliation and driver upgrades without a Clu…

aeca33f

…sterPolicy Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from f9e8163 to aeca33f Compare July 27, 2026 16:36

	logger.Error(err, "failed to get a cluster-wide configuration object")
	logger.Error(err, "failed to retrieve hostPaths information. No ClusterPolicy or GPUCluster object found in the cluster")

	if clusterPolicy == nil {
	if clusterPolicy == nil \|\| clusterPolicy.Spec.Driver.UseNvidiaDriverCRDType() {

Uh oh!

Conversation

karthikvetrivel commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

coveralls commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikvetrivel Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tariq1890 commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

karthikvetrivel commented Jun 23, 2026 •

edited

Loading

coveralls commented Jul 1, 2026 •

edited

Loading

karthikvetrivel Jul 27, 2026 •

edited

Loading