Skip to content

fix nvidiadriver status for multiple default nvidiadrivers - #2678

Merged
rahulait merged 1 commit into
NVIDIA:mainfrom
rahulait:fix-nvidiadriver-multiple-default-status
Jul 27, 2026
Merged

fix nvidiadriver status for multiple default nvidiadrivers#2678
rahulait merged 1 commit into
NVIDIA:mainfrom
rahulait:fix-nvidiadriver-multiple-default-status

Conversation

@rahulait

@rahulait rahulait commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Detect multiple default NVIDIADrivers during validation, mark the affected CRs NotReady with a ReconcileFailed condition, and retain the fail-closed owner-assignment behavior.

Add unit and E2E coverage for duplicate defaults and recovery after the duplicate is removed.

Bug

In PR #2569, we moved AssignOwners() from NVIDIADriverReconciler into the new NodeLabelingReconciler to centralize node-label writes.

Before that change, the NVIDIADriver reconciler called AssignOwners() directly. When multiple NVIDIADriver CRs had spec.default: true, AssignOwners() returned an error and the reconciler:

  • set the reconciled NVIDIADriver’s status.state to notReady
  • set a ReconcileFailed condition containing the duplicate-default error
  • returned the error to retry reconciliation

After PR #2569, AssignOwners() continued to detect duplicate defaults and fail closed, but it ran only in NodeLabelingReconciler. That controller returned the error without updating any NVIDIADriver status. At the same time, NVIDIADriverReconciler's selector validator intentionally skips default drivers while checking node-selector overlap, so duplicate defaults passed validation and could be marked ready.

As a result, duplicate defaults were still prevented from changing node owner labels, but the invalid CRs no longer reported notReady or exposed the error in their status conditions.

Fix

The fix restores status reporting without moving owner assignment back into NVIDIADriverReconciler.

  • Extend NVIDIADriver validation to reject multiple default NVIDIADrivers.
  • Preserve the existing owner-assignment behavior: NodeLabelingReconciler remains the sole writer of node owner labels, and AssignOwners() still fails closed before changing labels.
  • When duplicate defaults are detected, NVIDIADriverReconciler sets status.state=notReady, adds a ReconcileFailed condition with the duplicate-default message, and returns the error for retry.
  • Ignore NVIDIADrivers being deleted when checking duplicate defaults, consistent with owner assignment behavior.
  • Keep the existing selector-conflict behavior unchanged: normal overlapping non-default selectors continue to use the ConflictingNodeSelector condition reason.

The NVIDIADriver controller already requeues all NVIDIADrivers after a NVIDIADriver create, update, or delete. Therefore, both duplicate default CRs converge to notReady; after one is deleted or changed to non-default, the remaining default reconciles normally again.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

Deployed a cluster with multiple nodes and drivers managed by nvidiadriver.

Before the fix

If 2 default nvidiadrivers were created, their status was showing up as ready even though they conflicted with one another.

After the fix

Created a k8s cluster and added multiple default nvidiadrivers. They correctly went into notReady state on conflict.

@rahulait
rahulait force-pushed the fix-nvidiadriver-multiple-default-status branch from fe1f170 to 71c931b Compare July 25, 2026 23:15
@rahulait rahulait self-assigned this Jul 26, 2026
@rahulait rahulait added this to the v26.7 milestone Jul 27, 2026
Comment thread tests/scripts/update-nvidiadriver.sh
@rahulait
rahulait force-pushed the fix-nvidiadriver-multiple-default-status branch from 71c931b to 1cd642c Compare July 27, 2026 16:31
Detect multiple default NVIDIADrivers during validation, mark the
affected CRs NotReady with a ReconcileFailed condition, and retain the
fail-closed owner-assignment behavior.

Add unit and E2E coverage for duplicate defaults and recovery after
the duplicate is removed.

Signed-off-by: Rahul Sharma <rahulsharm@nvidia.com>
@rahulait
rahulait force-pushed the fix-nvidiadriver-multiple-default-status branch from 1cd642c to 2b85a75 Compare July 27, 2026 16:48
@rahulait
rahulait merged commit 6d3812a into NVIDIA:main Jul 27, 2026
37 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants