fix nvidiadriver status for multiple default nvidiadrivers - #2678
Merged
rahulait merged 1 commit intoJul 27, 2026
Merged
Conversation
rahulait
requested review from
cdesiniotis,
karthikvetrivel,
rajathagasthya,
shivamerla and
tariq1890
as code owners
July 25, 2026 23:09
rahulait
force-pushed
the
fix-nvidiadriver-multiple-default-status
branch
from
July 25, 2026 23:15
fe1f170 to
71c931b
Compare
tariq1890
reviewed
Jul 27, 2026
rahulait
force-pushed
the
fix-nvidiadriver-multiple-default-status
branch
from
July 27, 2026 16:31
71c931b to
1cd642c
Compare
Detect multiple default NVIDIADrivers during validation, mark the affected CRs NotReady with a ReconcileFailed condition, and retain the fail-closed owner-assignment behavior. Add unit and E2E coverage for duplicate defaults and recovery after the duplicate is removed. Signed-off-by: Rahul Sharma <rahulsharm@nvidia.com>
rahulait
force-pushed
the
fix-nvidiadriver-multiple-default-status
branch
from
July 27, 2026 16:48
1cd642c to
2b85a75
Compare
tariq1890
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detect multiple default NVIDIADrivers during validation, mark the affected CRs NotReady with a ReconcileFailed condition, and retain the fail-closed owner-assignment behavior.
Add unit and E2E coverage for duplicate defaults and recovery after the duplicate is removed.
Bug
In PR #2569, we moved AssignOwners() from NVIDIADriverReconciler into the new NodeLabelingReconciler to centralize node-label writes.
Before that change, the NVIDIADriver reconciler called AssignOwners() directly. When multiple NVIDIADriver CRs had
spec.default: true,AssignOwners()returned an error and the reconciler:After PR #2569,
AssignOwners()continued to detect duplicate defaults and fail closed, but it ran only in NodeLabelingReconciler. That controller returned the error without updating any NVIDIADriver status. At the same time, NVIDIADriverReconciler's selector validator intentionally skips default drivers while checking node-selector overlap, so duplicate defaults passed validation and could be marked ready.As a result, duplicate defaults were still prevented from changing node owner labels, but the invalid CRs no longer reported notReady or exposed the error in their status conditions.
Fix
The fix restores status reporting without moving owner assignment back into NVIDIADriverReconciler.
status.state=notReady, adds a ReconcileFailed condition with the duplicate-default message, and returns the error for retry.The NVIDIADriver controller already requeues all NVIDIADrivers after a NVIDIADriver create, update, or delete. Therefore, both duplicate default CRs converge to notReady; after one is deleted or changed to non-default, the remaining default reconciles normally again.
Checklist
make lint)make validate-generated-assets)make validate-modules)Testing
Deployed a cluster with multiple nodes and drivers managed by nvidiadriver.
Before the fix
If 2 default nvidiadrivers were created, their status was showing up as ready even though they conflicted with one another.
After the fix
Created a k8s cluster and added multiple default nvidiadrivers. They correctly went into notReady state on conflict.