CCO-834, CCO-835, CCO-836: Upgrade to Kubernetes 1.36 with CI and e2e tests by ericahinkleRH · Pull Request #1066 · openshift/cloud-credential-operator

ericahinkleRH · 2026-07-10T17:59:09Z

Summary

Upgrades Cloud Credential Operator to Kubernetes 1.36 with complete CI implementation and e2e testing.

Jira Tickets

CCO-834: Core K8s 1.36 upgrade (dependency updates)
CCO-835: CI implementation
CCO-836: E2E testing

Changes

CCO-834: Core Upgrade

Updated library-go to v0.0.0-20260716104731-fdf18b82797f
Includes HasSyncedChecker() support from OCPSTRAT-3036: Rebase 1.36.2 library-go#2171
Required for K8s 1.36 controller-runtime compatibility

CCO-835: CI Implementation

Documented CI strategy in docs/K8S_1.36_UPGRADE.md
Existing Prow jobs validate K8s 1.36 compatibility
No new CI configuration required (central config in openshift/release)

CCO-836: E2E Testing

Added test/e2e/k8s136/k8s_1_36_upgrade_test.go
Tests verify:
- HasSyncedChecker compatibility
- Controller manager startup
- CCO deployment health
- No K8s 1.36 compatibility errors in logs

Testing

✅ Build passes
✅ Unit tests pass
✅ E2E tests added and documented
✅ All existing CI jobs validate K8s 1.36

Documentation

docs/K8S_1.36_UPGRADE.md - Complete upgrade guide

Related PRs

OCPSTRAT-3036: Rebase 1.36.2 library-go#2171 - HasSyncedChecker implementation (merged)

Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com

openshift-ci-robot · 2026-07-10T17:59:13Z

@ericahinkleRH: This pull request references CCO-834 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Upgrade Kubernetes dependencies from v0.35.2 to v0.36.0

Upgrade controller-runtime from v0.23.3 to v0.24.0 (required pairing for Kubernetes 1.36)

Bump github.com/openshift/api to a version compatible with APIs removed in Kubernetes 1.36

Regenerate codegen (hack/update-codegen.sh); remove empty zz_generated.defaults.go (no longer emitted by k8s 1.36 code-generator)

Apply temporary vendor patch to library-go test helpers for HasSyncedChecker() until openshift/library-go#2171 merges

Dependency changes

Module Before After

k8s.io/* v0.35.2 v0.36.0

sigs.k8s.io/controller-runtime v0.23.3 v0.24.0

github.com/openshift/api v0.0.0-20260317165824-54a3998d81eb v0.0.0-20260709102940-580f1c1ba691

No application logic changes.

library-go vendor patch

Kubernetes 1.36 added HasSyncedChecker() to the SharedIndexInformer interface. Upstream library-go does not implement this yet (#2171). The vendor patch in vendor/github.com/openshift/library-go/pkg/operator/v1helpers/test_helpers.go matches the upstream proposed fix and only affects test helper fakes. It can be removed on the next library-go bump after #2171 merges.

Test plan

make build

make test

make vet

verify-vendored-crds

hack/verify-codegen.sh

CI make verify

CI e2e (if run on branch)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-10T18:04:53Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The pull request updates OpenShift and Kubernetes dependencies for Kubernetes 1.36, removes generated no-op default registration, adds Kubernetes 1.36 end-to-end tests, and documents validation, upgrade, and rollback procedures.

Changes

Kubernetes 1.36 upgrade

Layer / File(s)	Summary
Dependency and generated-code alignment `go.mod`, `pkg/apis/cloudcredential/v1/zz_generated.defaults.go`	OpenShift, Kubernetes, controller-runtime, observability, and indirect Go dependencies are refreshed; the generated no-op `RegisterDefaults` file is removed.
Kubernetes 1.36 end-to-end validation `test/e2e/k8s136/k8s_1_36_upgrade_test.go`	New tests validate CCO deployment and pod readiness, scan recent pod logs for compatibility errors, and verify controller-manager availability.
Upgrade procedure and verification `docs/K8S_1.36_UPGRADE.md`	Documents the dependency change, CI and e2e validation, verification checklist, rollback steps, and references.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: dlom, jstuever

Sequence Diagram(s)

sequenceDiagram
  participant TestMain
  participant KubernetesAPI
  participant CCOPods
  TestMain->>KubernetesAPI: resolve kubeconfig and inspect deployment
  KubernetesAPI->>CCOPods: identify running CCO pods
  TestMain->>CCOPods: fetch recent logs
  CCOPods-->>TestMain: return readiness and compatibility log data
  KubernetesAPI-->>TestMain: return Available condition

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	The e2e test logs raw pod output via `t.Logf(..., logStr)`, which can expose tokens, hostnames, or customer data.	Remove full log dumps; log only matched patterns and a short redacted excerpt, or avoid emitting pod logs entirely.
Test Structure And Quality	⚠️ Warning	FAIL: The new e2e test has no context timeout on cluster API calls (testTimeout is unused) and its TestMain diverges from the repo pattern by not calling os.Exit(testenv.Run(m)).	Wrap each feature/assessment in context.WithTimeout(..., testTimeout) and restore TestMain to os.Exit(testenv.Run(m)); optionally split the first feature further.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The new test titles are static string literals; none include generated names, timestamps, UUIDs, or other run-specific data.
Microshift Test Compatibility	✅ Passed	The new e2e test uses only Kubernetes client APIs and no MicroShift-blocked OpenShift API groups, namespaces, or unsupported feature assumptions.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The new e2e tests only verify CCO deployment/pod health and logs; they don’t require multiple nodes, cross-node scheduling, or HA failover.
Topology-Aware Scheduling Compatibility	✅ Passed	Only docs and e2e tests changed in HEAD; no deployment manifests, operator code, or controllers were modified, so no new topology/scheduling assumptions were introduced.
Ote Binary Stdout Contract	✅ Passed	No process-level stdout writes were added; TestMain only configures env/testenv and the file has no fmt.Print/log/klog-to-stdout calls.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The new e2e test only queries in-cluster Kubernetes APIs and pod logs; no IPv4-only literals, host/IP formatting, or external internet calls were found.
No-Weak-Crypto	✅ Passed	No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or non-constant-time secret comparisons appear in the changed files.
Container-Privileges	✅ Passed	The PR adds docs and e2e tests only; no changed file sets privileged, hostPID/Network/IPC, SYS_ADMIN, runAsRoot, or allowPrivilegeEscalation=true.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: upgrading to Kubernetes 1.36 and adding CI/e2e coverage.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-07-10T18:06:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.21%. Comparing base (aeca790) to head (46d1368).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1066   +/-   ##
=======================================
  Coverage   47.21%   47.21%           
=======================================
  Files          97       97           
  Lines       12650    12650           
=======================================
  Hits         5973     5973           
  Misses       6017     6017           
  Partials      660      660

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dlom · 2026-07-13T16:35:49Z

/approve

ericahinkleRH · 2026-07-13T17:14:18Z

/retest-required

ericahinkleRH · 2026-07-13T17:30:15Z

/lgtm

openshift-ci · 2026-07-13T17:30:17Z

@ericahinkleRH: you cannot LGTM your own PR.

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ericahinkleRH · 2026-07-13T18:07:00Z

The verify-deps test is still failing even with deps.diff added. The error shows the vendor patch isn't being recognized correctly. Any suggestions on how to fix the deps.diff format?

nader-ziada · 2026-07-13T18:09:05Z

/lgtm

dlom · 2026-07-13T18:43:18Z

I think we can get rid of deps.diff entirely and go with the clean version of the library-go test helpers in the vendor folder

ericahinkleRH · 2026-07-13T19:02:39Z

Thanks @dlom! Removed the deps.diff and vendor patch. Using the clean version from go mod vendor now.

dlom · 2026-07-13T19:06:07Z

/override ci/prow/security

openshift-ci · 2026-07-13T19:06:28Z

@dlom: Overrode contexts on behalf of dlom: ci/prow/security

Details

In response to this:

/override ci/prow/security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ericahinkleRH · 2026-07-13T19:25:31Z

@dlom The verify-deps test is still failing after removing deps.diff. It's showing differences in the library-go test_helpers.go file, but running go mod vendor locally produces the same file that's already there. The library-go version in go.mod (v0.0.0-20260318142011-72bf34f474bc) already includes HasSyncedChecker(). Should I override this test as well, or is there something else I'm missing?

dlom · 2026-07-13T19:54:20Z

please rerun go mod vendor as well

dlom · 2026-07-13T23:53:20Z

/override ci/prow/security

openshift-ci · 2026-07-13T23:53:39Z

@dlom: Overrode contexts on behalf of dlom: ci/prow/security

Details

In response to this:

/override ci/prow/security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dlom · 2026-07-14T00:01:46Z

I see... let's wait for openshift/library-go#2171 to merge first before moving forwards with this PR

dlom · 2026-07-14T00:13:21Z

top level PR we're waiting on is openshift/api#2813

Updated to library-go v0.0.0-20260716104731-fdf18b82797f which includes the HasSyncedChecker method required for K8s 1.36 compatibility. This matches the same library-go version used in CCO-834 PR openshift#1066. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

dlom · 2026-07-21T19:01:45Z

@ericahinkleRH can you squash to one commit please?

ericahinkleRH · 2026-07-21T20:53:17Z

@dlom Done!

ericahinkleRH · 2026-07-22T12:32:56Z

/retest e2e-hypershift

ericahinkleRH · 2026-07-22T12:41:56Z

/test e2e-hypershift

ericahinkleRH · 2026-07-22T13:07:01Z

/test e2e-hypershift

ericahinkleRH · 2026-07-22T14:31:22Z

/test e2e-hypershift

ericahinkleRH · 2026-07-23T12:45:45Z

Hi @dlom,

PR #1066 is ready for final review and merge. This PR completes the K8s 1.36 upgrade for CCO (covers CCO-834, CCO-835, and CCO-836):

Updated all K8s dependencies to v0.36.0
Updated library-go with HasSyncedChecker support (OCPSTRAT-3036: Rebase 1.36.2 library-go#2171 merged)
Added e2e tests and CI documentation
Squashed to a single commit for clean history

All tests are passing, including e2e-hypershift which passed after infrastructure flakes resolved.

Since I squashed the commits, the previous approval was invalidated. Could you review and add /lgtm and /approve when you get a chance?

Thanks!

bandrade · 2026-07-23T14:43:04Z

/lgtm

ericahinkleRH · 2026-07-29T16:49:24Z

@dlom @jstuever could you take a look at this? All tests pass and it has the /lgtm and /approved labels. Just needs /verified to merge!

… tests This upgrades the cloud-credential-operator to use Kubernetes 1.36.2 dependencies and adds comprehensive e2e tests to validate the upgrade. Key changes: - Upgrade k8s.io/* dependencies from 0.35.2 to 0.36.2 - Upgrade controller-runtime from 0.23.3 to 0.24.0 - Update OpenShift dependencies for K8s 1.36 compatibility: - openshift/api: updated to version with K8s 1.36 support - openshift/client-go: updated for K8s 1.36 - openshift/library-go: updated for K8s 1.36 - Remove auto-generated zz_generated.defaults.go (no longer needed) - Add K8s 1.36 upgrade e2e test suite - Add upgrade documentation Includes vendor updates from master (PR 1068 GCP universe domain support). Jira: https://issues.redhat.com/browse/CCO-834 Jira: https://issues.redhat.com/browse/CCO-835 Jira: https://issues.redhat.com/browse/CCO-836

ericahinkleRH · 2026-07-30T15:42:33Z

/test security

ericahinkleRH · 2026-07-30T17:49:44Z

/test security

dlom · 2026-07-31T06:12:11Z

/override ci/prow/security

dlom · 2026-07-31T06:12:26Z

/lgtm

openshift-ci · 2026-07-31T06:12:26Z

@dlom: Overrode contexts on behalf of dlom: ci/prow/security

Details

In response to this:

/override ci/prow/security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2026-07-31T06:12:28Z

@ericahinkleRH: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci · 2026-07-31T06:13:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bandrade, dlom, ericahinkleRH, nader-ziada

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dlom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dlom · 2026-07-31T06:14:06Z

/verified by ci but if k8s 1.36 breaks anything catastrophic for us it'll break for everyone catastrophically too

openshift-ci-robot · 2026-07-31T06:14:18Z

@dlom: This PR has been marked as verified by ci but if k8s 1.36 breaks anything catastrophic for us it'll break for everyone catastrophically too.

Details

In response to this:

/verified by ci but if k8s 1.36 breaks anything catastrophic for us it'll break for everyone catastrophically too

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 10, 2026

openshift-ci Bot requested review from dlom and jstuever July 10, 2026 17:59

ericahinkleRH marked this pull request as draft July 10, 2026 18:12

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 10, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 13, 2026

ericahinkleRH marked this pull request as ready for review July 13, 2026 17:08

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 13, 2026

openshift-ci Bot assigned nader-ziada Jul 13, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 13, 2026

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 13, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 17, 2026

ericahinkleRH force-pushed the CCO-834 branch from 6fd80a8 to 2efb149 Compare July 21, 2026 20:52

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 21, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 23, 2026

ericahinkleRH force-pushed the CCO-834 branch from 2efb149 to 46d1368 Compare July 30, 2026 15:15

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 30, 2026

openshift-ci Bot assigned dlom Jul 31, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 31, 2026

openshift-merge-bot Bot merged commit 5a92092 into openshift:master Jul 31, 2026
13 checks passed

Uh oh!

Conversation

ericahinkleRH commented Jul 10, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Jira Tickets

Changes

CCO-834: Core Upgrade

CCO-835: CI Implementation

CCO-836: E2E Testing

Testing

Documentation

Related PRs

Uh oh!

openshift-ci-robot commented Jul 10, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependency changes

library-go vendor patch

Test plan

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Pre-merge checks failed

❌ Failed checks (1 error, 1 warning)

Uh oh!

codecov Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dlom commented Jul 13, 2026

Uh oh!

ericahinkleRH commented Jul 13, 2026

Uh oh!

ericahinkleRH commented Jul 13, 2026

Uh oh!

openshift-ci Bot commented Jul 13, 2026

Uh oh!

ericahinkleRH commented Jul 13, 2026

Uh oh!

nader-ziada commented Jul 13, 2026

Uh oh!

dlom commented Jul 13, 2026

Uh oh!

ericahinkleRH commented Jul 13, 2026

Uh oh!

dlom commented Jul 13, 2026

Uh oh!

openshift-ci Bot commented Jul 13, 2026

Uh oh!

ericahinkleRH commented Jul 13, 2026

Uh oh!

dlom commented Jul 13, 2026

Uh oh!

dlom commented Jul 13, 2026

Uh oh!

openshift-ci Bot commented Jul 13, 2026

Uh oh!

dlom commented Jul 14, 2026

Uh oh!

dlom commented Jul 14, 2026

Uh oh!

dlom commented Jul 21, 2026

Uh oh!

ericahinkleRH commented Jul 21, 2026

Uh oh!

ericahinkleRH commented Jul 22, 2026

Uh oh!

ericahinkleRH commented Jul 22, 2026

Uh oh!

ericahinkleRH commented Jul 22, 2026

Uh oh!

ericahinkleRH commented Jul 22, 2026

Uh oh!

ericahinkleRH commented Jul 23, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ericahinkleRH commented Jul 10, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jul 10, 2026 •

edited by openshift-ci Bot

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 10, 2026 •

edited

Loading

ericahinkleRH commented Jul 23, 2026 •

edited by openshift-ci Bot

Loading