CNTRLPLANE-4009: feat(aro-hcp): add e2e-aro-hcp-presubmit job with CPO override and nightly support by bryan-cox · Pull Request #83108 · openshift/release

bryan-cox · 2026-08-07T15:08:14Z

Summary

Adds a new e2e-aro-hcp-presubmit CI job that tests both the HyperShift Operator (HO) and Control Plane Operator (CPO) from a hypershift PR against an ARO-HCP environment using nightly OCP release images
Makes CPO push conditional in the images-push step — only runs when HYPERSHIFT_CPO_IMAGE is provided via a job-level dependency override (existing e2e-aro-hcp HO-only job is unaffected)
Adds ARO_HCP_E2E_SUITE, ARO_HCP_OPENSHIFT_CHANNEL_GROUP, and ARO_HCP_OPENSHIFT_VERSION_ID env vars to the test step
Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel in the presubmit job so it runs the correct test suite

Why

CPO is not backwards compatible — it must run alongside matching control plane components from the same OCP development stream. Using a released stable version with CPO from main would crash. The solution uses nightly OCP release images (already discovered by Cluster Service from multi.ocp.releases.ci.openshift.org/graph, cached via ACR pull-through) so CPO from the PR gets a compatible release image.

The new job:

Pushes HO to SVC ACR (same as existing job)
Pushes CPO to OCP ACR (HCP namespaces only have OCP ACR pull creds)
Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel to run the presubmit-specific test suite
Sets ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0 so the test framework resolves the latest accepted 5.0 nightly
CPO override flows via ARM experimental tag → admission → Cosmos → CS dispatch → HostedCluster annotation

Companion PR: Azure/ARO-HCP#6456 (test, framework, and Makefile changes)

Test plan

Verify e2e-aro-hcp (HO-only) job is unaffected — no CPO push, no nightly env vars, same default suite
Verify e2e-aro-hcp-presubmit job pushes both HO and CPO images
Verify presubmit job runs hypershift-presubmit/parallel suite (not rp-api-compat-all/parallel)
Verify nightly version resolution works with ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0
Verify CPO override env file is sourced by test step

🤖 Generated with Claude Code

Summary by CodeRabbit

Adds the e2e-aro-hcp-presubmit CI job for HyperShift pull requests. The job tests HyperShift Operator and Control Plane Operator changes against ARO-HCP with nightly OpenShift 5.0 release images.

When HYPERSHIFT_CPO_IMAGE is set, CI pushes the Control Plane Operator image to the OCP ACR and passes its digest to the test workflow. CI continues to push the HyperShift Operator image to the SVC ACR.

The existing HO-only e2e-aro-hcp job remains unchanged.

coderabbitai · 2026-08-07T15:08:48Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b5ec73c8-efb4-492e-b93b-bff77ce9455c

📥 Commits

Reviewing files that changed from the base of the PR and between abb85e7 and 27673b2.

⛔ Files ignored due to path filters (1)

ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (5)

ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-commands.sh
ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-ref.yaml
ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-commands.sh
ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-ref.yaml

🚧 Files skipped from review as they are similar to previous changes (5)

ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-ref.yaml
ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-commands.sh
ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-commands.sh
ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-ref.yaml

Walkthrough

The change adds an optional ARO HCP nightly presubmit job, supports optional CPO image publishing, passes the CPO digest to local tests, and adds configurable OpenShift channel and version settings.

Changes

ARO HCP CPO presubmit

Layer / File(s)	Summary
Publish and record the CPO image `ci-operator/step-registry/aro-hcp/hypershift-images-push/*`	When enabled, the push step authenticates to the OCP ACR, mirrors the CPO image, resolves its digest, and writes the CPO override environment file.
Consume the CPO override in local tests `ci-operator/step-registry/aro-hcp/test/local/*`	The local test step optionally sources the CPO override. New settings select the e2e suite, OpenShift channel group, and nightly minor version.
Configure the nightly presubmit job `ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml`	An optional `e2e-aro-hcp-presubmit` job uses the ARO HCP development profile, nightly OpenShift settings, CPO image publishing, and the existing workflow.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Presubmit as e2e-aro-hcp-presubmit
  participant PushStep as hypershift-images-push
  participant OCPACR as OCP ACR
  participant LocalTest as aro-hcp-test-local
  Presubmit->>PushStep: Enable CPO image publishing
  PushStep->>OCPACR: Authenticate and mirror HYPERSHIFT_CPO_IMAGE
  OCPACR-->>PushStep: Return the CPO image digest
  PushStep->>LocalTest: Provide CPO_IMAGE_OVERRIDE
  LocalTest->>LocalTest: Apply suite and OpenShift version settings

Suggested reviewers: enxebre, deads2k

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	When PUSH_CPO_IMAGE=true, lines 33, 76, and 141-147 print the OCP ACR hostname and full CPO pullspec/override, exposing internal registry metadata in CI logs.	Log only fixed labels and status. Redact or omit OCP_ACR_URL, HYPERSHIFT_CPO_IMAGE, and the full CPO_IMAGE_OVERRIDE from stdout.
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: adding the ARO-HCP presubmit job, CPO override support, and nightly testing.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR changes only CI configuration, generated Prow YAML, and shell/ref files; it adds no Ginkgo It, Describe, Context, or When test titles.
Test Structure And Quality	✅ Passed	The pull request changes only four YAML files and two shell scripts; it adds no Ginkgo test code or It blocks to assess.
Microshift Test Compatibility	✅ Passed	The patch adds no Ginkgo tests or test declarations; it changes only CI YAML and shell workflow files, so MicroShift test compatibility checks do not apply.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The patch adds CI configuration and shell/YAML changes only; it adds no Ginkgo declarations or new e2e test source, so SNO compatibility checks are not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The six modified files are CI job/configuration and step scripts; the diff adds no anti-affinity, topology spread, replica, PDB, node selector, affinity, or toleration constraints.
Ote Binary Stdout Contract	✅ Passed	The PR changes only YAML configuration and shell scripts; no OTE binary source or process-level Go stdout/logging code was added.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The diff adds only CI configuration and shell/ref wiring; it adds no Ginkgo test declarations, IPv4 assumptions, or public endpoint usage in test code.
No-Weak-Crypto	✅ Passed	The diff adds CI configuration, ACR authentication, image mirroring, digest resolution, and environment sourcing; it adds no weak algorithms, custom crypto, or non-constant-time secret comparisons.
Container-Privileges	✅ Passed	The added job, workflow, step YAML, and scripts contain no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or root settings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai · 2026-08-07T15:20:07Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

openshift-ci-robot · 2026-08-07T15:24:53Z

@bryan-cox: This pull request references CNTRLPLANE-4009 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Adds a new e2e-aro-hcp-presubmit CI job that tests both the HyperShift Operator (HO) and Control Plane Operator (CPO) from a hypershift PR against an ARO-HCP environment using nightly OCP release images

Makes CPO push conditional in the images-push step — only runs when HYPERSHIFT_CPO_IMAGE is provided via a job-level dependency override (existing e2e-aro-hcp HO-only job is unaffected)

Adds ARO_HCP_E2E_SUITE, ARO_HCP_OPENSHIFT_CHANNEL_GROUP, and ARO_HCP_OPENSHIFT_VERSION_ID env vars to the test step

Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel in the presubmit job so it runs the correct test suite

Why

CNTRLPLANE-4009 / CNTRLPLANE-3980

CPO is not backwards compatible — it must run alongside matching control plane components from the same OCP development stream. Using a released stable version with CPO from main would crash. The solution uses nightly OCP release images (already discovered by Cluster Service from multi.ocp.releases.ci.openshift.org/graph, cached via ACR pull-through) so CPO from the PR gets a compatible release image.

The new job:

Pushes HO to SVC ACR (same as existing job)

Pushes CPO to OCP ACR (HCP namespaces only have OCP ACR pull creds)

Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel to run the presubmit-specific test suite

Sets ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0 so the test framework resolves the latest accepted 5.0 nightly

CPO override flows via ARM experimental tag → admission → Cosmos → CS dispatch → HostedCluster annotation

Companion PR: Azure/ARO-HCP#6456 (test, framework, and Makefile changes)

Test plan

Verify e2e-aro-hcp (HO-only) job is unaffected — no CPO push, no nightly env vars, same default suite

Verify e2e-aro-hcp-presubmit job pushes both HO and CPO images

Verify presubmit job runs hypershift-presubmit/parallel suite (not rp-api-compat-all/parallel)

Verify nightly version resolution works with ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0

Verify CPO override env file is sourced by test step

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-08-07T15:39:30Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

…ghtly support Adds a new `e2e-aro-hcp-presubmit` CI job that tests both the HyperShift Operator (HO) and Control Plane Operator (CPO) from a hypershift PR against an ARO-HCP environment using nightly OCP release images. Changes: - images-push: conditionally push CPO to OCP ACR when HYPERSHIFT_CPO_IMAGE is provided (via job-level dependency override), and write hypershift-cpo-override.env with the CPO_IMAGE_OVERRIDE ref - test-local: source hypershift-cpo-override.env if present, declare ARO_HCP_OPENSHIFT_CHANNEL_GROUP and ARO_HCP_OPENSHIFT_VERSION_ID env vars - ci-operator config: new e2e-aro-hcp-presubmit job that provides the hypershift (CPO) dependency and sets nightly channel group with OCP 5.0 The existing e2e-aro-hcp job (HO-only) is not affected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

coderabbitai · 2026-08-10T12:22:29Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

openshift-ci · 2026-08-10T12:22:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bryan-cox
Once this PR has been reviewed and has the lgtm label, please assign roivaz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

~~ci-operator/config/openshift/hypershift/OWNERS~~ [bryan-cox]
~~ci-operator/jobs/openshift/hypershift/OWNERS~~ [bryan-cox]
~~ci-operator/step-registry/aro-hcp/hypershift-images-push/OWNERS~~ [bryan-cox]
ci-operator/step-registry/aro-hcp/test/local/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-08-10T12:25:39Z

[REHEARSALNOTIFIER]
@bryan-cox: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-hypershift-main-e2e-aro-hcp-presubmit	openshift/hypershift	presubmit	Presubmit changed
pull-ci-openshift-hypershift-main-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-openshift-hypershift-release-5.1-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-openshift-hypershift-release-5.0-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-openshift-hypershift-release-4.23-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-Azure-ARO-HCP-main-e2e-parallel	Azure/ARO-HCP	presubmit	Registry content changed
pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel	Azure/ARO-HCP	presubmit	Registry content changed
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aro-hcp	N/A	periodic	Registry content changed
periodic-ci-openshift-hypershift-release-5.1-periodics-e2e-aro-hcp	N/A	periodic	Registry content changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

openshift-ci · 2026-08-10T12:34:26Z

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

bryan-cox · 2026-08-10T12:47:28Z

/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aro-hcp-presubmit

openshift-merge-bot · 2026-08-10T12:47:30Z

@bryan-cox: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci Bot requested review from deads2k and enxebre August 7, 2026 15:09

bryan-cox force-pushed the CNTRLPLANE-4009 branch from 80be42a to db0239d Compare August 7, 2026 15:19

bryan-cox mentioned this pull request Aug 7, 2026

CNTRLPLANE-4009: feat(test): add HyperShift presubmit e2e test and nightly version support Azure/ARO-HCP#6456

Open

5 tasks

bryan-cox changed the title ~~feat(aro-hcp): add e2e-aro-hcp-presubmit job with CPO override and nightly support~~ CNTRLPLANE-4009: feat(aro-hcp): add e2e-aro-hcp-presubmit job with CPO override and nightly support Aug 7, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 7, 2026

bryan-cox force-pushed the CNTRLPLANE-4009 branch from db0239d to cf24bfa Compare August 7, 2026 15:39

bryan-cox force-pushed the CNTRLPLANE-4009 branch from cf24bfa to 27673b2 Compare August 10, 2026 12:22

Navigation Menu

Uh oh!

CNTRLPLANE-4009: feat(aro-hcp): add e2e-aro-hcp-presubmit job with CPO override and nightly support - #83108

CNTRLPLANE-4009: feat(aro-hcp): add e2e-aro-hcp-presubmit job with CPO override and nightly support#83108
bryan-cox wants to merge 1 commit into
openshift:mainfrom
bryan-cox:CNTRLPLANE-4009

bryan-cox commented Aug 7, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

Pre-merge checks failed

Uh oh!

coderabbitai Bot commented Aug 7, 2026

Uh oh!

openshift-ci-robot commented Aug 7, 2026 •

edited by openshift-ci Bot

Loading

Summary

Why

Test plan

Uh oh!

coderabbitai Bot commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

bryan-cox commented Aug 10, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bryan-cox commented Aug 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Pre-merge checks failed

❌ Failed checks (1 error, 1 warning)

Uh oh!

coderabbitai Bot commented Aug 7, 2026

Uh oh!

openshift-ci-robot commented Aug 7, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Test plan

Uh oh!

coderabbitai Bot commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

bryan-cox commented Aug 10, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bryan-cox commented Aug 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

openshift-ci-robot commented Aug 7, 2026 •

edited by openshift-ci Bot

Loading