Skip to content

WIP: add openperouter deploy-verify test on AWS #82873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oribon wants to merge 1 commit into from
Closed

WIP: add openperouter deploy-verify test on AWS #82873

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 81 additions & 3 deletions ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
base_images:
cli-operator-sdk:
name: cli-operator-sdk
namespace: ocp
tag: v1.31.0
binary_build_commands: make build
build_root:
image_stream_tag:
Expand All @@ -9,7 +14,8 @@ images:
items:
- dockerfile_path: Dockerfile.openshift
to: openperouter-operator
- dockerfile_path: operator/bundle.Dockerfile.openshift
- context_dir: operator
dockerfile_path: bundle.Dockerfile
to: openperouter-operator-bundle
- build_args:
- name: BASE_IMAGE
Expand All @@ -19,10 +25,11 @@ images:
operator:
bundles:
- as: operator-bundle
dockerfile_path: operator/bundle.Dockerfile.openshift
context_dir: operator
dockerfile_path: bundle.Dockerfile
skip_building_index: true
substitutions:
- pullspec: quay.io/redhat-user-workloads/telco-5g-tenant/openperouter-operator-[\d].*
- pullspec: quay.io/openperouter/router:main
with: pipeline:openperouter-operator
promotion:
to:
Expand All @@ -47,6 +54,77 @@ resources:
requests:
cpu: 100m
memory: 200Mi
tests:
- as: operator-deploy-verify
steps:
allow_best_effort_post_steps: true
cluster_profile: openshift-org-aws
dependencies:
OO_BUNDLE: operator-bundle
env:
OO_INSTALL_MODE: AllNamespaces
OO_INSTALL_NAMESPACE: openshift-openperouter-system
test:
- as: verify
cli: latest
commands: |
set -exo pipefail

echo "### Label namespace privileged"
oc label --overwrite ns openshift-openperouter-system \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/audit=privileged \
pod-security.kubernetes.io/warn=privileged

echo "### Verify operator deployment"
oc wait --for condition=Available -n openshift-openperouter-system \
deployment --all --timeout=300s
oc get deployments -n openshift-openperouter-system

echo "### Create OpenPERouter CR"
cat <<'EOF' | oc apply -f -
apiVersion: network.openperouter.io/v1alpha1
kind: OpenPERouter
metadata:
name: openperouter
namespace: openshift-openperouter-system
spec:
logLevel: debug
EOF

echo "### Wait for controller daemonset"
until oc get daemonset controller -n openshift-openperouter-system &>/dev/null; do
sleep 5
done
oc rollout status daemonset/controller -n openshift-openperouter-system --timeout=300s

echo "### Wait for router daemonset"
until oc get daemonset router -n openshift-openperouter-system &>/dev/null; do
sleep 5
done
oc rollout status daemonset/router -n openshift-openperouter-system --timeout=300s
Comment on lines +95 to +105

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file exists and relevant lines =="
if [ -f ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml ]; then
  sed -n '1,140p' ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml | nl -ba | sed -n '80,115p'
else
  echo "missing target file"
fi

echo
echo "== kubectl/oc wait help if available =="
command -v oc || true
if command -v oc >/dev/null 2>&1; then
  oc wait --help | sed -n '1,160p'
fi
command -v kubectl || true
if command -v kubectl >/dev/null 2>&1; then
  kubectl wait --help | sed -n '1,160p'
fi

echo
echo "== search for wait_for_daemonset in repo =="
rg -n "wait_for_daemonset|for=create|oc wait" ci-operator/config/openshift-kni/openperouter ci-operator/config || true

Repository: openshift/release

Length of output: 232

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate target file =="
git ls-files | grep -F 'ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml' || true

echo
echo "== relevant target lines =="
awk '{printf "%4d\t%s\n", NR, $0}' ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml | sed -n '80,115p'

Repository: openshift/release

Length of output: 1879

🌐 Web query:

kubectl wait --for=create Kubernetes condition create support

💡 Result:

Yes, kubectl wait supports the --for=create option to wait for a resource to be created [1][2]. When you use kubectl wait --for=create [resource], the command will wait until the specified resource exists or until the defined timeout is reached [1][2]. This is particularly useful for workflows where you need to ensure a resource exists before proceeding to check its status [3]. Key details include: - Usage: You can combine it with other --for flags. For example, kubectl wait --for=create --for=condition=Ready [resource] will wait for the resource to be created and then for it to reach the Ready condition [1][3]. - Execution Order: If both --for=create and other conditions (like --for=condition=...) are provided, --for=create is always processed first [1][4]. - Label Selectors: The functionality supports label selectors, allowing you to wait for resources that match specific labels to be created [5]. - Default Behavior: If you do not provide --for=create, kubectl wait generally expects the target resource to already exist and may return an error if it cannot be found [5]. This feature was added to improve the reliability of scripts and CI/CD pipelines that depend on resources that might not be immediately available [6][3].

Citations:

Bound both DaemonSet creation waits.

Both until oc get daemonset loops can run indefinitely if the DaemonSet is not created. The 300-second oc rollout status timeout only starts after creation succeeds. Use a bounded creation wait with diagnostics on failure, and keep the rollout timeout for availability/progress.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml`
around lines 95 - 105, Bound the DaemonSet creation loops for controller and
router so they cannot run indefinitely before rollout status begins. Add a
finite timeout and failure diagnostics to each wait, while preserving the
existing 300-second oc rollout status timeout for availability and progress.


echo "### All pods"
oc get pods -n openshift-openperouter-system -o wide

echo "### Verify all containers are ready"
NOT_READY=$(oc get pods -n openshift-openperouter-system --no-headers \
| grep -v Completed | grep -v "Running" || true)
if [ -n "$NOT_READY" ]; then
echo "ERROR: Some pods are not healthy:"
echo "$NOT_READY"
oc describe pods -n openshift-openperouter-system
oc logs -n openshift-openperouter-system -l app=router --all-containers --tail=50
exit 1
fi
Comment on lines +110 to +119

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## Locate CI config"
git ls-files | rg 'ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml' || true

echo "## Relevant lines"
if [ -f ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml ]; then
  nl -ba ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml | sed -n '80,145p'
fi

echo "## Search rollout/status/health related patterns"
rg -n "rollout status|Verify all containers|openshift-openperouter-system|daemonset|desire|numberReady|desiredNumberScheduled|grep -v Running" ci-operator/config/openshift-kni/openperouter -S || true

echo "## Programmatic probe: current grep behavior for representative statuses"
python3 - <<'PY'
from subprocess import run

cases = {
    "empty": "",
    "running_ok": "openshift-openperouter-controller-controller-abcde   1/1     2/2     2     20s       Running",
    "running_unready": "openshift-openperouter-controller-controller-abcde   0/1     2/2     2     20s       Running",
    "mixed": """openshift-openperouter-controller-controller-abcde   0/1     2/2     2     20s       Running
openshift-openperouter-router-router-fghij   1/1     2/2     2     20s       Running""".replace("\n", " | "),
}
for name, lines in cases.items():
    args = ["grep", "-v", "Completed"]
    import pathlib
    p = pathlib.Path(f"/tmp/cases-{name}.txt")
    p.write_text(lines + "\n")
    # Simulate exactly the shell pipeline after file read, no oc/run outside.
    print(f"\nCASE {name}:")
    print("lines:", repr(lines))
    for label, cmd in [
        ("grep -v Completed", ["grep", "-v", "Completed"]),
        ("grep -v Running", ["grep", "-v", "Running"]),
        ("merged grep -v Completed | grep -v Running", ["bash", "-c", "grep -v Completed /tmp/cases-{}.txt | grep -v Running".format(name)]),
    ]:
        proc = run(cmd, text=True, input=lines if label.startswith("merged") else lines, capture_output=True)
        print(f"{label} => NOT_READY={bool(proc.stdout.strip())} stdout={proc.stdout.strip()!r}")
PY

echo "## Check git diff stat/name only for context"
git diff --stat || true

Repository: openshift/release

Length of output: 316

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## Relevant lines"
if [ -f ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml ]; then
  cat -n ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml | sed -n '1,160p'
fi

echo "## Search health/status patterns"
rg -n "rollout status|Verify all containers|openshift-openperouter-system|daemonset|desiredNumberScheduled|numberReady|grep -v Running|grep -v Completed" ci-operator/config/openswitch-kni/openperouter ci-operator/config/openshift-kni/openperouter -S || true

echo "## Programmatic probe: current grep behavior for representative statuses"
python3 - <<'PY'
from pathlib import Path
cases = {
    "empty": "",
    "running_ok": "openshift-openperouter-controller-controller-abcde   1/1     2/2     2     20s       Running",
    "running_unready": "openshift-openperouter-controller-controller-abcde   0/1     2/2     2     20s       Running",
    "mixed": """openshift-openperouter-controller-controller-abcde   0/1     2/2     2     20s       Running
openshift-openperouter-router-router-fghij   1/1     2/2     2     20s       Running""".replace("\n", " | "),
}
for name, lines in cases.items():
    print(f"\nCASE {name}:")
    print("lines:", repr(lines))
    import subprocess
    for label, cmd in [
        ("grep -v Completed", ["grep", "-v", "Completed"]),
        ("grep -v Running", ["grep", "-v", "Running"]),
    ]:
        proc = subprocess.run(cmd, text=True, input=lines, capture_output=True)
        print(f"{label} => stdout={proc.stdout.strip()!r}, exit={proc.returncode}")
PY

Repository: openshift/release

Length of output: 8905

Fail when DaemonSet Pods are not Ready or have no targets.

oc rollout status accepts a DaemonSet with desiredNumberScheduled=0, and the pod check only filters by Running; a pod with 0/1 READY passes. Check status.desiredNumberScheduled and status.numberReady for each DaemonSet, then reject empty readiness output after excluding Completed pods.

[low_effort_and_high_reward+]

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift-kni/openperouter/openshift-kni-openperouter-main.yaml`
around lines 110 - 119, Update the rollout verification around the pod readiness
check to validate every DaemonSet’s status.desiredNumberScheduled and
status.numberReady, failing when no targets are scheduled or the ready count is
insufficient. After excluding Completed pods, also reject empty or non-ready
readiness output so pods reporting 0/N READY cannot pass; preserve the existing
diagnostics before exiting.


echo "### Deploy verification passed"
from: src
resources:
requests:
cpu: 100m
memory: 200Mi
workflow: optional-operators-ci-operator-sdk-aws
zz_generated_metadata:
branch: main
org: openshift-kni
Expand Down
82 changes: 82 additions & 0 deletions ci-operator/jobs/openshift-kni/openperouter/openshift-kni-openperouter-main-presubmits.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,85 @@ presubmits:
secret:
secretName: result-aggregator
trigger: (?m)^/test( | .* )images,?($|\s.*)
- agent: kubernetes
always_run: true
branches:
- ^main$
- ^main-
cluster: build11
context: ci/prow/operator-deploy-verify
decorate: true
decoration_config: {}
labels:
ci-operator.openshift.io/cloud: aws
ci-operator.openshift.io/cloud-cluster-profile: openshift-org-aws
ci.openshift.io/generator: prowgen
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: pull-ci-openshift-kni-openperouter-main-operator-deploy-verify
path_alias: github.com/openperouter/openperouter
rerun_command: /test operator-deploy-verify
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=operator-deploy-verify
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: ci-pull-credentials
secret:
secretName: ci-pull-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
trigger: (?m)^/test( | .* )operator-deploy-verify,?($|\s.*)
1 change: 1 addition & 0 deletions ci-operator/step-registry/cluster-profiles/cluster-profiles-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ cluster_profiles:
repos:
- baremetal-deploy
- lifecycle-agent
- openperouter
- org: openshift-pipelines
repos:
- console-plugin
Expand Down