Test timeout client

[Jakob-Naucke]

follow up #313

Summary by Sourcery

Introduce a Kubernetes client with read timeouts and refactor operator, test utilities, and controllers to use separate clients for request/response calls and long‑lived watches, improving reliability of integrations and controllers.

New Features:

• Add must-gather collection and artifact upload to integration test workflow for post-failure diagnostics.

Bug Fixes:

• Handle errors during secret unmount cleanup in the keygen controller by surfacing them as controller errors instead of silently continuing.
• Avoid updating attestation key volumes when a Machine is already being deleted to prevent unnecessary reconciliation work.

Enhancements:

• Add a timed Kubernetes client with a bounded read timeout for request/response operations and keep a separate client without timeouts for watches.
• Refactor operator controllers, reflectors, and test utilities to use the dual Kubernetes clients (request vs. watch) to prevent watch stream timeouts and improve cache reliability.
• Add UTC timestamps to test info/warn logs to make debugging long-running integration tests easier.
• Increase some test cleanup and namespace-deletion timeouts to reduce flakiness under slower environments.
• Wire reference value image and job controllers and register-server keygen controller to use the new client pair and correct ownership/watch relationships.
• Update logging in the attestation key register controller to use structured warnings instead of bare stderr prints.

Build:

• Update default test VM image reference in the Makefile to a newer Fedora CoreOS kubevirt image.
• Enable the humantime feature on env_logger for improved log formatting.

CI:

• Adjust integration test workflow to use a fixed image registry/tag, remove the job timeout, repeatedly run integration tests until failure, and collect/upload must-gather diagnostics on completion.

Deployment:

• Remove the local registry push step from integration tests, relying on prebuilt images from a remote registry instead.

Tests:

• Refactor integration and attestation tests to construct and use the new dual Kubernetes clients, separating watch APIs from standard client calls and aligning with the new timeout behavior.
• Update virt/kubevirt test backend to use the new client pair, including dedicated watch client for VM readiness polling.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jakob-Naucke

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sourcery-ai]

Reviewer's Guide

Introduce a timed Kubernetes client alongside a separate watch client, refactor operator/tests/VM helpers to use the dual-client model for timeouts vs watches, improve logging and test diagnostics, and adjust CI and images for more reliable integration tests.

Sequence diagram for controller using watch and timed clients

sequenceDiagram
    participant OperatorMain
    participant LibRoot
    participant ReferenceValues
    participant Controller
    participant KubernetesAPI

    OperatorMain->>LibRoot: timed_client()
    LibRoot-->>OperatorMain: Client (timed)
    OperatorMain->>KubernetesAPI: Client::try_default()
    KubernetesAPI-->>OperatorMain: Client (watch)
    OperatorMain->>ReferenceValues: launch_rv_image_controller(clients)

    ReferenceValues->>KubernetesAPI: Api::default_namespaced(clients.watch)
    ReferenceValues->>Controller: Controller::new(images, Config::default())
    ReferenceValues->>Controller: owns(jobs, watcher::Config)
    ReferenceValues->>Controller: run(image_reconcile, controller_error_policy, Arc::new(clients.client))

    loop watch stream
        Controller->>KubernetesAPI: watch(images) via clients.watch
        Controller-->>ReferenceValues: image_reconcile(job, Arc<Client>)
        ReferenceValues->>KubernetesAPI: request/response via clients.client
    end

Loading

File-Level Changes

Change	Details	Files
Introduce timed Kubernetes client and dual-client abstraction for separating read-timeout and watch workloads.	Add KUBE_READ_TIMEOUT constant and timed_client() helper creating a Client with read_timeout configured. Define KubeClients struct bundling a timed request client and a non-timeout watch client. Update setup_test_clients() and TestContext to construct and expose KubeClients instead of a single Client. Switch various helpers like get_cluster_url, virt::VmConfig, and virt backends to accept/use KubeClients rather than bare Client instances.	lib/src/lib.rs test_utils/src/lib.rs test_utils/src/virt/mod.rs test_utils/src/virt/kubevirt.rs tests/attestation.rs
Refactor controllers and operator components to consistently use timed clients for mutations and non-timeout clients for controllers/reflectors.	Change AkContextData, ClusterContext, and controller launch functions to accept/use KubeClients and wire watch_client vs client appropriately for Api usage and reflectors. Update operator main to construct KubeClients via timed_client and Client::try_default and pass them into controllers instead of a single Client. Adjust reference_values controllers to accept KubeClients, using watch client for watches and timed client for reconcile operations. Modify register-server and attestation-key-register binaries to use timed_client for their Kubernetes client state.	operator/src/attestation_key_register.rs operator/src/main.rs operator/src/reference_values.rs operator/src/register_server.rs attestation-key-register/src/main.rs compute-pcrs/src/main.rs register-server/src/main.rs
Update tests to use dedicated watch clients for await_condition/wait_for_resource_* and extend some timeouts for stability.	Expose watch_client and clients accessors from TestContext and update tests to use watch client for await_condition and wait_for_resource_deleted/created calls. Adjust various Api instantiations in tests to use clients.watch for watch-based conditions and clients.client for direct CRUD operations. Increase cleanup timeout in TestContext::cleanup and some namespace deletion/other waits to reduce flakiness.	test_utils/src/lib.rs tests/trusted_execution_cluster.rs tests/attestation.rs
Improve logging and diagnostics for tests and operator.	Prefix test_info and test_warn macro logging with a UTC timestamp via new log_time helper. Enable env_logger humantime feature to improve log timestamp formatting. Change some error prints in secret reconcile to use log::warn instead of eprintln, and tweak a machine deletion log message to accurately describe skipped work.	test_utils/src/lib.rs Cargo.toml operator/src/attestation_key_register.rs
Adjust CI integration workflow, images, and Makefile defaults to support external registry and better failure diagnostics.	Change default REGISTRY and TAG env in integration-tests workflow, remove explicit job timeout, and loop make integration-tests until failure then run must-gather and upload artifacts. Update TEST_IMAGE and APPROVED_IMAGE defaults in Makefile to new image tags/digests. Add a must-gather/gather script placeholder (or updates) to support workflow’s must-gather step.	.github/workflows/integration-tests.yml Makefile must-gather/gather

Possibly linked issues

• #(unknown): PR introduces separate watch client and timeout handling to fix delayed AttestationKey approvals suspected from stale caches/timeouts.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[Jakob-Naucke]

Failure brought up in #248 (comment) but after 5h of successful tests I've seen enough here