From a1f76a9cf1a530b7d2d554f44d68f12160b7c775 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 26 Aug 2026 13:12:49 -0700 Subject: [PATCH 1/2] Set the project on the composed ProjectIAMMember Provisioning a GKE InferenceCluster failed at the IAM binding: Request `Create IAM Members roles/container.admin serviceAccount:...@.iam.gserviceaccount.com for project ""` returned error: Error retrieving IAM policy for project "": googleapi: got HTTP response code 404 `compose_iam_binding` built `forProvider` from `role` and `member` only, on the assumption -- written into `resolve_project`'s docstring -- that the provider defaults `projectID` from the ProviderConfig for every managed resource. It does for the Network, Subnetwork and ServiceAccount, which all come back with `project` late-initialized into their specs. It does not for ProjectIAMMember, which creates with an empty project first. `project` is optional on the CRD, so nothing catches the omission before the GCP call goes out. The XR never leaves `Unready resources: gke-cluster` as a result, so BackendReady never installs the serving stack and the InferenceCluster never reaches Ready. `self.project` is already resolved for the workloadPool, so this passes the same value. The test encoded the broken shape, asserting a forProvider of exactly `role` and `member`, so it is updated rather than added to. Two things found alongside this while provisioning a real cluster: The GCP roles a credential needs were undocumented. All seven are now listed in build-the-platform, with a note that `resourcemanager.projectIamAdmin` is required because Modelplane grants the node service account `container.admin` -- worth knowing before handing a key over. The local e2e control plane cannot provision cloud clusters, and says so now. Both of `lean-control-plane.yaml`'s trims block it silently: unactivated managed resources report no conditions at all, and the zero-replica ImageConfig is not undone by editing the ImageConfig alone. Signed-off-by: Dennis Ramdass --- .../getting-started/build-the-platform.md | 17 ++++++++++++++-- e2e/README.md | 20 +++++++++++++++++++ functions/compose-gke-cluster/function/fn.py | 14 ++++++++++--- .../compose-gke-cluster/tests/test_fn.py | 1 + 4 files changed, 47 insertions(+), 5 deletions(-) diff --git a/docs/content/getting-started/build-the-platform.md b/docs/content/getting-started/build-the-platform.md index 25e3b08c..09658cbd 100644 --- a/docs/content/getting-started/build-the-platform.md +++ b/docs/content/getting-started/build-the-platform.md @@ -19,8 +19,21 @@ against this capacity without knowing which cluster it runs on. - AWS access key ID and secret access key {{< /tab >}} {{< tab "GKE" >}} -- A GCP account with permissions to create GKE clusters, VPCs, and IAM roles -- A GCP service account JSON key +- A GCP service account JSON key, granted these roles on the project: + + | Role | Needed for | + |---|---| + | `roles/container.admin` | the cluster and its node pools | + | `roles/compute.admin` | the VPC network and subnetwork | + | `roles/serviceusage.serviceUsageAdmin` | enabling the APIs the cluster needs | + | `roles/iam.serviceAccountAdmin` | the node service account | + | `roles/iam.serviceAccountKeyAdmin` | the node service account's key | + | `roles/iam.serviceAccountUser` | attaching that account to the nodes | + | `roles/resourcemanager.projectIamAdmin` | granting the node account `container.admin` | + + The last one is worth a look before you hand the key over: Modelplane grants + the node service account `roles/container.admin`, so the credential doing the + provisioning has to be able to set project IAM policy. {{< /tab >}} {{< tab "AKS" >}} - An Azure account with permissions to create AKS clusters and managed identities diff --git a/e2e/README.md b/e2e/README.md index d915e428..c7aff122 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -51,6 +51,26 @@ server exposes both, so the pod goes Ready without a real model or GPU. | Control-plane `InferenceGateway` + cross-cluster routing to the replica | | | Status propagation and foreground-deletion ordering | | +### Why cloud provisioning cannot be tested here + +`lean-control-plane.yaml` trims the control plane to what a BYO cluster needs, +and both trims stop a cloud `InferenceCluster` from reconciling at all. Neither +announces itself, so this is what to expect if you point this control plane at a +real cloud: + +- The MRAP activates only `*.kubernetes.m.crossplane.io` and + `*.helm.m.crossplane.io`. A cloud provider's managed resources are then never + activated, so they sit with **no status conditions at all** — which reads as + nothing happening rather than as an error. +- The `dormant-cloud-providers` `ImageConfig` maps every + `xpkg.upbound.io/upbound/provider-*` to a zero-replica runtime config. Editing + that `ImageConfig` is not enough on its own: it is resolved when a package + revision reconciles, so existing Deployments keep their replica count until + something scales them. + +Undoing both is possible but leaves a control plane that is no longer the one CI +runs, so prefer a separate control plane for cloud work. + ## Prerequisites - **Docker** with real headroom — **≥ 16 GB memory** and **plenty of disk** diff --git a/functions/compose-gke-cluster/function/fn.py b/functions/compose-gke-cluster/function/fn.py index 9357df4b..b7dd834e 100644 --- a/functions/compose-gke-cluster/function/fn.py +++ b/functions/compose-gke-cluster/function/fn.py @@ -175,9 +175,11 @@ def _cred_name(self) -> str: def resolve_project(self) -> str | None: """Fetch the GCP provider config and return its projectID. - The GCP provider uses projectID from the ProviderConfig/ClusterProviderConfig - as the default for all managed resources, so we only need the value - explicitly for the workloadPool string in the GKE cluster spec. + The GCP provider late-initializes projectID from the + ProviderConfig/ClusterProviderConfig into most managed resources, so the + value is needed explicitly only where that does not happen: the + workloadPool string in the GKE cluster spec, and the ProjectIAMMember, + which creates with an empty project unless it is set. When the ProviderConfig is transiently gone but the cluster already exists, falls back to the project embedded in the observed cluster's @@ -506,6 +508,12 @@ def compose_iam_binding(self) -> None: forProvider=iamv1beta1.ForProvider( role="roles/container.admin", member=f"serviceAccount:{sa_email}", + # Set explicitly. The provider late-initializes `project` + # from the ProviderConfig for the Network, Subnetwork and + # ServiceAccount, but not for this resource: it goes to + # create with an empty project and the GCP call 404s on + # "Error retrieving IAM policy for project ''". + project=self.project, ), ), ), diff --git a/functions/compose-gke-cluster/tests/test_fn.py b/functions/compose-gke-cluster/tests/test_fn.py index 09813e66..a351bd16 100644 --- a/functions/compose-gke-cluster/tests/test_fn.py +++ b/functions/compose-gke-cluster/tests/test_fn.py @@ -274,6 +274,7 @@ def _iam_binding( "forProvider": { "role": "roles/container.admin", "member": f"serviceAccount:{sa_email}", + "project": "my-gcp-project", }, }, } From 21a6c8275c2dd4ae261e0fd007c8ba7e4562eda4 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Thu, 27 Aug 2026 10:01:04 -0700 Subject: [PATCH 2/2] Fix the Vale alerts in the new GKE prerequisites The docs-vale check fails on the added GKE role table: build-the-platform.md 27:49 error Spelling check: "subnetwork"? Modelplane.Spelling "Subnetwork" is the name of the Crossplane managed resource, but the row describes what the role is needed for, not which resource is composed, and GCP's own docs call it a subnet. The vocabulary already accepts "subnet", so this uses the word rather than growing the accept list. The note under the table also opened a sentence after a colon with a proper noun, which ai-tells.ColonUsage flags. It is a full sentence, so it takes a period. Signed-off-by: Dennis Ramdass --- docs/content/getting-started/build-the-platform.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/getting-started/build-the-platform.md b/docs/content/getting-started/build-the-platform.md index 09658cbd..f239bd15 100644 --- a/docs/content/getting-started/build-the-platform.md +++ b/docs/content/getting-started/build-the-platform.md @@ -24,14 +24,14 @@ against this capacity without knowing which cluster it runs on. | Role | Needed for | |---|---| | `roles/container.admin` | the cluster and its node pools | - | `roles/compute.admin` | the VPC network and subnetwork | + | `roles/compute.admin` | the VPC network and subnet | | `roles/serviceusage.serviceUsageAdmin` | enabling the APIs the cluster needs | | `roles/iam.serviceAccountAdmin` | the node service account | | `roles/iam.serviceAccountKeyAdmin` | the node service account's key | | `roles/iam.serviceAccountUser` | attaching that account to the nodes | | `roles/resourcemanager.projectIamAdmin` | granting the node account `container.admin` | - The last one is worth a look before you hand the key over: Modelplane grants + The last one is worth a look before you hand the key over. Modelplane grants the node service account `roles/container.admin`, so the credential doing the provisioning has to be able to set project IAM policy. {{< /tab >}}