STOR-1459: Add fine-grained RBAC roles instead of cluster-admin role for cluster-storage-operator

AGENTS.md

@@ -172,7 +172,38 @@ A PR review **must fail** if any of the following are true:
 ### RBAC Changes
 
 - RBAC files under `assets/csidriveroperators/<driver>/base/` govern the CSI driver *operator*, not the driver itself. Verify the subject, namespace, and rules match the minimum needed.
-- CSO itself currently runs with `cluster-admin` (`manifests/08_operator_rbac.yaml`) — this is a known open TODO, not a new concern introduced by a PR.
+- **CSO Operator RBAC** — CSO runs with fine-grained Role-based RBAC instead of `cluster-admin`:
+  - **Cluster-scoped permissions** — `manifests/08_0001_cluster_storage_operator_storage_role.yaml` contains a ClusterRole with 7 rules for cluster-wide resources:
+    - Cluster configuration observation (Infrastructure, Proxies, ClusterVersions, FeatureGates, ClusterOperators, APIServers)
+    - StorageClass and CSI driver management (storageclasses, csidrivers, volumeattributesclasses)
+    - ClusterRoleBindings for CSI sidecar RBAC (provisioner, attacher, resizer, snapshotter)
+    - Operator custom resources (Storage, ClusterCSIDriver CRs and status subresources)
+    - CustomResourceDefinitions management (for VolumePopulator CRD)
+    - ClusterOperator status updates (for health condition reporting)
+    - Monitoring resources (cluster-scoped ServiceMonitors and PrometheusRules)
+  - **Namespace-scoped permissions** — 4 namespace-specific Roles for fine-grained isolation:
+    - `manifests/08_0002_cluster_storage_operator_operator_namespace_role.yaml` in `openshift-cluster-storage-operator`: leases (leader election), events, serviceaccounts, services, configmaps, deployments, replicasets, roles/rolebindings, vSphere secrets (resourceNames-restricted), monitoring
+    - `manifests/08_0003_cluster_storage_operator_csi_drivers_namespace_role.yaml` in `openshift-cluster-csi-drivers`: deployments, configmaps, RBAC (roles/rolebindings), serviceaccounts, services, replicasets, events, monitoring
+    - `manifests/08_0004_cluster_storage_operator_config_namespace_role.yaml` in `openshift-config`: configmaps (read-only)
+    - `manifests/08_0005_cluster_storage_operator_config_managed_namespace_role.yaml` in `openshift-config-managed`: configmaps, RBAC (read-only)
+  - **RoleBindings** — `manifests/08_0010_cluster_storage_operator_rolebindings.yaml` contains 4 RoleBindings binding the cluster-storage-operator ServiceAccount to the above Roles
+  - **Key improvements:**
+    - **Least privilege by scope:** Namespace operations use namespace-scoped Roles, not cluster-wide ClusterRole
+    - **Secrets restricted:** Only 2 specific vSphere secrets readable (vsphere-cloud-credentials, vsphere-problem-detector-serving-cert) via resourceNames restriction
+    - **Removed unnecessary permissions:** Pods removed (informers cache without RBAC), Events scoped to namespaces (not cluster-wide)
+    - **Clear justification:** Each permission rule includes a comment explaining why it's necessary
+  - **Standalone path** (`manifests/08_operator_rbac.yaml`):
+    - Contains a deletion manifest (with `release.openshift.io/delete: "true"` annotation) pointing to the old `cluster-admin` binding
+    - Followed by a replacement ClusterRoleBinding pointing to `cluster-storage-operator-storage-role`
+    - CVO processes deletion first, then applies the replacement, avoiding immutable `roleRef` conflicts during upgrades
+  - **HyperShift path** (`manifests/08_operator_rbac-hypershift.yaml` + `manifests/08_0002_operator_rbac-hypershift-delete.yaml`):
+    - `08_operator_rbac-hypershift.yaml` is autogenerated via `make update` and must not be hand-edited
+    - `08_0002_operator_rbac-hypershift-delete.yaml` is a hand-maintained deletion manifest with `release.openshift.io/delete: "true"` annotation
+    - CVO processes both: deletion first from the delete manifest, then applies the autogenerated binding
+  - `manifests/08_operator_scc.yaml` — ClusterRoleBinding granting the `nonroot-v2` SecurityContextConstraint (required for pod scheduling).
+  - Permissions were discovered incrementally by deploying with empty RBAC and capturing "forbidden" errors from operator logs, ensuring only necessary permissions are granted.
+  - **Immutable field migration strategy:** ClusterRoleBinding `roleRef` is immutable after creation. Changing it requires deletion and replacement in the same CVO update cycle using `release.openshift.io/delete: "true"` annotation on the old binding.
+  - When reviewing RBAC changes, verify verbs are minimal and necessary — do not use `["*"]` for new permissions. Use resourceNames for secrets access when possible.
 - Sidecar RBAC for provisioner / attacher / resizer / snapshotter lives in `manifests/09_sidecar-*.yaml` and is shared across all drivers. Changes there affect every driver simultaneously.
 
 ### Adding a New CSI Driver

manifests/08_0001_cluster_storage_operator_storage_role.yaml

@@ -0,0 +1,55 @@
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-storage-role
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+rules:
+  # Cluster infrastructure and configuration observation (cluster-scoped singletons)
+  # Required for determining platform capabilities, proxy settings, feature gates, and cluster health
+  - apiGroups: ["config.openshift.io"]
+    resources: ["infrastructures", "proxies", "clusterversions", "featuregates", "clusteroperators", "apiservers"]
+    verbs: ["get", "list", "watch"]
+
+  # Default StorageClass management — CSO creates/reconciles the cluster default SC
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # CSI driver / VAC observation only — CSO watches these to make decisions and
+  # emit metrics; the objects themselves are managed by the CSI driver operators.
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["csidrivers", "volumeattributesclasses"]
+    verbs: ["get", "list", "watch"]
+
+  # ClusterRoleBindings for CSI driver sidecar RBAC (cluster-scoped)
+  # Required to grant sidecar containers (provisioner, attacher, resizer, snapshotter) cluster-level permissions
+  # Note: Roles and RoleBindings in namespaces are handled by namespace-scoped Roles
+  - apiGroups: ["rbac.authorization.k8s.io"]
+    resources: ["clusterrolebindings"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Storage operator custom resources (cluster-scoped)
+  # Required for managing the Storage CR and ClusterCSIDriver CRs for each platform
+  - apiGroups: ["operator.openshift.io"]
+    resources: ["storages", "clustercsidrivers"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["operator.openshift.io"]
+    resources: ["storages/status"]
+    verbs: ["get", "patch", "update"]
+
+  # Custom Resource Definitions management (cluster-scoped)
+  # Required for VolumeDataSourceValidator to deploy the VolumePopulator CRD
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # ClusterOperator status updates (cluster-scoped)
+  # Required by StatusSyncer controller to report operator health conditions
+  - apiGroups: ["config.openshift.io"]
+    resources: ["clusteroperators/status"]
+    verbs: ["patch", "update"]

manifests/08_0002_cluster_storage_operator_operator_namespace_role.yaml

@@ -0,0 +1,67 @@
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-role
+  namespace: openshift-cluster-storage-operator
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+rules:
+  # Leader election via leases in operator's own namespace
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Event recording for operator status and lifecycle
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["create", "update", "patch"]
+
+  # ServiceAccount management for vSphere problem detector
+  - apiGroups: [""]
+    resources: ["serviceaccounts"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Service management for metrics endpoint
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # CSO informs on all Secrets in its own namespace (informer requires list/watch,
+  # which already exposes full secret contents); vsphere-problem-detector reads
+  # vsphere-cloud-credentials and vsphere-problem-detector-serving-cert by name.
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "watch"]
+
+  # Monitoring resources for operator metrics
+  - apiGroups: ["monitoring.coreos.com"]
+    resources: ["servicemonitors", "prometheusrules"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Deployment management for VolumeDataSourceValidator (library-go DeploymentController:
+  # Get/Create/Update/Delete on the Deployment; status is read via this rule, scale is unused)
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # ConfigMap management for operator configuration
+  # Required by ConfigObserver and other controllers that manage operator configuration data
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # RBAC management in operator namespace
+  # Required to create and manage Roles and RoleBindings for operator components and sidecars
+  - apiGroups: ["rbac.authorization.k8s.io"]
+    resources: ["roles", "rolebindings"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # ReplicaSet observation for deployment status tracking
+  # Required to watch ReplicaSets created by Deployments for accurate status reporting
+  - apiGroups: ["apps"]
+    resources: ["replicasets"]
+    verbs: ["get", "list", "watch"]

manifests/08_0002_operator_rbac-hypershift-delete.yaml

@@ -0,0 +1,16 @@
+# Deletion manifest for old cluster-admin binding in HyperShift — marked for deletion to avoid immutable roleRef conflict
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: cluster-storage-operator-role
+  annotations:
+    release.openshift.io/delete: "true"
+    include.release.openshift.io/hypershift: "true"
+subjects:
+  - kind: ServiceAccount
+    name: cluster-storage-operator
+    namespace: openshift-cluster-storage-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin

manifests/08_0003_cluster_storage_operator_csi_drivers_namespace_role.yaml

@@ -0,0 +1,56 @@
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-csi-drivers-role
+  namespace: openshift-cluster-csi-drivers
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+rules:
+  # ReplicaSet observation for owner reference resolution in deployment controller
+  - apiGroups: ["apps"]
+    resources: ["replicasets"]
+    verbs: ["get", "list", "watch"]
+
+  # Event recording for CSI driver operator lifecycle events
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["create", "update", "patch"]
+
+  # Deployment management for CSI driver operators (DeploymentController)
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # ConfigMap management for driver operator configuration
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # RBAC management for CSI driver operators (roles and rolebindings)
+  - apiGroups: ["rbac.authorization.k8s.io"]
+    resources: ["roles", "rolebindings"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # ServiceAccount management for CSI driver operators
+  - apiGroups: [""]
+    resources: ["serviceaccounts"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Service management for CSI driver metrics and communication
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Monitoring resources for CSI driver metrics
+  - apiGroups: ["monitoring.coreos.com"]
+    resources: ["servicemonitors", "prometheusrules"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+  # Secrets for CSI driver operators (e.g., cloud provider credentials)
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "watch"]

manifests/08_0004_cluster_storage_operator_config_namespace_role.yaml

@@ -0,0 +1,17 @@
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-config-role
+  namespace: openshift-config
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+rules:
+  # ConfigMap observation for cluster configuration (e.g., cloud provider config)
+  # Read-only access to configuration data
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list", "watch"]

manifests/08_0005_cluster_storage_operator_config_managed_namespace_role.yaml

@@ -0,0 +1,17 @@
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-config-managed-role
+  namespace: openshift-config-managed
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+rules:
+  # ConfigMap observation for managed cluster configuration (CA bundle sync
+  # via resourcesynccontroller). Read-only.
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list", "watch"]

manifests/08_0010_cluster_storage_operator_rolebindings.yaml

@@ -0,0 +1,84 @@
+---
+# RoleBinding for openshift-cluster-storage-operator namespace
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-role
+  namespace: openshift-cluster-storage-operator
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+subjects:
+  - kind: ServiceAccount
+    name: cluster-storage-operator
+    namespace: openshift-cluster-storage-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cluster-storage-operator-role
+---
+# RoleBinding for openshift-cluster-csi-drivers namespace
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-csi-drivers-role
+  namespace: openshift-cluster-csi-drivers
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+subjects:
+  - kind: ServiceAccount
+    name: cluster-storage-operator
+    namespace: openshift-cluster-storage-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cluster-storage-operator-csi-drivers-role
+---
+# RoleBinding for openshift-config namespace
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-config-role
+  namespace: openshift-config
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+subjects:
+  - kind: ServiceAccount
+    name: cluster-storage-operator
+    namespace: openshift-cluster-storage-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cluster-storage-operator-config-role
+---
+# RoleBinding for openshift-config-managed namespace
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: cluster-storage-operator-config-managed-role
+  namespace: openshift-config-managed
+  annotations:
+    include.release.openshift.io/hypershift: "true"
+    include.release.openshift.io/ibm-cloud-managed: "true"
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"
+    capability.openshift.io/name: Storage
+subjects:
+  - kind: ServiceAccount
+    name: cluster-storage-operator
+    namespace: openshift-cluster-storage-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cluster-storage-operator-config-managed-role

manifests/08_operator_rbac-hypershift.yaml

@@ -2,11 +2,9 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
-  labels:
-    addonmanager.kubernetes.io/mode: Reconcile
-    kubernetes.io/cluster-service: "true"
   name: cluster-storage-operator-role
 roleRef:
+  apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: cluster-admin
 subjects: