Support multiple controller replicas for HA (split gRPC services from reconcilers)

[mangelajo]

Problem

The controller currently cannot run with more than 1 replica because it uses in-memory state for gRPC stream coordination:

• listenQueues (sync.Map) — pairs Dial and Listen calls via in-memory channels. When an exporter calls Listen, it blocks on a channel. When a client calls Dial, it sends a JWT token to that channel. If Dial and Listen land on different pods, the connection pairing breaks silently.
• leaseLocks (sync.Map) — ref-counted mutexes for lease operations, only valid within a single process.

Additionally, the ControllerService writes directly to the K8s API server (patches Exporters, creates Leases, rotates Secrets), and the Status streaming method patches Exporter.status.lastSeen every 10 seconds. Multiple replicas would create write conflicts.

The ControllerService does not implement NeedLeaderElection(), so it defaults to true (only runs on the leader). This means with multiple replicas, only the leader serves gRPC traffic, but:

1. Non-leader pods report ready (with healthz.Ping), so K8s routes ~50% of gRPC traffic to pods that aren't serving — causing connection refused errors
2. If readiness is tied to leader election (fix: tie readiness probe to leader election in all controllers #1012), rolling updates deadlock because K8s can't terminate the leader (it's the only available pod) and new pods can't become ready (not leader)

Current Mitigation

The operator now clamps controller.replicas to 1 with a warning event. See the PR that introduced this.

Proposed Solution

Split the controller into separate concerns:

1. Reconciler deployment (needs leader election, 1 active replica)

   • ExporterReconciler, ClientReconciler, LeaseReconciler
   • Only the leader reconciles; standby replicas are passive

2. gRPC service deployment (stateless, multiple replicas)

   • ControllerService and ClientService gRPC handlers
   • Move Dial/Listen coordination to an external store (Redis, or a K8s-native mechanism like a ConfigMap-based rendezvous, or keep a single-writer pattern)
   • Read-only methods (ListLeases, GetExporter, etc.) can safely run on any replica
   • Write methods use the K8s API server directly (optimistic concurrency handles conflicts)

3. Login/OIDC/Dashboard (already stateless, already NeedLeaderElection() → false)

   • These already run on all replicas, no changes needed

Alternative: keep single deployment, fix stream coordination

Instead of splitting, the Dial/Listen rendezvous could be moved to a shared data store or use the K8s API server itself (e.g., create a short-lived CR or annotation that the exporter watches). This avoids a new deployment but adds complexity.

References

• fix: tie readiness probe to leader election in all controllers #1012 — readiness probe tied to leader election (exposed the HA issues)
• fix(e2e): wait for controller leader election before running tests #1002 — E2E race condition with leader election