From c0e276fbd61145b11a39ec617e37da06e12453b7 Mon Sep 17 00:00:00 2001 From: Ramkumar Chinchani Date: Mon, 17 Aug 2026 19:29:01 -0700 Subject: [PATCH 1/4] docs: propose GEP-1748 Gateway API integration Define a fresh multi-cluster global ingress architecture that uses GatewayClass, Gateway, and HTTPRoute while preserving the existing networking.fleet.azure.com/v1alpha1 ServiceImport API. Align the Fleet hub configuration-cluster model with GKE multi-cluster Gateway guidance while retaining Fleet-specific ServiceImport support. Document environment prerequisites, ownership, status, fail-static behavior, repository placement, feasibility, and the phased tests-first implementation plan. Define optional annotations for Azure Front Door SKU, WAF attachment, health probes, host headers, and public or Private Link connectivity. Clarify that unknown or invalid explicit annotations must fail rather than silently falling back to defaults. Add separate operational guides for public LoadBalancer backends and AFD Premium Private Link to internal LoadBalancer and PLS backends. Identify the actor and placement for every hub, member-cluster, Azure, WAF, PLS approval, DNS, traffic-weight, and cleanup action. Document that the private AFD plus optional WAF plus PLS topology is intended to support SFI-NS253, while public member LoadBalancer origins are not that topology. Do not claim compliance until Private Link reconciliation, enforcement, and end-to-end evidence are complete. Pin references to the Gateway API v1.2.1 type definitions used by the implementation and keep the public documentation free of internal URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d8af5d6-9aba-40a2-8efe-9bf09675667b --- .../2026-08-17-2356-gep-1748-gateway-api.md | 189 ++++ docs/design/gep-1748-gateway-api.md | 927 ++++++++++++++++++ docs/design/gep-1748-implementation-plan.md | 602 ++++++++++++ docs/howtos/gateway-api-afd-configuration.md | 581 +++++++++++ 4 files changed, 2299 insertions(+) create mode 100644 .github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md create mode 100644 docs/design/gep-1748-gateway-api.md create mode 100644 docs/design/gep-1748-implementation-plan.md create mode 100644 docs/howtos/gateway-api-afd-configuration.md diff --git a/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md b/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md new file mode 100644 index 00000000..011ea9e7 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md @@ -0,0 +1,189 @@ +# GEP-1748 Gateway API Design + +## Requirements + +- Produce a fresh design focused on Gateway API and the GEP-1748 interaction model. +- Keep `networking.fleet.azure.com/v1alpha1 ServiceImport` unchanged. +- Use optional annotations for Azure Front Door, WAF, and Private Link configuration. +- Identify feasibility and repository ownership. +- Produce a detailed, tests-first implementation plan. +- Commit the documentation on a completely new pull request branch using Conventional Commits. + +## Additional comments from user + +- The design must not migrate to `multicluster.x-k8s.io/ServiceImport`. +- The public ingress API must not depend on the prior `FrontDoor*` CRD proposal. +- The working branch is `rchinchani/gep-1748-gateway-api`, created from the latest `origin/main`. +- The architecture should align with GKE's multi-cluster Gateway requirements where the concepts + apply to Fleet and Azure. +- The user approved proceeding with implementation after the design commit. + +## Plan + +### Phase 1: Verify current contracts + +- [x] **Task 1.1: Review repository instructions and documentation conventions.** + - Success criteria: repository-specific design, planning, and breadcrumb requirements are + reflected in the documentation. +- [x] **Task 1.2: Verify ServiceImport and internal export behavior.** + - Success criteria: the design accurately preserves the existing ServiceImport group, version, + schema, status, and aggregation behavior. + +### Phase 2: Create design documentation + +- [x] **Task 2.1: Write the architecture design.** + - Success criteria: the document covers API boundaries, GEP compatibility, annotations, + controllers, AFD/WAF/PLS behavior, status, ownership, security, feasibility, and repository + placement. +- [x] **Task 2.2: Write the implementation plan.** + - Success criteria: the plan is phased, tests-first, independently reviewable, and includes + measurable completion criteria. + +### Phase 3: Validate and commit + +- [x] **Task 3.1: Review documentation consistency and git diff.** + - Success criteria: links, examples, terminology, and referenced repository paths are coherent, + with no unrelated changes. +- [x] **Task 3.2: Commit the design package.** + - Success criteria: the design, plan, and breadcrumb are committed with a detailed Conventional + Commit message and required trailers. + +### Phase 4: Implement the Gateway foundation + +- [x] **Task 4.1: Add Gateway API dependency and scheme tests.** + - Success criteria: Gateway, HTTPRoute, ReferenceGrant, and the unchanged Fleet ServiceImport GVK + are registered. +- [x] **Task 4.2: Add typed AFD annotation parsing.** + - Success criteria: defaults, supported values, misspellings, malformed values, reserved keys, + and cross-resource compatibility are tested. +- [x] **Task 4.3: Add normalized model primitives.** + - Success criteria: listeners, routes, filters, Fleet ServiceImport backends, origins, probes, + WAF, deterministic ordering, and weight validation are represented. +- [x] **Task 4.4: Add the separate hub Gateway manager skeleton.** + - Success criteria: the binary registers schemes, validates controller-wide Azure configuration, + exposes health/readiness, uses repository-standard structured logging, and registers no + reconcilers while the feature is disabled. +- [x] **Task 4.5: Add public and private configuration documentation.** + - Success criteria: separate subsections identify actors and placement for public AFD origins, + optional WAF, internal load balancers, PLS, approval, and cleanup. +- [x] **Task 4.6: Run focused validation.** + - Success criteria: unit tests, build, vet, dependency tidy, and diff checks pass. Race validation + is delegated to CI because this Windows host does not have a C compiler. + +### Detailed checklist + +- [x] Phase 1 / Task 1.1 completed. +- [x] Phase 1 / Task 1.2 completed. +- [x] Phase 2 / Task 2.1 completed. +- [x] Phase 2 / Task 2.2 completed. +- [x] Phase 3 / Task 3.1 completed. +- [x] Phase 3 / Task 3.2 completed. +- [x] Phase 4 / Task 4.1 completed. +- [x] Phase 4 / Task 4.2 completed. +- [x] Phase 4 / Task 4.3 completed. +- [x] Phase 4 / Task 4.4 completed. +- [x] Phase 4 / Task 4.5 completed. +- [x] Phase 4 / Task 4.6 completed. + +### Overall success criteria + +- The design is independent of prior AFD CRD branches. +- Fleet ServiceImport remains unchanged. +- Gateway API is the only public AFD routing API. +- Azure-specific options are optional annotations. +- Formal upstream GEP conformance limitations are explicit. +- The implementation plan can be delivered as focused follow-up pull requests. + +## Decisions + +- Keep the Fleet ServiceImport API because existing controllers, users, and status consumers already + depend on its group and cluster list. +- Follow GEP-1748 semantics without claiming upstream Extended conformance, because the GEP names + the upstream MCS API group. +- Use annotations rather than Azure policy CRDs, accepting weaker API-server validation in exchange + for a smaller and more portable public API. +- Compensate for annotation limitations with typed parsing, admission validation, status + conditions, and warning events. +- Put all controller implementation in `fleet-networking`; `fleet` needs only optional examples or + placement documentation. +- Use a separate hub Gateway controller manager to isolate Azure permissions, rollout, failures, + and ARM throttling. +- Start with direct per-Service member origins; shared per-cluster ingress gateways are deferred. +- Treat the Fleet hub as the configuration cluster and install the multi-cluster GatewayClass there. +- Accept only Fleet ServiceImport backends for the multi-cluster GatewayClass, matching GKE's + separation between single-cluster and multi-cluster GatewayClasses. +- Require explicit feature enablement, Fleet/MCS health, managed identity, Azure provider + registration, quota readiness, and controller readiness. +- Fail static during hub/configuration-plane outages or migrations when Azure resource ownership + cannot be proven. + +## Implementation Details + +- `docs/design/gep-1748-gateway-api.md` defines the architecture and compatibility contract. +- `docs/design/gep-1748-implementation-plan.md` defines the tests-first delivery phases and pull + request sequence. +- Gateway API `v1.2.1` is the selected dependency because it matches Kubernetes `v0.31.1`; newer + Gateway API releases require a repository-wide Kubernetes/controller-runtime upgrade. +- The first implementation slice covers dependency and scheme registration, a separate manager + skeleton, typed annotations, and normalized model primitives. +- The design uses `networking.fleet.azure.com/afd-*` annotations on Gateway and ServiceImport + resources. +- `HTTPRoute.backendRefs` targets the unchanged Fleet ServiceImport group. +- Per-logical-backend weight uses `HTTPRoute.backendRefs.weight`; per-member-cluster weight retains + the existing ServiceExport weight annotation. +- PLS discovery is transported through internal resources rather than public ServiceImport status. +- `cmd/hub-gateway-controller-manager` is a separate process with health, readiness, leader + election, Gateway API schemes, feature gating, and controller-wide Azure configuration. +- `pkg/annotations` rejects unknown reserved AFD annotations and invalid explicit values rather + than silently applying defaults. +- `pkg/controllers/hub/gatewaymodel` provides a provider-neutral, deeply copied, deterministic + model with distinct Gateway API backend weights and Fleet ServiceExport origin weights. + +## Changes Made + +- Added a new design directory for the GEP-1748 Gateway API proposal. +- Added the architecture design. +- Added the detailed implementation plan. +- Added a configuration guide with separate public-origin and private + internal-load-balancer/PLS subsections, each including optional WAF. +- Added an actor-and-placement matrix and labeled each configuration step with + who applies it and whether it belongs in the Fleet hub, member clusters, or Azure. +- Documented the private AFD + optional WAF + PLS topology's SFI-NS253 alignment and clarified that + public member load balancers are not the SFI-NS253 topology. The foundation PR does not claim + compliance until Private Link reconciliation, enforcement, and end-to-end evidence are complete. +- Added the Gateway API dependency, scheme registration tests, manager skeleton, annotation parser, + normalized model, Makefile build target, and focused unit tests. +- Focused tests pass with 81.8% coverage for the manager, 95.1% for annotation parsing, and 89.7% + for the normalized model. Race mode could not run locally because `gcc` is not installed; CI + will provide race validation. +- Added this breadcrumb as the decision and documentation trail. + +## Before/After Comparison + +- **Before:** The main branch had Service export/import and Traffic Manager documentation but no + Gateway API global-ingress design. +- **After:** The branch has a standalone Gateway API design and phased plan for AFD, optional WAF, + and optional PLS while preserving Fleet ServiceImport. + +## References + +- `api/v1alpha1/serviceimport_types.go`: Existing ServiceImport contract. +- `api/v1alpha1/serviceexport_types.go`: Existing ServiceExport and cluster weight annotation. +- `api/v1alpha1/internalserviceexport_types.go`: Existing member-to-hub transport. +- `pkg/controllers/hub/serviceimport/controller.go`: ServiceImport aggregation behavior. +- `pkg/controllers/hub/trafficmanagerbackend/controller.go`: Existing Azure reconciliation + patterns. +- `cmd/hub-net-controller-manager/main.go`: Existing hub controller process boundary. +- `docs/concepts/ExportingService/README.md`: Service export/import user model. +- `docs/concepts/DNSBasedGlobalLoadBalancing/README.md`: Existing global-routing ownership model. +- `docs/howtos/gateway-api-afd-configuration.md`: Proposed public and Private Link configuration + workflow. +- GEP-1748: +- Gateway API v1.2.1 type definitions: + +- GKE multi-cluster Gateway requirements: + +- Azure Front Door Private Link: +- Azure Front Door WAF: +- Repository domain knowledge: no files were present under `.github/.copilot/domain_knowledge`. +- Repository specifications: no files were present under `.github/.copilot/specifications`. diff --git a/docs/design/gep-1748-gateway-api.md b/docs/design/gep-1748-gateway-api.md new file mode 100644 index 00000000..e6f8aa9a --- /dev/null +++ b/docs/design/gep-1748-gateway-api.md @@ -0,0 +1,927 @@ +# GEP-1748 Gateway API for Fleet Global Ingress + +## Document status + +- **Status:** Proposed +- **Date:** 2026-08-17 +- **Target repository:** `Azure/fleet-networking` +- **Primary API:** Kubernetes Gateway API +- **Gateway API dependency:** `sigs.k8s.io/gateway-api v1.2.1` +- **Multi-cluster backend API:** `networking.fleet.azure.com/v1alpha1`, kind `ServiceImport` +- **Initial Azure provider:** Azure Front Door Standard/Premium + +## Summary + +This design adds HTTP(S) global ingress to Fleet by combining Kubernetes Gateway API routing with +Fleet's existing multi-cluster `ServiceImport` API. It follows the interaction model described by +[GEP-1748](https://gateway-api.sigs.k8s.io/geps/gep-1748/): an `HTTPRoute` can reference a +`ServiceImport` as a backend, and the Gateway implementation resolves that logical service into +endpoints across the Fleet. + +The design intentionally keeps the existing Fleet `ServiceImport` group, version, kind, schema, and +controller behavior unchanged: + +```text +networking.fleet.azure.com/v1alpha1, kind ServiceImport +``` + +Azure-specific behavior is configured through optional annotations instead of new user-facing +Front Door CRDs. The portable routing contract remains `GatewayClass`, `Gateway`, `HTTPRoute`, and +`ServiceImport`. When another Gateway implementation ignores the Azure annotations, the portable +listener and routing intent remains understandable, although the Fleet-specific `ServiceImport` +group requires explicit support from that implementation. + +Azure Front Door (AFD) is the first provider implementation. Web Application Firewall (WAF) and +Private Link Service (PLS) are optional capabilities layered onto the same Gateway API model. + +## Motivation + +Fleet already provides: + +- Member-cluster `ServiceExport` resources. +- A hub-cluster `ServiceImport` representing one logical service exported by multiple clusters. +- Internal transport resources that identify the contributing member clusters. +- Per-cluster traffic weights through the `networking.fleet.azure.com/weight` annotation. +- Azure global routing integration through Traffic Manager for DNS-based scenarios. + +Fleet does not currently provide a Gateway API implementation for HTTP(S) global ingress. Users +need a model that supports: + +- Host, path, header, and method routing. +- Multiple multi-cluster backends. +- Weighted backend references. +- Edge TLS termination. +- Optional WAF attachment. +- Optional private connectivity from AFD to AKS through PLS. +- Standard Gateway API status and ownership behavior. + +Creating a second set of Front Door-specific routing CRDs would duplicate concepts already defined +by Gateway API. This design instead makes Gateway API the public ingress API and limits Azure +extensions to annotations. + +## Goals + +1. Use `GatewayClass`, `Gateway`, and `HTTPRoute` as the only user-facing AFD routing resources. +2. Allow `HTTPRoute.backendRefs` to target the existing Fleet `ServiceImport`. +3. Preserve the existing Fleet `ServiceImport` API without schema, group, version, or semantic + changes. +4. Implement AFD Standard/Premium as a Gateway controller. +5. Support optional existing WAF policy attachment. +6. Support optional AFD-to-PLS private origin connectivity. +7. Preserve portable Gateway API listener and route configuration. +8. Report standard Gateway API conditions and actionable validation failures. +9. Keep Traffic Manager behavior and APIs independent. +10. Isolate AFD reconciliation from the existing hub networking controller manager. + +## Non-goals + +1. Claim upstream GEP-1748 Extended conformance for the Fleet `ServiceImport` group. +2. Replace or migrate Fleet's `ServiceExport` or `ServiceImport` APIs. +3. Introduce `FrontDoorProfile`, `FrontDoorBackend`, or `FrontDoorCustomDomain` APIs. +4. Model every Azure Front Door property in Kubernetes. +5. Provision WAF policy definitions or managed rule sets. +6. Provision the member-cluster internal load balancer or PLS directly from the hub controller. +7. Implement Gateway API mesh/GAMMA behavior. +8. Make Azure Traffic Manager a GatewayClass. +9. Support arbitrary Azure resource adoption in the initial release. +10. Support shared per-cluster ingress gateways in the initial release. + +## Compatibility with GEP-1748 + +### Dependency baseline + +| Component | Version | Rationale | +|---|---|---| +| Go | Repository baseline | No toolchain upgrade is introduced by Gateway API. | +| Kubernetes libraries | `v0.31.1` | Existing fleet-networking dependency baseline. | +| controller-runtime | `v0.19.0` | Existing fleet-networking dependency baseline. | +| Gateway API | `v1.2.1` | Uses Kubernetes `v0.31.1` and is compatible with the repository baseline. | + +Gateway API `v1.3.0` and later require newer Kubernetes libraries and controller-runtime versions. +They are deferred to avoid coupling this feature to a repository-wide dependency upgrade. + +GEP-1748 defines an `HTTPRoute` backend reference to: + +```yaml +group: multicluster.x-k8s.io +kind: ServiceImport +``` + +Fleet currently owns and reconciles: + +```yaml +group: networking.fleet.azure.com +kind: ServiceImport +``` + +This design adopts the GEP's behavior but not its upstream API group. The Fleet Gateway controller +recognizes the following backend reference as an implementation-specific Extended capability: + +```yaml +backendRefs: +- group: networking.fleet.azure.com + kind: ServiceImport + name: store + port: 8080 +``` + +Consequences: + +- Fleet `ServiceImport` support must be documented as an implementation-specific Extended + capability. +- The implementation must not advertise upstream GEP-1748 Extended conformance unless it also + supports `multicluster.x-k8s.io/ServiceImport`. +- The backend behavior should otherwise follow GEP-1748: routes, filters, weights, namespace + authorization, and status apply consistently to `Service` and `ServiceImport` backends. +- The initial multi-cluster GatewayClass accepts only Fleet `ServiceImport` backend references, + matching the separation used by GKE multi-cluster GatewayClasses. A `Service` backend is rejected + for this class. +- Gateway API conformance claims must describe this class-specific backend restriction. A future + single-cluster GatewayClass may support `Service`, but it is outside this design. + +## Alignment with the GKE multi-cluster Gateway model + +The GKE multi-cluster Gateway setup is a useful operational reference even though the Azure +resource model and network topology differ. This design intentionally follows its major control +plane patterns: + +| GKE multi-cluster Gateway requirement or behavior | Fleet/AFD equivalent | Alignment | +|---|---|---| +| All workload clusters are registered to one fleet | All target clusters are joined to one Fleet hub and have healthy `MemberCluster` state | Aligned | +| Multi-cluster Services is enabled | Fleet member and hub networking controllers reconcile `ServiceExport` and `ServiceImport` | Aligned | +| A selected configuration cluster hosts Gateway resources | The Fleet hub is the configuration cluster for `Gateway`, `HTTPRoute`, and `ServiceImport` | Aligned | +| The platform installs multi-cluster GatewayClasses | The Fleet Gateway chart installs `azure-fleet-afd` | Aligned | +| A hosted controller programs global infrastructure | A dedicated hub Gateway controller manager programs AFD | Same responsibility; different hosting model | +| Workload identity is required | The controller uses Azure workload identity or another approved managed identity mechanism | Aligned | +| Multi-cluster Gateway supports only `ServiceImport` backends | `azure-fleet-afd` accepts only Fleet `ServiceImport` backends | Aligned | +| MCS requirements also apply to Gateway backends | A ServiceImport must be healthy and have resolvable member exports before a route is programmed | Aligned | +| Controller/API enablement is explicit and observable | The feature is enabled explicitly and GatewayClass acceptance confirms readiness | Aligned | +| Configuration-cluster changes can orphan resources | Hub/configuration-plane migration uses an explicit handoff procedure and fails static without ownership proof | Aligned risk treatment | +| Regional control-plane failure causes fail-static behavior | Existing AFD data-plane state remains unchanged while the hub controller is unavailable | Aligned | +| Load balancer quotas apply | AFD profile, endpoint, route, origin, WAF, and Private Link quotas are preflighted and monitored | Aligned | +| Clusters must share supported project/VPC topology | Azure topology is defined by AFD origin reachability and identity authorization, not a same-VNet rule | Intentionally different | + +The Azure implementation does not copy GKE-specific requirements such as VPC-native clusters, +proxy-only subnets, Google APIs, Shared VPC roles, or the `HttpLoadBalancing` add-on. Their Azure +equivalents are AKS/Fleet registration, Azure resource-provider registration, managed identity, +AFD reachability, and PLS readiness. + +## Portability model + +Annotations are used to keep Azure configuration outside the portable Gateway API routing model. +They do not make Azure-specific behavior portable by themselves. Portability is achieved through +graceful degradation: + +1. Listener and routing intent stays in Gateway API fields. +2. Azure annotations are optional and use documented defaults. +3. A non-Azure controller can ignore the annotations. +4. Removing the Azure annotations does not alter host, path, header, method, or backend routing + intent. +5. No Azure resource identifier appears in `HTTPRoute.spec`. + +There is one explicit limitation: Fleet's custom `ServiceImport` group is not directly portable to +controllers that only support the upstream MCS API. Such controllers must add support for Fleet's +group or translate the reference outside this design. + +## User-facing resource model + +### Hub cluster + +The Fleet hub acts as the Gateway API configuration cluster. It contains: + +- One platform-managed `GatewayClass` for the AFD implementation. +- One or more user-created `Gateway` resources. +- User-created `HTTPRoute` resources. +- Existing Fleet `ServiceImport` resources. +- `ReferenceGrant` resources when a route references a backend in another namespace. + +The initial implementation supports exactly one active configuration plane: the Fleet hub where +the Gateway controller is installed. Gateway resources are not copied to member clusters. + +### Member clusters + +Each member cluster contains: + +- The application workload. +- A Kubernetes `Service`. +- A Fleet `ServiceExport`. +- For private origins, an internal `LoadBalancer` Service configured to request PLS creation. + +### Azure + +The Gateway controller owns: + +- AFD profile and endpoint. +- Origin groups and origins. +- Routes and custom domains. +- Security-policy attachment to an existing WAF policy. +- Private Link origin configuration and approval status observation. + +The controller does not own: + +- The AKS-managed internal load balancer. +- The PLS created for the member Service. +- An externally managed WAF policy. +- Public DNS records unless a later design explicitly adds DNS integration. + +## Environment requirements + +These requirements are the Fleet/AFD counterpart of the GKE multi-cluster Gateway preparation +requirements. + +### Fleet and hub requirements + +- All target AKS clusters are registered as healthy members of the same Fleet. +- Fleet member and hub networking controllers are installed and healthy. +- Gateway API CRDs for the selected supported version are installed on the hub. +- The dedicated hub Gateway controller manager and its `azure-fleet-afd` GatewayClass are installed. +- The GatewayClass reports `Accepted=True` before users create Gateways. +- The hub has a durable lifecycle and backup/recovery process because it is the configuration + cluster. + +### Member-cluster requirements + +- The Fleet member networking agent is installed and can publish internal export state to the hub. +- Each multi-cluster backend has a valid `Service` and matching `ServiceExport`. +- Service names, namespaces, and ports satisfy current Fleet ServiceImport conflict rules. +- Public-origin Services expose the public endpoint information required by the controller. +- Private-origin Services use a supported internal load balancer and PLS configuration. + +Gateway API CRDs are not required on member clusters for the initial architecture because Gateway +and HTTPRoute resources live only on the hub. + +### Azure requirements + +- Required Azure resource providers, including `Microsoft.Cdn` and `Microsoft.Network`, are + registered in every subscription used by controller-owned resources or origins. +- The controller has an approved managed identity and scoped Azure RBAC. +- The identity can manage AFD resources in the configured resource group. +- The identity can read externally managed WAF policies that users attach. +- The member or hub identity used for origin discovery can read required public IP, load balancer, + and PLS state. +- Subscription and AFD quotas are sufficient for the requested Gateways, routes, origins, custom + domains, WAF associations, and private endpoints. +- Private Link origins use supported AFD SKU, origin type, Azure region, and approval topology. + +### Enablement and readiness + +Feature enablement is explicit: + +1. Install Gateway API CRDs on the hub. +2. Install the hub Gateway controller with AFD disabled. +3. Configure identity, subscription, resource group, and Azure cloud. +4. Enable the AFD feature. +5. Verify controller health and readiness. +6. Verify `GatewayClass/azure-fleet-afd` reports `Accepted=True`. +7. Verify Fleet ServiceExport/ServiceImport reconciliation is healthy. +8. Create Gateway and HTTPRoute resources. + +## Example configuration + +### GatewayClass + +The platform installs the class. Users reference it but do not modify it. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: azure-fleet-afd +spec: + controllerName: networking.fleet.azure.com/afd +``` + +Provider-wide defaults such as subscription, default resource group, identity, tags, and Azure +cloud environment are controller deployment configuration. They are not repeated on every +Gateway. + +### Gateway without WAF + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: global-ingress + namespace: store +spec: + gatewayClassName: azure-fleet-afd + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: store.example.com + allowedRoutes: + namespaces: + from: Same +``` + +### HTTPRoute to Fleet ServiceImport + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store + namespace: store +spec: + parentRefs: + - name: global-ingress + hostnames: + - store.example.com + rules: + - matches: + - path: + type: PathPrefix + value: /api + backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store-api + port: 8080 + weight: 100 +``` + +The referenced `ServiceImport` remains unchanged: + +```yaml +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: ServiceImport +metadata: + name: store-api + namespace: store +``` + +### Cross-namespace backend + +Gateway API `ReferenceGrant` controls cross-namespace access: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: allow-store-route + namespace: store-backends +spec: + from: + - group: gateway.networking.k8s.io + kind: HTTPRoute + namespace: store + to: + - group: networking.fleet.azure.com + kind: ServiceImport +``` + +## Annotation contract + +All annotations are optional unless stated otherwise. Unknown annotations under the +`networking.fleet.azure.com` prefix are rejected only when they use the reserved AFD annotation +family. Other Fleet annotations remain unaffected. + +### Gateway annotations + +| Annotation | Values | Default | Purpose | +|---|---|---|---| +| `networking.fleet.azure.com/afd-sku` | `Standard_AzureFrontDoor`, `Premium_AzureFrontDoor` | Controller default | Selects the AFD SKU. | +| `networking.fleet.azure.com/afd-resource-group` | Reserved | None | Rejected until a separate per-Gateway ownership design is approved. | +| `networking.fleet.azure.com/afd-waf-policy-id` | Full Azure resource ID | None | Attaches an existing AFD WAF policy. | + +The controller generates stable Azure resource names from the Gateway UID. User-selected Azure +profile names are not part of the initial annotation contract because they complicate ownership, +adoption, collision handling, and deletion. + +The initial implementation does not allow a per-Gateway resource-group override. The resource +group is controller-wide configuration so that Azure RBAC, ownership, quota accounting, and +deletion remain bounded. The `afd-resource-group` annotation is reserved but rejected until a +separate ownership design enables it. + +Annotation keys and values are an implementation API. Existing meanings and defaults cannot change +in place. A replacement must use a new annotation key, support a documented overlap period, and +emit deprecation warnings before the old key is removed. + +### ServiceImport annotations + +| Annotation | Values | Default | Purpose | +|---|---|---|---| +| `networking.fleet.azure.com/afd-origin-connectivity` | `auto`, `public`, `private-link` | `auto` | Selects how AFD reaches all member origins for the logical service. | +| `networking.fleet.azure.com/afd-health-probe-path` | Absolute HTTP path | `/` | Selects the origin-group health probe path. | +| `networking.fleet.azure.com/afd-origin-host-header` | Valid DNS hostname | Derived endpoint hostname | Overrides the Host header sent to origins. | + +The connectivity annotation belongs on `ServiceImport`, rather than `HTTPRoute`, because: + +- Multiple routes can share one logical backend. +- Connectivity is an origin property, not an HTTP match property. +- The same backend cannot safely be public in one route and private in another while sharing an + origin group. +- Backend-level configuration avoids repeating annotations across routes. + +### Existing ServiceExport weight annotation + +The existing annotation remains the source of per-cluster origin weight: + +```text +networking.fleet.azure.com/weight +``` + +`HTTPRoute.backendRefs[*].weight` controls traffic between logical backends. `ServiceExport` +weight controls traffic between member-cluster origins behind one logical `ServiceImport`. + +## Annotation validation + +Annotations are strings and do not receive CRD OpenAPI validation. Misspelled keys and invalid +values may be accepted by the Kubernetes API server. The implementation compensates with: + +1. A single typed annotation parsing package. +2. Table-driven unit tests for all defaults, accepted values, malformed values, and conflicts. +3. A validating admission webhook for deterministic failures where practical. +4. Reconciliation-time validation as the authoritative fallback. +5. Gateway API conditions and Kubernetes warning events with actionable messages. +6. No silent fallback from an explicitly requested capability. + +Examples: + +- An unknown SKU sets the Gateway `Accepted` condition to `False`. +- A malformed WAF policy ID sets `Accepted=False`. +- `private-link` with a non-compatible SKU sets `Accepted=False`. +- A private backend missing PLS information sets the route `ResolvedRefs=False`. +- A backend containing a mixture of public and private member origins sets + `ResolvedRefs=False`. + +The controller must never interpret a malformed value as the default. Defaults apply only when an +annotation is absent. + +## Controller architecture + +### Process boundary + +AFD support is implemented in a separate hub-side controller manager: + +```text +cmd/hub-gateway-controller-manager +``` + +This binary runs in the hub cluster and contains the Gateway API controllers and Azure Front Door +clients. It remains separate from `cmd/hub-net-controller-manager` to provide: + +- Independent feature enablement and rollout. +- Separate Azure permissions. +- Independent leader election and scaling. +- Failure isolation from Service export/import and Traffic Manager. +- Clear ownership of ARM rate limits and metrics. + +### Proposed packages + +```text +cmd/hub-gateway-controller-manager/ +pkg/annotations/ +pkg/controllers/hub/gatewayclass/ +pkg/controllers/hub/gateway/ +pkg/controllers/hub/httproute/ +pkg/controllers/hub/gatewaymodel/ +pkg/providers/azure/frontdoor/ +pkg/providers/azure/origin/ +``` + +### Reconciliation flow + +1. The existing controllers aggregate member `ServiceExport` resources into a hub + `ServiceImport`. +2. The Gateway controller watches `GatewayClass`, `Gateway`, `HTTPRoute`, `ReferenceGrant`, + `ServiceImport`, and relevant internal export resources. +3. It validates the GatewayClass controller name and listener support. +4. It resolves each accepted `HTTPRoute` backend reference. +5. A Fleet `ServiceImport` resolves to its contributing member clusters. +6. Internal export state resolves each member cluster to a public endpoint or PLS. +7. The controller builds a provider-neutral normalized model. +8. The AFD provider reconciles the normalized model into Azure resources. +9. The controller updates Gateway API status only after observing the desired Azure state. + +### Normalized model + +Gateway API objects should not be translated directly into imperative ARM calls. A normalized +model separates Kubernetes validation from provider operations: + +```text +GlobalGateway + listeners[] + routes[] + matches[] + filters[] + backends[] + serviceImport + routeWeight + origins[] + cluster + endpoint + clusterWeight + connectivity + privateLinkResourceID + privateLinkLocation + healthProbe + wafPolicyID +``` + +The model is internal Go code, not a CRD. It allows deterministic comparison, unit testing, and +future provider implementations without changing the public API. + +## ServiceImport resolution + +The public `ServiceImport` schema remains unchanged. The Gateway controller uses: + +- `ServiceImport.status.ports` to validate the backend port. +- `ServiceImport.status.clusters` to identify contributing member clusters. +- `InternalServiceExport.spec.serviceReference` to map an export to its source cluster. +- Existing member-networking state for public IP information and per-cluster weight. +- New internal-only PLS discovery fields when Private Link is implemented. + +Adding PLS information to `InternalServiceExport` or another internal transport resource is allowed +because this design's compatibility promise applies to the public `ServiceImport` API. Internal +schema changes must remain backward-compatible during mixed-version upgrades. + +## HTTPRoute behavior + +### Initial supported features + +- Fleet `ServiceImport` is the only supported `backendRef` kind for the multi-cluster GatewayClass. +- Hostname matching. +- Exact and path-prefix matching. +- Header matching supported by AFD. +- HTTP method matching supported by AFD. +- Weighted backend references. +- Request redirects supported by AFD. +- URL rewrite and header modification where AFD behavior matches Gateway API semantics. +- Same-namespace and `ReferenceGrant`-authorized cross-namespace ServiceImport references. + +### Unsupported features + +An unsupported listener, match, or filter must produce the applicable Gateway API condition and +must not be silently ignored. The initial implementation publishes its supported Gateway API +features through `GatewayClass.status.supportedFeatures` when supported by the selected Gateway API +version. + +A core Kubernetes `Service` backend is unsupported by `azure-fleet-afd`. This is a deliberate +multi-cluster class restriction, consistent with the GKE multi-cluster Gateway model, rather than +an attempt to reinterpret `Service` as a multi-cluster backend. + +### Backend validation + +A Fleet ServiceImport backend is resolved only when: + +- Group is exactly `networking.fleet.azure.com`. +- Kind is exactly `ServiceImport`. +- The object exists and is authorized. +- The requested port exists in `ServiceImport.status.ports`. +- At least one valid member origin can be resolved. +- All origins are compatible with the selected connectivity mode. + +## AFD resource model + +The initial mapping is: + +| Gateway API concept | AFD resource | +|---|---| +| `Gateway` | Profile and endpoint | +| Gateway listener hostname | Custom domain and route domain association | +| `HTTPRoute` | One or more AFD routes/rule sets | +| `ServiceImport` backend | Origin group | +| Member-cluster export | Origin | +| `backendRefs.weight` | Logical backend distribution | +| `ServiceExport` weight | Per-cluster origin weight | +| WAF annotation | Security policy association | + +One Gateway owns one AFD profile in the initial implementation. This provides a simple ownership +and deletion boundary. Profile sharing can be considered separately after the controller's +isolation and quota behavior are understood. + +All owned Azure resources receive tags containing: + +- Fleet hub identity. +- Kubernetes namespace and name. +- Kubernetes UID. +- Controller identifier. + +The Kubernetes UID, not only the object name, is used for ownership checks. + +## Optional WAF + +WAF is enabled by adding a full Azure resource ID to the Gateway: + +```yaml +metadata: + annotations: + networking.fleet.azure.com/afd-waf-policy-id: >- + /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/security-rg/providers/Microsoft.Network/frontdoorWebApplicationFirewallPolicies/store-waf +``` + +Behavior: + +- Absence means no WAF security-policy association. +- The controller attaches the existing policy; it does not create or modify policy rules. +- The controller validates the resource ID shape, policy accessibility, and SKU compatibility. +- Removing the annotation removes only the association, not the WAF policy. +- Deleting the Gateway never deletes an externally managed WAF policy. +- Policy detection/prevention mode remains owned by the WAF policy resource. + +This keeps security policy lifecycle separate from application routing lifecycle. + +## SFI-NS253 alignment for AFD, WAF, and PLS + +The private-backend topology is designed to support SFI-NS253: + +```text +Internet + → Azure Front Door Premium + → optional Azure Front Door WAF policy + → Azure Private Link + → member-cluster Private Link Service + → internal Azure Load Balancer + → application +``` + +In this topology: + +- Member workloads do not require public IP addresses or public load balancer frontends. +- AFD is the public edge and reaches every member origin through Private Link. +- WAF is optional in the API contract; environments that require WAF for SFI or another security + baseline must enforce an approved policy through platform admission or deployment policy. +- Explicit `private-link` connectivity cannot silently downgrade to a public origin. +- A backend remains unresolved until every required PLS and private endpoint connection is ready. + +The public-backend topology described by this design is not the SFI-NS253 topology because member +Services have public load balancer origins. It remains available only for environments where that +exposure is permitted. + +This PR establishes the design, configuration contract, validation primitives, and controller +foundation. It does not by itself demonstrate SFI-NS253 compliance: PLS discovery, AFD Private Link +origin reconciliation, policy enforcement, deployment evidence, and end-to-end validation remain +implementation-plan work. + +## Optional Private Link Service connectivity + +### Member Service + +The member-cluster Service requests an internal load balancer and PLS through supported AKS cloud +provider annotations: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store-api + namespace: store + annotations: + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" +spec: + type: LoadBalancer + ports: + - name: http + port: 8080 + targetPort: 8080 +``` + +The matching `ServiceExport` remains unchanged. + +### Connectivity modes + +`auto`: + +- Use Private Link when every valid member origin has discoverable PLS state. +- Otherwise use public connectivity when every valid member origin has a valid public endpoint. +- Reject mixed public/private resolution instead of guessing. + +`public`: + +- Require all valid member origins to expose a usable public endpoint. +- Ignore PLS state for origin creation. + +`private-link`: + +- Require an AFD SKU that supports Private Link origins. +- Require all valid member origins to expose discoverable PLS resource IDs and locations. +- Reject the backend until required private endpoint approvals are complete. + +### Discovery + +The member networking controller discovers: + +- PLS resource ID. +- Azure location. +- Provisioning state. +- Any stable information required to correlate the PLS with the exported Service. + +The information is transported to the hub through an internal Fleet networking resource. It is +not added to public `ServiceImport.status`. + +### Constraints + +- Public and Private Link origins are not mixed within one origin group. +- The controller does not silently downgrade explicitly requested `private-link` connectivity. +- Private endpoint approval may be asynchronous; status remains not programmed until Azure reports + a usable connection. +- PLS deletion or replacement triggers origin reconciliation without changing the ServiceImport. +- Direct per-Service origins are the initial model. Shared per-cluster Gateway origins are deferred. + +## TLS and certificates + +The initial milestone should support HTTP so that routing and origin reconciliation can be +validated independently. HTTPS requires a separate implementation decision because AFD certificate +sources do not map perfectly to Kubernetes TLS Secrets. + +The preferred first HTTPS option is an AFD-managed certificate for a listener hostname. Bring-your- +own certificate support, including Key Vault/LUMA-managed certificates, requires a documented +reference mechanism and identity permissions. Certificate identifiers must not be stored as secret +values in annotations. + +HTTPS design completion is required before declaring the feature production ready, but it does not +block the initial controller and public-origin implementation. + +## Status and events + +The implementation follows Gateway API status conventions. + +### GatewayClass + +- `Accepted=True` when `controllerName` is recognized and controller configuration is valid. +- `Accepted=False` for invalid platform configuration. + +### Gateway + +- `Accepted` reflects API, annotation, and listener validation. +- `Programmed` becomes true only after the AFD profile, endpoint, domains, and required security + associations match the desired state. +- `addresses` reports the AFD endpoint hostname. +- Listener conditions report unsupported protocols, hostname conflicts, and route attachment state. + +### HTTPRoute + +For each parent: + +- `Accepted` reports listener attachment. +- `ResolvedRefs` reports ServiceImport, port, namespace authorization, and origin resolution. +- Implementation-specific conditions may report Azure programming failures without replacing + standard conditions. + +### Events + +Warning events are emitted for: + +- Invalid annotations. +- Unsupported Gateway API features. +- Missing or incompatible ServiceImport ports. +- Missing public endpoint or PLS information. +- Azure authorization or quota failures. +- Private endpoint approval requirements. + +Repeated events must be rate limited. + +## Ownership, deletion, and drift + +- The Gateway receives a finalizer only before the controller creates an owned Azure resource. +- Deleting a route removes only route-owned configuration. +- Deleting a Gateway removes its owned AFD resources before removing the finalizer. +- Externally managed WAF policies and member-cluster PLS resources are never deleted. +- Azure resources not tagged with the expected Gateway UID are not adopted or deleted. +- Reconciliation is idempotent and compares normalized desired state with observed Azure state. +- Out-of-band edits to owned properties are corrected. +- Out-of-band edits to unowned external resources are observed but not overwritten. + +### Configuration-plane migration + +Changing or rebuilding the hub configuration plane is an explicit migration, not an ordinary +controller restart: + +1. Preserve all Gateway, HTTPRoute, ReferenceGrant, ServiceImport, and ownership identity data. +2. Start the replacement controller in observation-only mode. +3. Verify every Azure resource is tagged with and attributable to the expected Gateway UID. +4. Transfer reconciliation ownership. +5. Disable the old controller only after the replacement reports complete observation. + +If ownership cannot be proven, the replacement controller fails static: it reports status and does +not create, adopt, update, or delete Azure resources. Disabling the feature while Gateways still +exist is rejected or prominently warned because it can leave intentionally persistent Azure +resources without an active reconciler. + +### Control-plane outage behavior + +AFD continues serving the last successfully programmed configuration when the Fleet hub, +Kubernetes API, controller, or Azure management plane is unavailable. During the outage: + +- No speculative changes are made. +- Kubernetes desired-state changes remain pending. +- Existing AFD data-plane traffic continues subject to Azure Front Door availability. +- Reconciliation resumes from observed state after recovery. + +## Security and authorization + +- The controller uses workload identity or another supported managed identity mechanism. +- Azure permissions are scoped to the configured AFD resource group where practical. +- WAF policy access is read plus association; WAF policy mutation is not required. +- Member networking requires read access to load balancer and PLS state needed for discovery. +- Full Azure resource IDs are identifiers, not credentials. +- Secrets, private keys, tokens, and connection strings are forbidden in annotations. +- `ReferenceGrant` is mandatory for cross-namespace ServiceImport references. +- Gateway and route status must not expose credentials or sensitive Azure responses. + +## Coexistence with Traffic Manager + +Traffic Manager remains the DNS-based global routing solution for scenarios such as: + +- Non-HTTP protocols. +- DNS-level failover. +- Existing public endpoints. +- Workloads that do not need an edge proxy, WAF, or HTTP routing. + +AFD handles HTTP(S), edge proxying, application routing, WAF, and optional Private Link. + +The controllers are independent. A ServiceImport may be referenced by both products, but each +controller owns only its Azure resources. The implementation should emit a warning when +configuration creates an unsupported or ambiguous endpoint topology, but it must not modify +Traffic Manager resources. + +## Repository ownership + +### `fleet-networking` + +All implementation changes belong in this repository: + +- Gateway API dependencies and scheme registration. +- Gateway controller manager binary and chart. +- GatewayClass, Gateway, HTTPRoute, and ReferenceGrant reconcilers. +- Annotation parsing and validation. +- ServiceImport and internal export resolution. +- AFD, WAF association, and PLS origin clients. +- Status, events, metrics, tests, examples, and documentation. + +### `fleet` + +No controller or API changes are required. Optional follow-up changes may add: + +- Placement examples that deploy Services and ServiceExports. +- Documentation that identifies the hub as the Gateway configuration cluster. +- Cluster capability labels if placement later depends on Private Link support. + +### AKS cloud provider + +The existing Service annotation contract remains responsible for internal load balancer and PLS +provisioning. Changes are required only if current status does not expose enough stable information +for Fleet member networking to discover the PLS. + +## Feasibility + +| Capability | Feasibility | Key consideration | +|---|---|---| +| GatewayClass/Gateway/HTTPRoute controller | High | Standard controller-runtime integration. | +| Existing Fleet ServiceImport backend | High | Group is implementation-specific and cannot claim upstream GEP Extended conformance. | +| GKE-style hub/config-cluster model | High | Fleet hub already provides a central configuration and membership plane. | +| Public AFD origins | High | Existing internal export state already includes public IP information. | +| Host/path routing | High | Directly maps to AFD routes and rule sets for a defined supported subset. | +| Existing WAF policy attachment | High | Requires resource ID validation, permissions, and SKU checks. | +| PLS discovery | Medium | Internal transport must carry PLS resource ID, location, and readiness. | +| AFD Private Link origins | Medium | Requires Premium capability, asynchronous approval handling, and strict topology validation. | +| Annotation validation | Medium-high | Webhook and reconciliation validation compensate for lack of CRD schema. | +| HTTPS with AFD-managed certificate | Medium-high | Requires domain ownership and provisioning lifecycle handling. | +| Key Vault/LUMA certificate integration | Medium | Requires a separate certificate reference and authorization design. | +| Formal GEP-1748 Extended conformance | Not targeted | Requires upstream `multicluster.x-k8s.io/ServiceImport`. | + +The design is technically feasible. The largest delivery risks are Private Link lifecycle handling, +certificate integration, and maintaining clear status through asynchronous ARM operations. Keeping +the public ServiceImport unchanged is not a technical blocker, but it is a deliberate conformance +tradeoff. + +## Rollout + +1. Ship the controller and chart behind an explicit feature flag. +2. Require healthy Fleet membership, Service export/import, managed identity, and Azure provider + registration before accepting the GatewayClass. +3. Start with HTTP and public AFD origins. +4. Add WAF attachment. +5. Add member PLS discovery and private origins. +6. Add HTTPS and certificate lifecycle. +7. Run Gateway API conformance tests for the features applicable to the multi-cluster class. +8. Publish Fleet ServiceImport behavior as implementation-specific Extended support. +9. Promote only after upgrade, configuration-plane migration, deletion, Azure throttling, quota, + and control-plane failure testing. + +## Open questions + +1. Which HTTPRoute filters map precisely enough to AFD to advertise as supported? +2. Should AFD-managed certificates be the only initial HTTPS mode? +3. How should Private Link approval ownership be represented when manual approval is required? +4. Should one Gateway always own one AFD profile, or should profile sharing be designed later? +5. What mixed-version compatibility window is required for internal PLS transport fields? + +## References + +- [Configure Gateway API with Azure Front Door](../howtos/gateway-api-afd-configuration.md) +- [GEP-1748: Gateway API Interaction with Multi-Cluster Services](https://gateway-api.sigs.k8s.io/geps/gep-1748/) +- [Gateway API v1.2.1 type definitions](https://github.com/kubernetes-sigs/gateway-api/blob/v1.2.1/apis/v1/gateway_types.go) +- [Gateway API cross-namespace routing](https://gateway-api.sigs.k8s.io/guides/multiple-ns/) +- [GKE multi-cluster Gateway requirements](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/prepare-environment-multi-cluster-gateways#requirements) +- [Azure Front Door Private Link](https://learn.microsoft.com/azure/frontdoor/private-link) +- [Azure Web Application Firewall on Azure Front Door](https://learn.microsoft.com/azure/web-application-firewall/afds/afds-overview) +- `api/v1alpha1/serviceimport_types.go`: Existing Fleet ServiceImport contract. +- `api/v1alpha1/serviceexport_types.go`: Existing Fleet ServiceExport and weight contract. +- `api/v1alpha1/internalserviceexport_types.go`: Current member-to-hub exported Service state. +- `pkg/controllers/hub/serviceimport/controller.go`: Existing ServiceImport aggregation. +- `pkg/controllers/hub/trafficmanagerbackend/controller.go`: Existing Azure global-routing reconciliation patterns. +- `cmd/hub-net-controller-manager/main.go`: Existing hub networking controller process. diff --git a/docs/design/gep-1748-implementation-plan.md b/docs/design/gep-1748-implementation-plan.md new file mode 100644 index 00000000..12bd6990 --- /dev/null +++ b/docs/design/gep-1748-implementation-plan.md @@ -0,0 +1,602 @@ +# GEP-1748 Gateway API Implementation Plan + +## Plan status + +- **Status:** Proposed +- **Date:** 2026-08-17 +- **Design:** [GEP-1748 Gateway API for Fleet Global Ingress](gep-1748-gateway-api.md) +- **Target repository:** `Azure/fleet-networking` +- **Branch:** `rchinchani/gep-1748-gateway-api` + +## Delivery principles + +1. Preserve `networking.fleet.azure.com/v1alpha1 ServiceImport` without API changes. +2. Use Gateway API resources as the public global-ingress API. +3. Use optional annotations for Azure-specific configuration. +4. Write unit tests before implementation whenever practical. +5. Build a provider-neutral normalized model before ARM reconciliation. +6. Deliver public AFD origins before WAF, PLS, and HTTPS. +7. Keep AFD in a separate hub controller manager. +8. Fail explicitly through Gateway API conditions; never silently downgrade requested behavior. +9. Keep each implementation pull request independently reviewable and testable. + +## Phase 0: Confirm API and dependency baselines + +### Task 0.0: Define environment prerequisites + +- [x] Map GKE multi-cluster Gateway prerequisites to Fleet, AKS, and Azure. +- [x] Define Fleet membership and ServiceExport/ServiceImport health requirements. +- [x] Define hub/configuration-plane selection, enablement, readiness, and migration procedures. +- [x] Define required Azure resource providers, identity, RBAC, quotas, and network reachability. + +**Success criteria** + +- A preflight checklist can determine whether the environment is ready. +- The `azure-fleet-afd` GatewayClass remains unaccepted when controller prerequisites are invalid. +- GKE-specific VPC and API requirements are not incorrectly applied to Azure. + +### Task 0.1: Select the Gateway API version + +- [x] Choose Gateway API `v1.2.1`, which uses the repository's Kubernetes `v0.31.1` + dependency baseline. +- [x] Record the supported Core and Extended feature set. +- [x] Confirm the selected release includes the status fields used by the controller. + +**Success criteria** + +- The selected module version builds with the repository. +- A compatibility table identifies supported Gateway API and Kubernetes versions. +- The design's status behavior matches the selected API version. + +### Task 0.2: Freeze the annotation contract + +- [x] Review annotation names, values, defaults, and ownership. +- [x] Keep resource-group configuration controller-wide; reserve and reject the annotation. +- [x] Define additive versioning and deprecation rules for annotations. + +**Success criteria** + +- Every annotation has a documented parser, default, validation rule, and failure condition. +- No credential or secret values are permitted. +- Removing Azure annotations leaves portable Gateway routing intent. + +### Task 0.3: Define the supported AFD mapping + +- [x] Map Gateway listeners, HTTPRoute matches, filters, and backends to AFD resources. +- [x] Enforce Fleet ServiceImport as the only backend kind for the multi-cluster GatewayClass. +- [x] Identify unsupported or semantically incompatible features. +- [x] Define resource and quota limits that must be preflighted before production enablement. + +**Success criteria** + +- A reviewed mapping table exists. +- Unsupported features have explicit Gateway API condition behavior. +- The initial conformance profile is testable. + +## Phase 1: Add test infrastructure and dependencies + +### Task 1.1: Add dependency compile tests + +- [x] Add a minimal test that registers Gateway API and Fleet networking schemes. +- [x] Verify existing Fleet ServiceImport GVK registration remains unchanged. + +**Success criteria** + +- The test fails before dependencies and scheme registration are added. +- The test passes after registration. +- No generated Fleet CRD changes are produced. + +### Task 1.2: Add Gateway API dependencies + +- [x] Add `sigs.k8s.io/gateway-api v1.2.1`. +- [x] Run `go mod tidy`. +- [x] Register Gateway API types with the new manager scheme. + +**Success criteria** + +- Target packages compile. +- Existing dependency replacements remain valid. +- `ServiceImport` continues using `networking.fleet.azure.com/v1alpha1`. + +### Task 1.3: Create controller-manager skeleton + +- [x] Add `cmd/hub-gateway-controller-manager`. +- [x] Add health, readiness, metrics, leader election, repository-standard logging, and signal + handling. +- [x] Add a feature-enable flag for AFD reconciliation. +- [x] Add placeholder Azure configuration loading without creating resources. + +**Success criteria** + +- The binary builds and starts against envtest. +- Health and readiness endpoints pass. +- No controllers run when the feature is disabled. + +## Phase 2: Implement typed annotation handling + +### Task 2.1: Write annotation parser tests + +- [x] Cover absent values and defaults. +- [x] Cover every accepted SKU and connectivity value. +- [x] Cover malformed Azure resource IDs, hostnames, and probe paths. +- [x] Cover unknown reserved AFD annotation names. +- [x] Cover incompatible combinations. + +**Success criteria** + +- Tests are table-driven. +- Explicit invalid values never fall back to defaults. +- Error messages identify object, key, value, and expected format. + +### Task 2.2: Implement annotation package + +- [x] Add constants and typed configuration structures. +- [x] Add Gateway annotation parsing. +- [x] Add ServiceImport annotation parsing. +- [x] Add focused Azure Front Door WAF policy resource ID validation. + +**Success criteria** + +- All parser tests pass. +- Reconciler packages consume typed values instead of reading annotation maps directly. +- No annotation parsing logic is duplicated. + +### Task 2.3: Add admission validation + +- [ ] Add validating webhook handlers for Gateway and annotated ServiceImport resources. +- [ ] Keep reconciliation validation as the authoritative fallback. +- [ ] Add webhook chart and certificate configuration. + +**Success criteria** + +- Deterministic annotation errors are rejected at admission when the webhook is enabled. +- Reconciliation reports the same errors when the webhook is unavailable. +- Existing non-AFD Fleet annotations remain unaffected. + +## Phase 3: Build ServiceImport origin resolution + +### Task 3.1: Write public-origin resolver tests + +- [ ] Resolve a ServiceImport port. +- [ ] Resolve contributing clusters. +- [ ] Resolve public IP/DNS information from internal exports. +- [ ] Apply existing ServiceExport cluster weights. +- [ ] Reject missing ports, empty clusters, invalid exports, and mixed endpoint states. + +**Success criteria** + +- Resolver behavior is deterministic regardless of Kubernetes list order. +- Tests cover member deletion and conflicting exports. +- Public `ServiceImport` objects are not mutated. + +### Task 3.2: Implement origin resolver + +- [ ] Add indexes for ServiceImport-to-route and internal-export-to-ServiceImport relationships. +- [ ] Resolve public origins from existing internal resources. +- [ ] Normalize cluster ordering and weights. +- [ ] Expose resolver errors suitable for `ResolvedRefs` conditions. + +**Success criteria** + +- ServiceImport updates enqueue affected routes. +- Internal export updates enqueue only affected routes. +- No polling is required for Kubernetes state changes. + +### Task 3.3: Define the internal normalized model + +- [ ] Write model validation and equality tests. +- [x] Add listeners, routes, matches, filters, logical backends, origins, WAF, and probes. +- [ ] Add stable sorting and deterministic naming inputs. + +**Success criteria** + +- Equal Kubernetes intent produces an equal normalized model. +- Model serialization used in tests is stable. +- Provider packages do not read Kubernetes resources directly. + +## Phase 4: Implement GatewayClass and Gateway reconciliation + +### Task 4.1: Write GatewayClass controller tests + +- [ ] Test recognized and unrecognized controller names. +- [ ] Test invalid controller Azure configuration. +- [ ] Test missing Azure resource-provider registration and insufficient quota preflight results. +- [ ] Test status update conflict retries. + +**Success criteria** + +- `Accepted` follows Gateway API semantics. +- The controller does not modify classes owned by another implementation. + +### Task 4.2: Implement GatewayClass controller + +- [ ] Watch `GatewayClass`. +- [ ] Publish accepted status and supported features. +- [ ] Enqueue Gateways when class acceptance changes. + +**Success criteria** + +- Only `networking.fleet.azure.com/afd` is accepted. +- Status is idempotent and generation-aware where supported. + +### Task 4.3: Write Gateway validation tests + +- [ ] Test listener protocols, ports, hostnames, and route namespace policy. +- [ ] Test AFD SKU and WAF annotation validation. +- [ ] Test unsupported and conflicting listener configurations. + +**Success criteria** + +- Invalid configuration produces expected listener and Gateway conditions. +- Unsupported listeners never create Azure resources. + +### Task 4.4: Implement Gateway controller foundation + +- [ ] Resolve GatewayClass and typed annotations. +- [ ] Generate stable Azure resource names from Gateway UID. +- [ ] Add ownership tags and finalizer rules. +- [ ] Update Gateway and listener status. + +**Success criteria** + +- A valid Gateway reaches `Accepted=True`. +- A finalizer is added only when Azure resource ownership begins. +- Deleting a never-programmed Gateway does not block. + +## Phase 5: Add Azure Front Door provider and public Gateway + +### Task 5.1: Define provider interfaces and fakes + +- [ ] Add narrow interfaces for AFD profile, endpoint, origin group, origin, route, domain, rule + set, and security policy operations. +- [ ] Add deterministic fake clients. +- [ ] Add Azure error classification tests. + +**Success criteria** + +- Controller tests do not require live Azure. +- Retryable, authorization, conflict, not-found, and terminal errors are distinguishable. + +### Task 5.2: Add Azure SDK clients + +- [ ] Add required `armcdn` SDK dependencies. +- [ ] Initialize clients from the controller's Azure configuration. +- [ ] Apply repository-standard retry, rate limit, and user-agent behavior. + +**Success criteria** + +- Client initialization tests pass. +- No credentials are logged. +- Azure calls use bounded contexts and actionable wrapped errors. + +### Task 5.3: Write Gateway programming tests + +- [ ] Test create, update, no-op, drift correction, and deletion. +- [ ] Test ownership collision and foreign-resource rejection. +- [ ] Test partial ARM failure and retry. + +**Success criteria** + +- Repeated reconciliation is idempotent. +- Foreign resources are not adopted or deleted. +- `Programmed=True` is set only after observing desired state. + +### Task 5.4: Implement AFD profile and endpoint reconciliation + +- [ ] Create one profile and endpoint per Gateway. +- [ ] Apply stable names and ownership tags. +- [ ] Publish the AFD hostname in Gateway addresses. +- [ ] Implement cleanup through the Gateway finalizer. + +**Success criteria** + +- A valid HTTP Gateway becomes programmed. +- Deletion removes only owned resources. +- Out-of-band drift in owned fields is corrected. + +## Phase 6: Implement HTTPRoute and public origins + +### Task 6.1: Write route attachment tests + +- [ ] Test parent references and listener selection. +- [ ] Test hostname intersection. +- [ ] Test allowed route namespaces. +- [ ] Test `ReferenceGrant` for cross-namespace ServiceImport. +- [ ] Test unsupported filters and matches. + +**Success criteria** + +- Route parent status matches Gateway API semantics. +- Unauthorized references set `ResolvedRefs=False`. + +### Task 6.2: Write ServiceImport backend tests + +- [ ] Test the exact Fleet group and kind. +- [ ] Test that core Kubernetes Service backends are rejected for the multi-cluster GatewayClass. +- [ ] Test port resolution. +- [ ] Test weighted logical backends and per-cluster origin weights. +- [ ] Test missing and deleted ServiceImports. +- [ ] Test references to unsupported groups and kinds. + +**Success criteria** + +- Fleet ServiceImport behavior follows the GEP-1748 routing model. +- The controller does not claim support for upstream MCS ServiceImport. + +### Task 6.3: Implement HTTPRoute controller + +- [ ] Build normalized routes from accepted Gateway parents. +- [ ] Resolve Fleet ServiceImport backends. +- [ ] Reconcile AFD origin groups, origins, routes, and supported rule sets. +- [ ] Update `Accepted` and `ResolvedRefs`. + +**Success criteria** + +- Host and path traffic routes to every valid member origin. +- Route updates do not recreate the AFD profile. +- Removing a route cleans only its owned route resources. + +### Task 6.4: Add public-origin integration tests + +- [ ] Run controllers with envtest and fake Azure clients. +- [ ] Create ServiceImport/internal export/Gateway/HTTPRoute fixtures. +- [ ] Verify desired Azure operations and final statuses. + +**Success criteria** + +- End-to-end controller reconciliation succeeds without live Azure. +- Changes to member endpoints update the origin set. + +## Phase 7: Add optional WAF attachment + +### Task 7.1: Write WAF validation and ownership tests + +- [ ] Test absent policy, valid policy ID, malformed ID, inaccessible policy, and incompatible + SKU. +- [ ] Test annotation removal and Gateway deletion. + +**Success criteria** + +- WAF policy objects are never created, modified, or deleted. +- Invalid explicit configuration never falls back to no WAF. + +### Task 7.2: Implement WAF association + +- [ ] Read and validate the existing WAF policy. +- [ ] Create or update the AFD security-policy association. +- [ ] Remove only the association when the annotation is removed. +- [ ] Reflect Azure errors in Gateway status. + +**Success criteria** + +- WAF is optional. +- The correct custom domains are associated. +- Gateway deletion leaves the external policy intact. + +## Phase 8: Add Private Link Service discovery + +### Task 8.1: Define backward-compatible internal transport + +- [ ] Select `InternalServiceExport` fields or a dedicated internal resource for PLS state. +- [ ] Define resource ID, location, provisioning state, and readiness fields. +- [ ] Define mixed-version defaulting and compatibility. + +**Success criteria** + +- Public ServiceImport API and generated CRD remain unchanged. +- Old member agents and new hub agents can coexist safely. +- Missing PLS fields mean unknown, not public. + +### Task 8.2: Write member PLS discovery tests + +- [ ] Test internal load balancer with PLS. +- [ ] Test public Service, missing PLS, provisioning PLS, replacement, and deletion. +- [ ] Test Azure authorization and transient errors. + +**Success criteria** + +- Discovery reports stable state without leaking credentials. +- PLS replacement triggers a hub update. + +### Task 8.3: Implement member PLS discovery + +- [ ] Read Service and Azure load balancer/PLS state. +- [ ] Publish PLS resource ID, location, and readiness through internal transport. +- [ ] Add required member-controller Azure permissions and documentation. + +**Success criteria** + +- Hub state converges after PLS creation, approval, replacement, and deletion. +- Public-origin behavior remains unchanged for non-PLS Services. + +## Phase 9: Add AFD Private Link origins + +### Task 9.1: Write connectivity-mode tests + +- [ ] Test `auto`, `public`, and `private-link`. +- [ ] Test all-public, all-private, mixed, incomplete, and empty origin sets. +- [ ] Test incompatible SKU and asynchronous approval. + +**Success criteria** + +- Mixed topology is rejected. +- Explicit private mode never downgrades to public. +- `auto` selects only a fully valid topology. + +### Task 9.2: Implement Private Link origin reconciliation + +- [ ] Add PLS resource ID and location to normalized origins. +- [ ] Configure AFD Private Link origins. +- [ ] Observe private endpoint connection and approval state. +- [ ] Update route and Gateway status through asynchronous transitions. + +**Success criteria** + +- A fully private ServiceImport becomes programmed. +- Pending approval is visible and retryable. +- PLS deletion removes or disables the affected origin without impacting unrelated backends. + +### Task 9.3: Add live Azure PLS end-to-end tests + +- [ ] Provision two member-cluster internal Services with PLS. +- [ ] Program an AFD Premium Gateway and HTTPRoute. +- [ ] Approve connections using the documented ownership workflow. +- [ ] Verify traffic, failover, weight changes, and cleanup. + +**Success criteria** + +- Traffic reaches both private member origins. +- No public origin path is created. +- Cleanup leaves no controller-owned AFD resources. + +## Phase 10: Add HTTPS and certificate lifecycle + +### Task 10.1: Finalize certificate design + +- [ ] Select the first supported certificate mode. +- [ ] Define listener validation and status. +- [ ] Define domain ownership validation and renewal behavior. +- [ ] Document future Key Vault/LUMA integration boundaries. + +**Success criteria** + +- Certificate identifiers do not expose secret material. +- Ownership and deletion behavior are explicit. +- The design supports safe rotation. + +### Task 10.2: Implement AFD-managed certificate support + +- [ ] Add HTTPS listener translation. +- [ ] Create and associate custom domains. +- [ ] Observe domain validation and certificate deployment. +- [ ] Publish listener status throughout provisioning. + +**Success criteria** + +- HTTPS becomes programmed only after the certificate is deployed. +- HTTP-only Gateways remain unaffected. +- Domain or certificate failure is actionable. + +## Phase 11: Package, observe, and document + +### Task 11.1: Add Helm chart and image targets + +- [ ] Add controller deployment, service account, workload identity, RBAC, webhook, service, + metrics, and configuration. +- [ ] Add Makefile, Dockerfile, and CI targets. +- [ ] Keep installation explicitly disabled by default until promoted. + +**Success criteria** + +- Chart rendering and installation tests pass. +- Controller has only required Kubernetes and Azure permissions. +- Existing controller charts remain unchanged unless shared helpers require updates. + +### Task 11.2: Add metrics and diagnostics + +- [ ] Add reconciliation latency, status, Azure operation, throttling, and resource-count metrics. +- [ ] Add structured logs keyed by Kubernetes and Azure resource identity. +- [ ] Add rate-limited events. + +**Success criteria** + +- Operators can identify validation, Kubernetes, ARM, quota, authorization, and approval failures. +- Logs do not expose secrets. + +### Task 11.3: Add examples and operational documentation + +- [ ] Add public AFD example. +- [ ] Add optional WAF example. +- [ ] Add private PLS example. +- [ ] Add required Azure roles, limits, troubleshooting, and cleanup documentation. +- [ ] Document the implementation-specific GEP-1748 compatibility statement. +- [ ] Add a GKE-to-Fleet prerequisite comparison and an environment preflight procedure. + +**Success criteria** + +- Each supported topology has a runnable example. +- Limitations and non-conformance are prominent. +- Troubleshooting maps status reasons to operator actions. + +## Phase 12: Conformance, upgrade, and release + +### Task 12.1: Run Gateway API conformance tests + +- [ ] Run conformance tests applicable to the selected multi-cluster GatewayClass feature set. +- [ ] Add implementation-specific tests for Fleet ServiceImport backend references. +- [ ] Verify core Kubernetes Service backends are rejected as documented. +- [ ] Publish supported and unsupported features. + +**Success criteria** + +- All claimed Gateway API features pass. +- Fleet ServiceImport support is tested without claiming upstream MCS Extended conformance. + +### Task 12.2: Test upgrades and deletion + +- [ ] Test controller upgrade with existing programmed Gateways. +- [ ] Test configuration-plane migration using observation-only ownership verification. +- [ ] Test fail-static behavior when ownership cannot be proven. +- [ ] Test annotation default changes are prohibited or safely versioned. +- [ ] Test member/hub mixed versions for PLS. +- [ ] Test deletion during partial Azure outages. + +**Success criteria** + +- Upgrades do not recreate stable AFD resources. +- Finalizers do not become permanently stuck on terminal authorization failures without operator + guidance. +- Mixed-version behavior matches the compatibility contract. + +### Task 12.3: Complete production-readiness review + +- [ ] Review quotas, scale, rate limits, retry budgets, and disaster recovery. +- [ ] Review identity, RBAC, WAF ownership, and private connectivity. +- [ ] Review regional and Azure control-plane failure modes. +- [ ] Define support and rollback procedures. + +**Success criteria** + +- Release criteria, rollback, monitoring, and ownership are approved. +- The feature flag can be enabled for the target environment. + +## Proposed pull request sequence + +1. **Design:** design document, implementation plan, and examples of the API contract. +2. **Foundation:** Gateway API dependency, annotation parser, normalized model, and manager skeleton. +3. **Gateway status:** GatewayClass and Gateway validation/status without Azure mutation. +4. **Public AFD:** provider clients, profile/endpoint, HTTPRoute, ServiceImport public origins. +5. **WAF:** existing policy attachment. +6. **PLS discovery:** internal transport and member discovery. +7. **Private origins:** AFD Premium Private Link reconciliation. +8. **HTTPS:** domain and AFD-managed certificate lifecycle. +9. **Packaging:** chart, permissions, observability, examples, and conformance evidence. + +Each PR must include focused unit or integration tests and must leave the repository buildable. + +## Detailed checklist + +- [ ] Phase 0: API, annotation, and AFD mapping decisions complete. +- [ ] Phase 1: Dependencies and manager skeleton complete. +- [ ] Phase 2: Typed annotation handling complete. +- [ ] Phase 3: ServiceImport origin resolution and normalized model complete. +- [ ] Phase 4: GatewayClass and Gateway reconciliation complete. +- [ ] Phase 5: Public AFD Gateway programming complete. +- [ ] Phase 6: HTTPRoute and public ServiceImport origins complete. +- [ ] Phase 7: Optional WAF attachment complete. +- [ ] Phase 8: Member PLS discovery complete. +- [ ] Phase 9: AFD Private Link origins complete. +- [ ] Phase 10: HTTPS and certificates complete. +- [ ] Phase 11: Packaging, observability, and documentation complete. +- [ ] Phase 12: Conformance and production readiness complete. + +## Overall success criteria + +- Users configure global HTTP(S) ingress with Gateway API resources. +- `HTTPRoute` references the unchanged Fleet `ServiceImport` API. +- Public, WAF-protected, and Private Link origin modes work as documented. +- Invalid annotations and unsupported Gateway features fail explicitly. +- Standard Gateway API status accurately represents Kubernetes and Azure state. +- Existing Service export/import and Traffic Manager functionality remains compatible. +- All claimed Gateway API conformance tests pass. +- Upgrade, deletion, throttling, identity, and failure scenarios are documented and tested. diff --git a/docs/howtos/gateway-api-afd-configuration.md b/docs/howtos/gateway-api-afd-configuration.md new file mode 100644 index 00000000..432a3fd4 --- /dev/null +++ b/docs/howtos/gateway-api-afd-configuration.md @@ -0,0 +1,581 @@ +# Configure Gateway API with Azure Front Door + +> [!IMPORTANT] +> This guide describes the proposed GEP-1748 implementation on the +> `rchinchani/gep-1748-gateway-api` branch. The controller is not yet released. +> The manifests define the intended user contract and will become runnable as +> the implementation phases are completed. + +This guide configures a Fleet multi-cluster HTTP application behind Azure Front +Door (AFD). It covers two independent backend topologies: + +1. [AFD with public backends](#configure-afd-with-public-backends) +2. [AFD with Private Link to internal load balancers](#configure-afd-with-private-link-to-internal-load-balancers) + +Both topologies use: + +- Gateway API resources in the Fleet hub. +- A Fleet `ServiceImport` as the only supported `HTTPRoute` backend. +- A matching `Service` and `ServiceExport` in every participating member + cluster. +- Optional attachment to an existing Azure Front Door WAF policy. + +The examples configure HTTP listeners. HTTPS and certificate lifecycle are a +later implementation phase. + +## Common prerequisites + +Before configuring either topology: + +1. Register every workload cluster as a healthy member of the same Fleet. +2. Install the Fleet hub and member networking controllers. +3. Install Gateway API `v1.2.1` CRDs in the Fleet hub. +4. Install the proposed `hub-gateway-controller-manager` in the Fleet hub. +5. Configure the controller with: + - An Azure subscription. + - A controller-wide AFD resource group. + - An approved workload or managed identity. + - Permission to manage AFD resources in the configured resource group. +6. Confirm the platform-installed GatewayClass is accepted: + + ```bash + kubectl get gatewayclass azure-fleet-afd + ``` + + Expected status: + + ```text + NAME CONTROLLER ACCEPTED + azure-fleet-afd networking.fleet.azure.com/afd True + ``` + +The examples use these placeholders: + +| Placeholder | Description | +|---|---| +| `${MEMBER_CONTEXT}` | `kubectl` context for one Fleet member cluster. | +| `${HUB_CONTEXT}` | `kubectl` context for the Fleet hub. | +| `${NAMESPACE}` | Application namespace, such as `store`. | +| `${SERVICE_NAME}` | Service and ServiceExport name, such as `store-api`. | +| `${APP_HOSTNAME}` | Public application hostname, such as `store.example.com`. | +| `${WAF_POLICY_ID}` | Optional full resource ID of an existing AFD WAF policy. | + +Repeat the member-cluster steps for every cluster that should contribute an +origin to the multi-cluster service. + +## Responsibility and resource placement + +| Configuration or action | Applied or performed by | Where | +|---|---|---| +| Install Gateway API CRDs | Fleet platform operator | Fleet hub | +| Install and configure `hub-gateway-controller-manager` | Fleet platform operator | Fleet hub | +| Create `GatewayClass/azure-fleet-afd` | Fleet platform operator, normally through the controller Helm chart | Fleet hub; cluster-scoped | +| Configure controller subscription, AFD resource group, identity, and feature flag | Fleet platform operator | Gateway controller deployment in the Fleet hub | +| Grant AFD and origin-discovery Azure permissions | Azure subscription or resource-group owner | Azure RBAC | +| Deploy application workload | Application team or Fleet placement controller | Every selected member cluster | +| Create the application `Service` and `ServiceExport` | Application team or Fleet placement controller | Every selected member cluster | +| Create the public Load Balancer | AKS cloud provider, in response to the member `Service` | Azure resources associated with each member cluster | +| Create the internal Load Balancer and PLS | AKS cloud provider, in response to the member `Service` annotations | Azure resources associated with each member cluster | +| Create and aggregate `ServiceImport` | Fleet networking controllers | Fleet hub | +| Add AFD backend annotations to `ServiceImport` | Application or networking owner | Existing generated `ServiceImport` in the Fleet hub | +| Create `Gateway` and `HTTPRoute` | Application ingress owner | Fleet hub | +| Create and maintain WAF policy | Security owner | Azure | +| Add the optional WAF policy annotation to `Gateway` | Application ingress owner after authorization from the security owner | Fleet hub | +| Create and reconcile AFD resources | Hub Gateway controller | Azure, in the configured AFD resource group | +| Approve AFD-to-PLS private endpoint connections | Network or security owner | Azure PLS for each member cluster | +| Create application DNS records | DNS owner | Authoritative public DNS zone | + +Users do not create `ServiceImport` directly. Fleet creates it from matching +member-cluster exports. Users annotate the generated hub object after it +appears. A production GitOps workflow should use a controller or patch that +waits for the generated object instead of attempting to own its full manifest. + +## Configure AFD with public backends + +In this topology, every member cluster exposes the application through a public +Azure Load Balancer. AFD connects to those public origins. + +### 1. Create a public Service in each member cluster + +**Actor:** Application team or Fleet placement controller + +**Target:** Every participating member cluster + +Use a unique Azure DNS label in each member cluster. The label gives AFD a +stable origin hostname and must be unique within the Azure region. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store-api + namespace: store + annotations: + service.beta.kubernetes.io/azure-dns-label-name: store-api-member-east +spec: + type: LoadBalancer + selector: + app: store-api + ports: + - name: http + protocol: TCP + port: 8080 + targetPort: 8080 +--- +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: ServiceExport +metadata: + name: store-api + namespace: store +``` + +Apply the manifest to each member cluster: + +```bash +kubectl --context "${MEMBER_CONTEXT}" apply -f public-backend.yaml +``` + +Verify that the Service has a public address and that the export is valid: + +```bash +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get service "${SERVICE_NAME}" + +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get serviceexport "${SERVICE_NAME}" -o yaml +``` + +Do not continue until each `ServiceExport` reports `Valid=True` and +`Conflict=False`. + +### 2. Select public origin connectivity on the ServiceImport + +**Actor:** Application or networking owner + +**Target:** Generated `ServiceImport` in the Fleet hub + +Fleet creates the `ServiceImport` in the hub after at least one valid export is +observed. Keep its API unchanged and add only the AFD connectivity annotation: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate serviceimport "${SERVICE_NAME}" \ + networking.fleet.azure.com/afd-origin-connectivity=public \ + networking.fleet.azure.com/afd-health-probe-path=/healthz \ + --overwrite +``` + +The `public` value is explicit: if any member origin lacks a usable public +endpoint, the route receives `ResolvedRefs=False`. The controller must not +silently omit the invalid member or switch the backend to Private Link. + +### 3. Create the public AFD Gateway and route + +**Actor:** Application ingress owner + +**Target:** Fleet hub + +AFD Standard is sufficient for public origins. Premium is also valid. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: store-global + namespace: store + annotations: + networking.fleet.azure.com/afd-sku: Standard_AzureFrontDoor +spec: + gatewayClassName: azure-fleet-afd + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: store.example.com + allowedRoutes: + namespaces: + from: Same +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store-api + namespace: store +spec: + parentRefs: + - name: store-global + sectionName: http + hostnames: + - store.example.com + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store-api + port: 8080 + weight: 100 +``` + +Apply the resources to the hub: + +```bash +kubectl --context "${HUB_CONTEXT}" apply -f public-gateway.yaml +``` + +The Fleet group in `backendRefs` is intentional. This implementation follows +the GEP-1748 behavior while preserving the existing +`networking.fleet.azure.com/v1alpha1 ServiceImport` API. + +### 4. Optionally attach WAF to the public Gateway + +**Actors:** Security owner creates and maintains the policy; application +ingress owner attaches the approved policy + +**Targets:** WAF policy in Azure; annotation on the `Gateway` in the Fleet hub + +WAF is optional and is attached at the Gateway, not at an individual route. +The referenced policy must already exist and must be compatible with the +selected AFD SKU. + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate gateway store-global \ + networking.fleet.azure.com/afd-waf-policy-id="${WAF_POLICY_ID}" \ + --overwrite +``` + +The controller creates only the AFD security-policy association. It does not +create, modify, or delete the external WAF policy. + +To remove WAF while retaining the Gateway: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate gateway store-global \ + networking.fleet.azure.com/afd-waf-policy-id- +``` + +### 5. Verify the public configuration + +**Actor:** Application ingress owner or Fleet platform operator + +**Targets:** Gateway API status in the Fleet hub and AFD resources in Azure + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get gateway store-global -o yaml + +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get httproute store-api -o yaml +``` + +Verify: + +- Gateway `Accepted=True`. +- Gateway `Programmed=True`. +- Listener `Programmed=True`. +- HTTPRoute parent `Accepted=True`. +- HTTPRoute parent `ResolvedRefs=True`. +- Gateway `status.addresses` contains the AFD endpoint hostname. + +Create the required public DNS record for `${APP_HOSTNAME}` only after the +Gateway reports its AFD hostname. DNS ownership and custom-domain validation +must follow the status and instructions emitted by the controller. The DNS +owner performs this step in the authoritative public DNS zone. + +## Configure AFD with Private Link to internal load balancers + +In this topology, each member application is exposed through an internal Azure +Load Balancer and an Azure Private Link Service (PLS). AFD Premium creates a +private endpoint connection to each PLS. The member Services are not exposed as +public origins. + +This is the topology intended to support SFI-NS253 because the member workloads +do not require public IP addresses. The public backend configuration in the +preceding section is not the SFI-NS253 topology. The current PR documents and +validates the contract but does not yet implement or certify the complete +Private Link data path. + +### 1. Create an internal LoadBalancer Service and PLS in each member cluster + +**Actor:** Application team or Fleet placement controller + +**Target:** Every participating member cluster + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store-api + namespace: store + annotations: + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" + service.beta.kubernetes.io/azure-pls-name: store-api-pls +spec: + type: LoadBalancer + selector: + app: store-api + ports: + - name: http + protocol: TCP + port: 8080 + targetPort: 8080 +--- +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: ServiceExport +metadata: + name: store-api + namespace: store +``` + +Apply the manifest to every private-backend member cluster: + +```bash +kubectl --context "${MEMBER_CONTEXT}" apply -f private-backend.yaml +``` + +Verify that the Service address is private and the export is valid: + +```bash +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get service "${SERVICE_NAME}" -o wide + +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get serviceexport "${SERVICE_NAME}" -o yaml +``` + +Also verify that Azure created the PLS in the AKS node resource group: + +```bash +az network private-link-service show \ + --resource-group "${AKS_NODE_RESOURCE_GROUP}" \ + --name store-api-pls \ + --query "{id:id,provisioningState:provisioningState}" \ + --output yaml +``` + +The AKS cloud provider creates the internal load balancer and PLS. The member +networking controller then discovers the PLS resource ID, Azure location, and +readiness and transports them to the hub through internal Fleet networking +state. These fields are not added to public `ServiceImport.status`. + +### 2. Require Private Link connectivity on the ServiceImport + +**Actor:** Application or networking owner + +**Target:** Generated `ServiceImport` in the Fleet hub + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate serviceimport "${SERVICE_NAME}" \ + networking.fleet.azure.com/afd-origin-connectivity=private-link \ + networking.fleet.azure.com/afd-health-probe-path=/healthz \ + --overwrite +``` + +The explicit `private-link` value prevents unsafe downgrade. If any member +origin lacks a ready PLS, the route remains unresolved rather than becoming +public. + +All origins represented by one ServiceImport must use the same connectivity +mode. Public and Private Link origins cannot be mixed in one AFD origin group. + +### 3. Create the Premium AFD Gateway and route + +**Actor:** Application ingress owner + +**Target:** Fleet hub + +Private Link origins require the Premium AFD SKU: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: store-private-global + namespace: store + annotations: + networking.fleet.azure.com/afd-sku: Premium_AzureFrontDoor +spec: + gatewayClassName: azure-fleet-afd + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: store.example.com + allowedRoutes: + namespaces: + from: Same +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store-private-api + namespace: store +spec: + parentRefs: + - name: store-private-global + sectionName: http + hostnames: + - store.example.com + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store-api + port: 8080 + weight: 100 +``` + +Apply the resources to the hub: + +```bash +kubectl --context "${HUB_CONTEXT}" apply -f private-gateway.yaml +``` + +Using `private-link` with `Standard_AzureFrontDoor` is invalid and must produce +an explicit condition rather than a fallback. + +### 4. Optionally attach WAF to the private Gateway + +**Actors:** Security owner creates and maintains the policy; application +ingress owner attaches the approved policy + +**Targets:** WAF policy in Azure; annotation on the `Gateway` in the Fleet hub + +Attach an existing AFD Premium-compatible WAF policy: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate gateway store-private-global \ + networking.fleet.azure.com/afd-waf-policy-id="${WAF_POLICY_ID}" \ + --overwrite +``` + +As with the public topology, the WAF policy lifecycle remains external to the +Gateway. Removing the annotation removes only the AFD association. + +### 5. Approve the AFD private endpoint connections + +**Actor:** Network or security owner + +**Target:** Each member cluster's PLS in Azure + +AFD Private Link approval is asynchronous. Repeat this process for the PLS in +every participating member cluster. + +Get the PLS resource ID: + +```bash +export PLS_ID=$(az network private-link-service show \ + --resource-group "${AKS_NODE_RESOURCE_GROUP}" \ + --name store-api-pls \ + --query id \ + --output tsv) +``` + +List pending private endpoint connections: + +```bash +az network private-endpoint-connection list \ + --id "${PLS_ID}" \ + --query "[?privateLinkServiceConnectionState.status=='Pending'].{name:name,id:id}" \ + --output table +``` + +Approve each connection after verifying that its request belongs to the +controller-owned AFD origin: + +```bash +az network private-endpoint-connection approve \ + --id "${PRIVATE_ENDPOINT_CONNECTION_ID}" \ + --description "Approve Fleet Gateway AFD origin" +``` + +Do not configure broad PLS auto-approval unless the platform security owner has +approved that trust boundary. + +### 6. Verify the private configuration + +**Actor:** Application ingress owner, network owner, or Fleet platform operator + +**Targets:** Gateway API status in the Fleet hub, AFD resources, and member PLS +connections in Azure + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get gateway store-private-global -o yaml + +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get httproute store-private-api -o yaml +``` + +During approval, the route can remain `ResolvedRefs=False` or the Gateway can +remain `Programmed=False` with a reason identifying pending Private Link +approval. After every required connection is approved, verify: + +- Gateway `Accepted=True`. +- Gateway `Programmed=True`. +- HTTPRoute parent `Accepted=True`. +- HTTPRoute parent `ResolvedRefs=True`. +- Every AFD origin uses Private Link. +- No public AFD origin was created for the ServiceImport. + +## Configure per-cluster traffic weights + +**Actor:** Application or traffic-management owner + +**Target:** `ServiceExport` in each member cluster + +For either topology, the existing ServiceExport annotation controls traffic +between member-cluster origins: + +```bash +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate serviceexport "${SERVICE_NAME}" \ + networking.fleet.azure.com/weight=100 \ + --overwrite +``` + +Use: + +- `HTTPRoute.backendRefs[*].weight` to split traffic between logical + ServiceImport backends. +- `ServiceExport` weight to split traffic between member clusters behind one + ServiceImport. + +## Remove the configuration + +**Actors:** Application ingress owner removes hub routing; application team or +Fleet placement controller removes member resources + +**Order:** Fleet hub first, then member clusters + +Delete hub routing resources before deleting member Services: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" delete httproute --all + +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" delete gateway --all +``` + +Wait for Gateway finalizers to finish deleting controller-owned AFD resources. +Then delete ServiceExports and Services from each member cluster. Externally +managed WAF policies are never deleted by the Gateway controller. + +## Related design documents + +- [GEP-1748 Gateway API for Fleet Global Ingress](../design/gep-1748-gateway-api.md) +- [GEP-1748 Gateway API Implementation Plan](../design/gep-1748-implementation-plan.md) +- [Exporting Services](../concepts/ExportingService/README.md) From 7bf9918ce41b10ae268a9acfdd21193cd92411ad Mon Sep 17 00:00:00 2001 From: Ramkumar Chinchani Date: Mon, 17 Aug 2026 19:29:01 -0700 Subject: [PATCH 2/4] feat: add Gateway API controller foundation Pin Gateway API v1.2.1 to the repository's Kubernetes v0.31.1 baseline and register Gateway, HTTPRoute, ReferenceGrant, and the unchanged Fleet ServiceImport types in a separate hub Gateway manager. Add controller-wide Azure configuration validation, feature gating, health and readiness endpoints, leader election, structured klog events, and repository build targets. The foundation intentionally registers no Azure reconcilers yet. Add typed annotation parsing for AFD SKU, optional WAF attachment, origin connectivity, health probes, and host headers. Reject unknown reserved keys, malformed values, unsupported resource-group overrides, and incompatible Private Link settings. Add a provider-neutral normalized Gateway model with deterministic ordering and separate validation for Gateway API backend weights and Fleet ServiceExport per-cluster weights. Focused tests cover manager startup and failure paths, annotations, and the normalized model. Local race testing is deferred to CI because this Windows host does not have gcc installed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d8af5d6-9aba-40a2-8efe-9bf09675667b --- Makefile | 5 + cmd/hub-gateway-controller-manager/main.go | 187 ++++++++++ .../main_test.go | 348 ++++++++++++++++++ go.mod | 2 + go.sum | 2 + pkg/annotations/afd.go | 211 +++++++++++ pkg/annotations/afd_test.go | 271 ++++++++++++++ pkg/controllers/hub/gatewaymodel/model.go | 206 +++++++++++ .../hub/gatewaymodel/model_test.go | 212 +++++++++++ 9 files changed, 1444 insertions(+) create mode 100644 cmd/hub-gateway-controller-manager/main.go create mode 100644 cmd/hub-gateway-controller-manager/main_test.go create mode 100644 pkg/annotations/afd.go create mode 100644 pkg/annotations/afd_test.go create mode 100644 pkg/controllers/hub/gatewaymodel/model.go create mode 100644 pkg/controllers/hub/gatewaymodel/model_test.go diff --git a/Makefile b/Makefile index d8bc01b6..1ef91a0a 100644 --- a/Makefile +++ b/Makefile @@ -183,9 +183,14 @@ generate: $(CONTROLLER_GEN) .PHONY: build build: generate fmt vet ## Build binaries. go build -o bin/hub-net-controller-manager cmd/hub-net-controller-manager/main.go + go build -o bin/hub-gateway-controller-manager cmd/hub-gateway-controller-manager/main.go go build -o bin/member-net-controller-manager cmd/member-net-controller-manager/main.go go build -o bin/mcs-controller-manager cmd/mcs-controller-manager/main.go +.PHONY: run-hub-gateway-controller-manager +run-hub-gateway-controller-manager: manifests generate fmt vet ## Run the hub Gateway controller from your host. + go run ./cmd/hub-gateway-controller-manager/main.go + .PHONY: run-hub-net-controller-manager run-hub-net-controller-manager: manifests generate fmt vet ## Run a controllers from your host. go run ./cmd/hub-net-controller-manager/main.go diff --git a/cmd/hub-gateway-controller-manager/main.go b/cmd/hub-gateway-controller-manager/main.go new file mode 100644 index 00000000..7778f799 --- /dev/null +++ b/cmd/hub-gateway-controller-manager/main.go @@ -0,0 +1,187 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Binary hub-gateway-controller-manager watches Gateway API and Fleet +// ServiceImport resources in the hub cluster. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "os" + + "k8s.io/apimachinery/pkg/runtime" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + _ "k8s.io/client-go/plugin/pkg/client/auth" + "k8s.io/client-go/rest" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/healthz" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" + + "go.goms.io/fleet/pkg/utils/cloudconfig/azure" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +var ( + metricsAddr = flag.String("metrics-bind-address", ":8082", "The address the metrics endpoint binds to.") + probeAddr = flag.String("health-probe-bind-address", ":8083", "The address the health probe endpoint binds to.") + + enableLeaderElection = flag.Bool("leader-elect", true, "Enable leader election for the controller manager.") + leaderElectionNamespace = flag.String("leader-election-namespace", "fleet-system", "The namespace used for leader election.") + + enableAFD = flag.Bool("enable-afd", false, "Enable Azure Front Door reconciliation.") + cloudConfigFile = flag.String("cloud-config", "/etc/kubernetes/provider/azure.json", "The Azure cloud configuration file.") + afdResourceGroup = flag.String("afd-resource-group", "", "The resource group in which the controller manages Azure Front Door resources.") +) + +func init() { + // Register klog flags before flag.Parse so production verbosity and output + // settings are available consistently with the existing controller binaries. + klog.InitFlags(nil) +} + +func main() { + flag.Parse() + defer klog.Flush() + + ctrl.SetLogger(zap.New(zap.UseDevMode(true))) + + if err := run(currentOptions(), productionDependencies()); err != nil { + klog.ErrorS(err, "Problem running hub Gateway controller manager") + os.Exit(1) + } +} + +func currentOptions() managerOptions { + return managerOptions{ + metricsAddress: *metricsAddr, + probeAddress: *probeAddr, + leaderElection: *enableLeaderElection, + leaderElectionNamespace: *leaderElectionNamespace, + enableAFD: *enableAFD, + cloudConfigFile: *cloudConfigFile, + afdResourceGroup: *afdResourceGroup, + } +} + +type managerOptions struct { + metricsAddress string + probeAddress string + leaderElection bool + leaderElectionNamespace string + enableAFD bool + cloudConfigFile string + afdResourceGroup string +} + +type controllerManager interface { + AddHealthzCheck(string, healthz.Checker) error + AddReadyzCheck(string, healthz.Checker) error + Start(context.Context) error +} + +type dependencies struct { + getConfig func() *rest.Config + newManager func(*rest.Config, ctrl.Options) (controllerManager, error) + signalHandler func() context.Context + loadAFDConfig func(string, string) (*azure.CloudConfig, error) + newScheme func() (*runtime.Scheme, error) +} + +func productionDependencies() dependencies { + return dependencies{ + getConfig: ctrl.GetConfigOrDie, + newManager: func(config *rest.Config, options ctrl.Options) (controllerManager, error) { + return ctrl.NewManager(config, options) + }, + signalHandler: ctrl.SetupSignalHandler, + loadAFDConfig: loadAFDConfiguration, + newScheme: newScheme, + } +} + +func run(options managerOptions, deps dependencies) error { + scheme, err := deps.newScheme() + if err != nil { + return fmt.Errorf("register controller schemes: %w", err) + } + + if options.enableAFD { + if _, err := deps.loadAFDConfig(options.cloudConfigFile, options.afdResourceGroup); err != nil { + return fmt.Errorf("load Azure Front Door configuration: %w", err) + } + // The feature flag intentionally performs configuration validation only + // until the Gateway reconcilers are introduced in the next slice. + klog.InfoS("Azure Front Door configuration is valid; no reconcilers are registered in the foundation release") + } else { + klog.InfoS("Azure Front Door reconciliation is disabled") + } + + mgr, err := deps.newManager(deps.getConfig(), ctrl.Options{ + Scheme: scheme, + Metrics: metricsserver.Options{ + BindAddress: options.metricsAddress, + }, + HealthProbeBindAddress: options.probeAddress, + LeaderElection: options.leaderElection, + LeaderElectionNamespace: options.leaderElectionNamespace, + LeaderElectionID: "hub-gateway-controller-manager.networking.fleet.azure.com", + }) + if err != nil { + return fmt.Errorf("create hub Gateway controller manager: %w", err) + } + + if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil { + return fmt.Errorf("set up health check: %w", err) + } + if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil { + return fmt.Errorf("set up readiness check: %w", err) + } + + klog.InfoS("Starting hub Gateway controller manager") + if err := mgr.Start(deps.signalHandler()); err != nil { + return fmt.Errorf("start hub Gateway controller manager: %w", err) + } + return nil +} + +func newScheme() (*runtime.Scheme, error) { + scheme := runtime.NewScheme() + installers := []func(*runtime.Scheme) error{ + clientgoscheme.AddToScheme, + gatewayv1.Install, + // ReferenceGrant remains v1beta1 in Gateway API v1.2.1. + gatewayv1beta1.Install, + fleetnetv1alpha1.AddToScheme, + } + for _, install := range installers { + if err := install(scheme); err != nil { + return nil, err + } + } + return scheme, nil +} + +func loadAFDConfiguration(cloudConfigPath, resourceGroup string) (*azure.CloudConfig, error) { + if resourceGroup == "" { + return nil, errors.New("afd-resource-group must be configured when Azure Front Door reconciliation is enabled") + } + cloudConfig, err := azure.NewCloudConfigFromFile(cloudConfigPath) + if err != nil { + return nil, err + } + if cloudConfig.SubscriptionID == "" { + return nil, errors.New("Azure subscription ID must be configured") + } + cloudConfig.SetUserAgent("fleet-hub-gateway-controller-manager") + return cloudConfig, nil +} diff --git a/cmd/hub-gateway-controller-manager/main_test.go b/cmd/hub-gateway-controller-manager/main_test.go new file mode 100644 index 00000000..4d8f5d14 --- /dev/null +++ b/cmd/hub-gateway-controller-manager/main_test.go @@ -0,0 +1,348 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package main + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/rest" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/healthz" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" + + "go.goms.io/fleet/pkg/utils/cloudconfig/azure" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +func TestNewScheme_RegistersGatewayAndFleetTypes(t *testing.T) { + scheme, err := newScheme() + if err != nil { + t.Fatalf("newScheme() error = %v", err) + } + + tests := []struct { + name string + gvk schema.GroupVersionKind + }{ + { + name: "Gateway", + gvk: gatewayv1.SchemeGroupVersion.WithKind("Gateway"), + }, + { + name: "HTTPRoute", + gvk: gatewayv1.SchemeGroupVersion.WithKind("HTTPRoute"), + }, + { + name: "ReferenceGrant", + gvk: gatewayv1beta1.SchemeGroupVersion.WithKind("ReferenceGrant"), + }, + { + name: "Fleet ServiceImport", + gvk: fleetnetv1alpha1.GroupVersion.WithKind("ServiceImport"), + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if !scheme.Recognizes(tt.gvk) { + t.Errorf("scheme does not recognize %s", tt.gvk) + } + }) + } +} + +func TestNewScheme_PreservesFleetServiceImportGVK(t *testing.T) { + scheme, err := newScheme() + if err != nil { + t.Fatalf("newScheme() error = %v", err) + } + + gvks, _, err := scheme.ObjectKinds(&fleetnetv1alpha1.ServiceImport{}) + if err != nil { + t.Fatalf("scheme.ObjectKinds(ServiceImport) error = %v", err) + } + + want := fleetnetv1alpha1.GroupVersion.WithKind("ServiceImport") + for _, got := range gvks { + if got == want { + return + } + } + t.Errorf("ServiceImport GVKs = %v, want to contain %s", gvks, want) +} + +func TestCurrentOptions_UsesCommandLineConfiguration(t *testing.T) { + got := currentOptions() + + if got.metricsAddress != *metricsAddr || + got.probeAddress != *probeAddr || + got.leaderElection != *enableLeaderElection || + got.leaderElectionNamespace != *leaderElectionNamespace || + got.enableAFD != *enableAFD || + got.cloudConfigFile != *cloudConfigFile || + got.afdResourceGroup != *afdResourceGroup { + t.Errorf("currentOptions() = %#v, want values from command-line flags", got) + } +} + +func TestProductionDependencies_AreConfigured(t *testing.T) { + got := productionDependencies() + + if got.getConfig == nil || got.newManager == nil || got.signalHandler == nil || got.loadAFDConfig == nil || got.newScheme == nil { + t.Errorf("productionDependencies() = %#v, want all dependencies configured", got) + } + + // Constructing a manager exercises the production adapter without starting + // it or contacting a Kubernetes API server. + if _, err := got.newManager(&rest.Config{Host: "https://127.0.0.1"}, ctrl.Options{}); err != nil { + t.Fatalf("production newManager() error = %v", err) + } +} + +func TestRun_ManagesStartupLifecycle(t *testing.T) { + tests := []struct { + name string + options managerOptions + schemeError error + loadConfigError error + newManagerError error + healthError error + readyError error + startError error + wantError string + wantConfigLoads int + wantManagerStarts int + }{ + { + name: "scheme registration failure stops startup", + schemeError: errors.New("scheme failed"), + wantError: "register controller schemes", + }, + { + name: "disabled AFD starts manager without loading Azure configuration", + options: managerOptions{ + metricsAddress: ":8082", + probeAddress: ":8083", + leaderElection: true, + leaderElectionNamespace: "fleet-system", + }, + wantManagerStarts: 1, + }, + { + name: "enabled AFD validates configuration and starts manager", + options: managerOptions{ + enableAFD: true, + cloudConfigFile: "provider.json", + afdResourceGroup: "afd-rg", + }, + wantConfigLoads: 1, + wantManagerStarts: 1, + }, + { + name: "configuration failure stops startup", + options: managerOptions{ + enableAFD: true, + }, + loadConfigError: errors.New("invalid cloud config"), + wantError: "load Azure Front Door configuration", + wantConfigLoads: 1, + }, + { + name: "manager creation failure is returned", + newManagerError: errors.New("manager failed"), + wantError: "create hub Gateway controller manager", + }, + { + name: "health registration failure is returned", + healthError: errors.New("health failed"), + wantError: "set up health check", + }, + { + name: "readiness registration failure is returned", + readyError: errors.New("ready failed"), + wantError: "set up readiness check", + }, + { + name: "manager start failure is returned", + startError: errors.New("start failed"), + wantError: "start hub Gateway controller manager", + wantManagerStarts: 1, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + manager := &fakeControllerManager{ + healthError: tt.healthError, + readyError: tt.readyError, + startError: tt.startError, + } + configLoads := 0 + var receivedOptions ctrl.Options + deps := dependencies{ + getConfig: func() *rest.Config { + return &rest.Config{Host: "https://hub.example.com"} + }, + newManager: func(_ *rest.Config, options ctrl.Options) (controllerManager, error) { + receivedOptions = options + if tt.newManagerError != nil { + return nil, tt.newManagerError + } + return manager, nil + }, + signalHandler: func() context.Context { + return context.Background() + }, + loadAFDConfig: func(_, _ string) (*azure.CloudConfig, error) { + configLoads++ + if tt.loadConfigError != nil { + return nil, tt.loadConfigError + } + return &azure.CloudConfig{SubscriptionID: "00000000-0000-0000-0000-000000000000"}, nil + }, + newScheme: func() (*runtime.Scheme, error) { + if tt.schemeError != nil { + return nil, tt.schemeError + } + return newScheme() + }, + } + + err := run(tt.options, deps) + if tt.wantError != "" { + if err == nil { + t.Fatalf("run() error = nil, want containing %q", tt.wantError) + } + if !strings.Contains(err.Error(), tt.wantError) { + t.Errorf("run() error = %q, want containing %q", err, tt.wantError) + } + } else if err != nil { + t.Fatalf("run() error = %v", err) + } + + if configLoads != tt.wantConfigLoads { + t.Errorf("Azure configuration loads = %d, want %d", configLoads, tt.wantConfigLoads) + } + if manager.starts != tt.wantManagerStarts { + t.Errorf("manager starts = %d, want %d", manager.starts, tt.wantManagerStarts) + } + if tt.schemeError == nil && tt.newManagerError == nil && tt.loadConfigError == nil { + if receivedOptions.Scheme == nil { + t.Error("manager options Scheme = nil, want registered scheme") + } + if receivedOptions.Metrics.BindAddress != tt.options.metricsAddress { + t.Errorf("metrics address = %q, want %q", receivedOptions.Metrics.BindAddress, tt.options.metricsAddress) + } + } + }) + } +} + +func TestLoadAFDConfiguration_RejectsInvalidInput(t *testing.T) { + tests := []struct { + name string + cloudConfig string + resourceGroup string + wantError string + }{ + { + name: "missing resource group", + cloudConfig: "provider.json", + wantError: "afd-resource-group", + }, + { + name: "missing cloud config file", + cloudConfig: "does-not-exist.json", + resourceGroup: "afd-rg", + wantError: "does-not-exist.json", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := loadAFDConfiguration(tt.cloudConfig, tt.resourceGroup) + if err == nil { + t.Fatalf("loadAFDConfiguration() error = nil, want containing %q", tt.wantError) + } + if !strings.Contains(err.Error(), tt.wantError) { + t.Errorf("loadAFDConfiguration() error = %q, want containing %q", err, tt.wantError) + } + }) + } +} + +func TestLoadAFDConfiguration_LoadsValidConfiguration(t *testing.T) { + path := filepath.Join(t.TempDir(), "provider.json") + const subscriptionID = "00000000-0000-0000-0000-000000000000" + config := `{ + "cloud":"AzurePublicCloud", + "location":"eastus", + "subscriptionID":"` + subscriptionID + `", + "resourceGroup":"fleet-rg", + "useManagedIdentityExtension":true + }` + if err := os.WriteFile(path, []byte(config), 0o600); err != nil { + t.Fatalf("os.WriteFile() error = %v", err) + } + + got, err := loadAFDConfiguration(path, "afd-rg") + if err != nil { + t.Fatalf("loadAFDConfiguration() error = %v", err) + } + if got.SubscriptionID != subscriptionID { + t.Errorf("SubscriptionID = %q, want %q", got.SubscriptionID, subscriptionID) + } +} + +func TestLoadAFDConfiguration_RejectsMissingSubscription(t *testing.T) { + path := filepath.Join(t.TempDir(), "provider.json") + config := `{ + "cloud":"AzurePublicCloud", + "location":"eastus", + "resourceGroup":"fleet-rg", + "useManagedIdentityExtension":true + }` + if err := os.WriteFile(path, []byte(config), 0o600); err != nil { + t.Fatalf("os.WriteFile() error = %v", err) + } + + _, err := loadAFDConfiguration(path, "afd-rg") + if err == nil { + t.Fatal("loadAFDConfiguration() error = nil, want missing subscription error") + } + if !strings.Contains(err.Error(), "subscription ID") { + t.Errorf("loadAFDConfiguration() error = %q, want missing subscription ID", err) + } +} + +type fakeControllerManager struct { + healthError error + readyError error + startError error + starts int +} + +func (f *fakeControllerManager) AddHealthzCheck(_ string, _ healthz.Checker) error { + return f.healthError +} + +func (f *fakeControllerManager) AddReadyzCheck(_ string, _ healthz.Checker) error { + return f.readyError +} + +func (f *fakeControllerManager) Start(_ context.Context) error { + f.starts++ + return f.startError +} diff --git a/go.mod b/go.mod index c698ae16..2bc0afb9 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,8 @@ go 1.25.12 require go.goms.io/fleet v0.14.0 +require sigs.k8s.io/gateway-api v1.2.1 + require ( github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.0 diff --git a/go.sum b/go.sum index a4654e6b..72d0435f 100644 --- a/go.sum +++ b/go.sum @@ -315,6 +315,8 @@ sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50 h1:l9igMANNptVwYmZrqGS51oW sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50/go.mod h1:1M90A+akyTabHVnveSKlvIO/Kk9kEr1LjRx+08twKVU= sigs.k8s.io/controller-runtime v0.19.0 h1:nWVM7aq+Il2ABxwiCizrVDSlmDcshi9llbaFbC0ji/Q= sigs.k8s.io/controller-runtime v0.19.0/go.mod h1:iRmWllt8IlaLjvTTDLhRBXIEtkCK6hwVBJJsYS9Ajf4= +sigs.k8s.io/gateway-api v1.2.1 h1:fZZ/+RyRb+Y5tGkwxFKuYuSRQHu9dZtbjenblleOLHM= +sigs.k8s.io/gateway-api v1.2.1/go.mod h1:EpNfEXNjiYfUJypf0eZ0P5iXA9ekSGWaS1WgPaM42X0= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= diff --git a/pkg/annotations/afd.go b/pkg/annotations/afd.go new file mode 100644 index 00000000..f229efcc --- /dev/null +++ b/pkg/annotations/afd.go @@ -0,0 +1,211 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Package annotations parses the Azure Front Door annotations supported by the +// Fleet Gateway controller. +package annotations + +import ( + "fmt" + "net/url" + "regexp" + "strings" + + "k8s.io/apimachinery/pkg/util/validation" +) + +const ( + annotationPrefix = "networking.fleet.azure.com/" + afdAnnotationPrefix = annotationPrefix + "afd-" + + // AFDSKUAnnotation selects the Azure Front Door SKU for a Gateway. + AFDSKUAnnotation = afdAnnotationPrefix + "sku" + // AFDResourceGroupAnnotation is reserved for a future ownership model. + AFDResourceGroupAnnotation = afdAnnotationPrefix + "resource-group" + // AFDWAFPolicyIDAnnotation attaches an existing Azure Front Door WAF policy to a Gateway. + AFDWAFPolicyIDAnnotation = afdAnnotationPrefix + "waf-policy-id" + + // AFDOriginConnectivityAnnotation selects how Azure Front Door reaches a ServiceImport. + AFDOriginConnectivityAnnotation = afdAnnotationPrefix + "origin-connectivity" + // AFDHealthProbePathAnnotation configures the origin-group HTTP health probe path. + AFDHealthProbePathAnnotation = afdAnnotationPrefix + "health-probe-path" + // AFDOriginHostHeaderAnnotation overrides the Host header sent to origins. + AFDOriginHostHeaderAnnotation = afdAnnotationPrefix + "origin-host-header" +) + +var subscriptionIDPattern = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) + +// SKU is an Azure Front Door Standard/Premium SKU name. +type SKU string + +const ( + // SKUStandard is the Azure Front Door Standard SKU. + SKUStandard SKU = "Standard_AzureFrontDoor" + // SKUPremium is the Azure Front Door Premium SKU. + SKUPremium SKU = "Premium_AzureFrontDoor" +) + +// Connectivity selects the origin connectivity mode for every member origin +// represented by one ServiceImport. +type Connectivity string + +const ( + // ConnectivityAuto selects a fully private topology when available and otherwise a fully public topology. + ConnectivityAuto Connectivity = "auto" + // ConnectivityPublic requires all origins to expose public endpoints. + ConnectivityPublic Connectivity = "public" + // ConnectivityPrivateLink requires all origins to expose Azure Private Link Services. + ConnectivityPrivateLink Connectivity = "private-link" +) + +// GatewayConfig is the typed Azure configuration derived from Gateway annotations. +type GatewayConfig struct { + SKU SKU + WAFPolicyID string +} + +// ServiceImportConfig is the typed Azure configuration derived from ServiceImport annotations. +type ServiceImportConfig struct { + Connectivity Connectivity + HealthProbePath string + OriginHostHeader string +} + +// ParseGatewayConfig parses annotations that are valid on a Gateway. +func ParseGatewayConfig(annotations map[string]string, defaultSKU SKU) (GatewayConfig, error) { + if !defaultSKU.valid() { + return GatewayConfig{}, fmt.Errorf("default SKU %q is not supported", defaultSKU) + } + + allowed := map[string]struct{}{ + AFDSKUAnnotation: {}, + AFDResourceGroupAnnotation: {}, + AFDWAFPolicyIDAnnotation: {}, + } + if err := rejectUnsupportedAFDAnnotations(annotations, allowed); err != nil { + return GatewayConfig{}, err + } + + // Resource-group selection is intentionally controller-wide until resource + // adoption, RBAC scoping, and deletion ownership are designed together. + if value, found := annotations[AFDResourceGroupAnnotation]; found { + return GatewayConfig{}, fmt.Errorf("annotation %q with value %q is reserved and not supported", AFDResourceGroupAnnotation, value) + } + + config := GatewayConfig{SKU: defaultSKU} + if value, found := annotations[AFDSKUAnnotation]; found { + config.SKU = SKU(value) + if !config.SKU.valid() { + return GatewayConfig{}, fmt.Errorf("annotation %q has unsupported value %q; expected %q or %q", AFDSKUAnnotation, value, SKUStandard, SKUPremium) + } + } + + if value, found := annotations[AFDWAFPolicyIDAnnotation]; found { + if err := validateWAFPolicyID(value); err != nil { + return GatewayConfig{}, fmt.Errorf("annotation %q has invalid value %q: %w", AFDWAFPolicyIDAnnotation, value, err) + } + config.WAFPolicyID = value + } + return config, nil +} + +// ParseServiceImportConfig parses annotations that are valid on a Fleet ServiceImport. +func ParseServiceImportConfig(annotations map[string]string) (ServiceImportConfig, error) { + allowed := map[string]struct{}{ + AFDOriginConnectivityAnnotation: {}, + AFDHealthProbePathAnnotation: {}, + AFDOriginHostHeaderAnnotation: {}, + } + if err := rejectUnsupportedAFDAnnotations(annotations, allowed); err != nil { + return ServiceImportConfig{}, err + } + + config := ServiceImportConfig{ + Connectivity: ConnectivityAuto, + HealthProbePath: "/", + } + if value, found := annotations[AFDOriginConnectivityAnnotation]; found { + config.Connectivity = Connectivity(value) + if !config.Connectivity.valid() { + return ServiceImportConfig{}, fmt.Errorf("annotation %q has unsupported value %q; expected %q, %q, or %q", AFDOriginConnectivityAnnotation, value, ConnectivityAuto, ConnectivityPublic, ConnectivityPrivateLink) + } + } + if value, found := annotations[AFDHealthProbePathAnnotation]; found { + if err := validateHealthProbePath(value); err != nil { + return ServiceImportConfig{}, fmt.Errorf("annotation %q has invalid value %q: %w", AFDHealthProbePathAnnotation, value, err) + } + config.HealthProbePath = value + } + if value, found := annotations[AFDOriginHostHeaderAnnotation]; found { + if errs := validation.IsDNS1123Subdomain(value); len(errs) != 0 { + return ServiceImportConfig{}, fmt.Errorf("annotation %q has invalid value %q: expected a lowercase DNS hostname: %s", AFDOriginHostHeaderAnnotation, value, strings.Join(errs, "; ")) + } + config.OriginHostHeader = value + } + return config, nil +} + +// ValidateCompatibility checks settings that span a Gateway and one of its ServiceImport backends. +func ValidateCompatibility(gateway GatewayConfig, serviceImport ServiceImportConfig) error { + if serviceImport.Connectivity == ConnectivityPrivateLink && gateway.SKU != SKUPremium { + return fmt.Errorf("connectivity %q requires Azure Front Door SKU %q", ConnectivityPrivateLink, SKUPremium) + } + return nil +} + +func (s SKU) valid() bool { + return s == SKUStandard || s == SKUPremium +} + +func (c Connectivity) valid() bool { + return c == ConnectivityAuto || c == ConnectivityPublic || c == ConnectivityPrivateLink +} + +func rejectUnsupportedAFDAnnotations(annotations map[string]string, allowed map[string]struct{}) error { + for key, value := range annotations { + if !strings.HasPrefix(key, afdAnnotationPrefix) { + continue + } + if _, found := allowed[key]; !found { + // Rejecting unknown reserved keys prevents misspellings from becoming + // silent no-ops, which is the primary validation risk of annotations. + return fmt.Errorf("annotation %q with value %q is not supported on this resource", key, value) + } + } + return nil +} + +func validateHealthProbePath(value string) error { + if !strings.HasPrefix(value, "/") { + return fmt.Errorf("expected an absolute path beginning with '/'") + } + parsed, err := url.ParseRequestURI(value) + if err != nil { + return fmt.Errorf("expected a valid URI path: %w", err) + } + if parsed.IsAbs() || parsed.Host != "" || parsed.RawQuery != "" || parsed.Fragment != "" { + return fmt.Errorf("expected a path without scheme, host, query, or fragment") + } + return nil +} + +func validateWAFPolicyID(value string) error { + parts := strings.Split(strings.Trim(value, "/"), "/") + if len(parts) != 8 || + !strings.EqualFold(parts[0], "subscriptions") || + !strings.EqualFold(parts[2], "resourceGroups") || + !strings.EqualFold(parts[4], "providers") || + !strings.EqualFold(parts[5], "Microsoft.Network") || + !strings.EqualFold(parts[6], "frontdoorWebApplicationFirewallPolicies") { + return fmt.Errorf("expected an Azure Front Door WAF policy resource ID") + } + if !subscriptionIDPattern.MatchString(parts[1]) { + return fmt.Errorf("subscription ID %q is not a GUID", parts[1]) + } + if parts[3] == "" || parts[7] == "" { + return fmt.Errorf("resource group and policy name must not be empty") + } + return nil +} diff --git a/pkg/annotations/afd_test.go b/pkg/annotations/afd_test.go new file mode 100644 index 00000000..a38674a3 --- /dev/null +++ b/pkg/annotations/afd_test.go @@ -0,0 +1,271 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package annotations + +import ( + "strings" + "testing" +) + +func TestParseGatewayConfig(t *testing.T) { + const validWAFPolicyID = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/security-rg/providers/Microsoft.Network/frontdoorWebApplicationFirewallPolicies/store-waf" + + tests := []struct { + name string + annotations map[string]string + defaultSKU SKU + want GatewayConfig + wantErr string + }{ + { + name: "uses controller default", + defaultSKU: SKUPremium, + want: GatewayConfig{ + SKU: SKUPremium, + }, + }, + { + name: "parses explicit supported values", + annotations: map[string]string{ + AFDSKUAnnotation: string(SKUStandard), + AFDWAFPolicyIDAnnotation: validWAFPolicyID, + }, + defaultSKU: SKUStandard, + want: GatewayConfig{ + SKU: SKUStandard, + WAFPolicyID: validWAFPolicyID, + }, + }, + { + name: "rejects invalid explicit SKU", + annotations: map[string]string{ + AFDSKUAnnotation: "premium", + }, + defaultSKU: SKUStandard, + wantErr: AFDSKUAnnotation, + }, + { + name: "rejects invalid controller default", + defaultSKU: SKU("invalid"), + wantErr: "default SKU", + }, + { + name: "rejects malformed WAF resource ID", + annotations: map[string]string{ + AFDWAFPolicyIDAnnotation: "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Network/notWaf/name", + }, + defaultSKU: SKUPremium, + wantErr: AFDWAFPolicyIDAnnotation, + }, + { + name: "rejects reserved resource group override", + annotations: map[string]string{ + AFDResourceGroupAnnotation: "application-rg", + }, + defaultSKU: SKUPremium, + wantErr: AFDResourceGroupAnnotation, + }, + { + name: "rejects unknown reserved annotation", + annotations: map[string]string{ + annotationPrefix + "afd-skku": "Premium_AzureFrontDoor", + }, + defaultSKU: SKUPremium, + wantErr: annotationPrefix + "afd-skku", + }, + { + name: "rejects backend annotation on Gateway", + annotations: map[string]string{ + AFDOriginConnectivityAnnotation: string(ConnectivityPublic), + }, + defaultSKU: SKUPremium, + wantErr: AFDOriginConnectivityAnnotation, + }, + { + name: "ignores unrelated annotation", + annotations: map[string]string{ + "example.com/owner": "team-a", + }, + defaultSKU: SKUStandard, + want: GatewayConfig{ + SKU: SKUStandard, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ParseGatewayConfig(tt.annotations, tt.defaultSKU) + if tt.wantErr != "" { + if err == nil { + t.Fatalf("ParseGatewayConfig() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("ParseGatewayConfig() error = %q, want containing %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("ParseGatewayConfig() error = %v", err) + } + if got != tt.want { + t.Errorf("ParseGatewayConfig() = %#v, want %#v", got, tt.want) + } + }) + } +} + +func TestParseServiceImportConfig(t *testing.T) { + tests := []struct { + name string + annotations map[string]string + want ServiceImportConfig + wantErr string + }{ + { + name: "uses defaults", + want: ServiceImportConfig{ + Connectivity: ConnectivityAuto, + HealthProbePath: "/", + }, + }, + { + name: "parses explicit values", + annotations: map[string]string{ + AFDOriginConnectivityAnnotation: string(ConnectivityPrivateLink), + AFDHealthProbePathAnnotation: "/healthz", + AFDOriginHostHeaderAnnotation: "api.example.com", + }, + want: ServiceImportConfig{ + Connectivity: ConnectivityPrivateLink, + HealthProbePath: "/healthz", + OriginHostHeader: "api.example.com", + }, + }, + { + name: "rejects invalid connectivity", + annotations: map[string]string{ + AFDOriginConnectivityAnnotation: "private", + }, + wantErr: AFDOriginConnectivityAnnotation, + }, + { + name: "rejects relative health probe", + annotations: map[string]string{ + AFDHealthProbePathAnnotation: "healthz", + }, + wantErr: AFDHealthProbePathAnnotation, + }, + { + name: "rejects health probe query", + annotations: map[string]string{ + AFDHealthProbePathAnnotation: "/healthz?deep=true", + }, + wantErr: AFDHealthProbePathAnnotation, + }, + { + name: "rejects invalid origin host header", + annotations: map[string]string{ + AFDOriginHostHeaderAnnotation: "https://api.example.com", + }, + wantErr: AFDOriginHostHeaderAnnotation, + }, + { + name: "rejects Gateway annotation on ServiceImport", + annotations: map[string]string{ + AFDSKUAnnotation: string(SKUPremium), + }, + wantErr: AFDSKUAnnotation, + }, + { + name: "ignores existing Fleet weight annotation", + annotations: map[string]string{ + annotationPrefix + "weight": "100", + }, + want: ServiceImportConfig{ + Connectivity: ConnectivityAuto, + HealthProbePath: "/", + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ParseServiceImportConfig(tt.annotations) + if tt.wantErr != "" { + if err == nil { + t.Fatalf("ParseServiceImportConfig() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("ParseServiceImportConfig() error = %q, want containing %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("ParseServiceImportConfig() error = %v", err) + } + if got != tt.want { + t.Errorf("ParseServiceImportConfig() = %#v, want %#v", got, tt.want) + } + }) + } +} + +func TestValidateCompatibility(t *testing.T) { + tests := []struct { + name string + gateway GatewayConfig + service ServiceImportConfig + wantErr string + }{ + { + name: "private link with Premium is valid", + gateway: GatewayConfig{ + SKU: SKUPremium, + }, + service: ServiceImportConfig{ + Connectivity: ConnectivityPrivateLink, + }, + }, + { + name: "private link with Standard is invalid", + gateway: GatewayConfig{ + SKU: SKUStandard, + }, + service: ServiceImportConfig{ + Connectivity: ConnectivityPrivateLink, + }, + wantErr: string(SKUPremium), + }, + { + name: "public with Standard is valid", + gateway: GatewayConfig{ + SKU: SKUStandard, + }, + service: ServiceImportConfig{ + Connectivity: ConnectivityPublic, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := ValidateCompatibility(tt.gateway, tt.service) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("ValidateCompatibility() error = %v", err) + } + return + } + if err == nil { + t.Fatalf("ValidateCompatibility() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("ValidateCompatibility() error = %q, want containing %q", err, tt.wantErr) + } + }) + } +} diff --git a/pkg/controllers/hub/gatewaymodel/model.go b/pkg/controllers/hub/gatewaymodel/model.go new file mode 100644 index 00000000..767ed195 --- /dev/null +++ b/pkg/controllers/hub/gatewaymodel/model.go @@ -0,0 +1,206 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Package gatewaymodel defines the provider-neutral desired state produced from +// Gateway API and Fleet ServiceImport resources. +package gatewaymodel + +import ( + "fmt" + "sort" +) + +const maxServiceExportWeight = 1000 +const maxHTTPRouteBackendWeight = 1000000 + +// GlobalGateway is the normalized desired state for one Gateway. +type GlobalGateway struct { + Namespace string + Name string + UID string + Listeners []Listener + Routes []Route + WAFPolicyID string +} + +// Listener is a normalized Gateway listener. +type Listener struct { + Name string + Protocol string + Port int32 + Hostname string +} + +// Route is a normalized HTTPRoute attached to a Gateway. +type Route struct { + Namespace string + Name string + Hostnames []string + Matches []HTTPMatch + Filters []HTTPFilter + Backends []Backend +} + +// HTTPMatch contains the portable HTTP match fields supported by the initial provider. +type HTTPMatch struct { + PathType string + Path string + Method string +} + +// HTTPFilter contains the provider-neutral representation of one supported HTTPRoute filter. +type HTTPFilter struct { + Type string + StatusCode int32 + Hostname string + Path string +} + +// Backend represents one Fleet ServiceImport referenced by an HTTPRoute. +type Backend struct { + Namespace string + Name string + Port int32 + // RouteWeight splits traffic between logical ServiceImport backends and + // follows the Gateway API HTTPBackendRef range [0, 1,000,000]. + RouteWeight int32 + HealthProbePath string + Origins []Origin +} + +// Origin represents one member-cluster endpoint behind a ServiceImport. +type Origin struct { + Cluster string + Endpoint string + // Weight splits traffic between member clusters behind one ServiceImport + // and follows the existing Fleet ServiceExport range [0, 1000]. + Weight int64 + Connectivity string + PrivateLinkResourceID string + PrivateLinkLocation string +} + +// Normalize validates a model and returns a deeply copied, deterministically +// ordered representation suitable for equality checks and provider reconciliation. +func Normalize(model GlobalGateway) (GlobalGateway, error) { + normalized := clone(model) + if err := validate(normalized); err != nil { + return GlobalGateway{}, err + } + + sort.Slice(normalized.Listeners, func(i, j int) bool { + return normalized.Listeners[i].Name < normalized.Listeners[j].Name + }) + sort.Slice(normalized.Routes, func(i, j int) bool { + left, right := normalized.Routes[i], normalized.Routes[j] + if left.Namespace != right.Namespace { + return left.Namespace < right.Namespace + } + return left.Name < right.Name + }) + for routeIndex := range normalized.Routes { + route := &normalized.Routes[routeIndex] + sort.Strings(route.Hostnames) + sort.Slice(route.Backends, func(i, j int) bool { + left, right := route.Backends[i], route.Backends[j] + if left.Namespace != right.Namespace { + return left.Namespace < right.Namespace + } + if left.Name != right.Name { + return left.Name < right.Name + } + return left.Port < right.Port + }) + for backendIndex := range route.Backends { + sort.Slice(route.Backends[backendIndex].Origins, func(i, j int) bool { + return route.Backends[backendIndex].Origins[i].Cluster < route.Backends[backendIndex].Origins[j].Cluster + }) + } + } + return normalized, nil +} + +func clone(model GlobalGateway) GlobalGateway { + result := model + result.Listeners = append([]Listener(nil), model.Listeners...) + result.Routes = make([]Route, len(model.Routes)) + for routeIndex := range model.Routes { + result.Routes[routeIndex] = model.Routes[routeIndex] + result.Routes[routeIndex].Hostnames = append([]string(nil), model.Routes[routeIndex].Hostnames...) + result.Routes[routeIndex].Matches = append([]HTTPMatch(nil), model.Routes[routeIndex].Matches...) + result.Routes[routeIndex].Filters = append([]HTTPFilter(nil), model.Routes[routeIndex].Filters...) + result.Routes[routeIndex].Backends = make([]Backend, len(model.Routes[routeIndex].Backends)) + for backendIndex := range model.Routes[routeIndex].Backends { + result.Routes[routeIndex].Backends[backendIndex] = model.Routes[routeIndex].Backends[backendIndex] + result.Routes[routeIndex].Backends[backendIndex].Origins = append([]Origin(nil), model.Routes[routeIndex].Backends[backendIndex].Origins...) + } + } + return result +} + +func validate(model GlobalGateway) error { + if model.Namespace == "" || model.Name == "" { + return fmt.Errorf("Gateway identity requires namespace and name") + } + + listeners := make(map[string]struct{}, len(model.Listeners)) + for _, listener := range model.Listeners { + if listener.Name == "" { + return fmt.Errorf("listener name must not be empty") + } + if _, found := listeners[listener.Name]; found { + return fmt.Errorf("duplicate listener %q", listener.Name) + } + listeners[listener.Name] = struct{}{} + if listener.Port < 1 || listener.Port > 65535 { + return fmt.Errorf("listener %q port %d is outside [1, 65535]", listener.Name, listener.Port) + } + } + + routes := make(map[string]struct{}, len(model.Routes)) + for _, route := range model.Routes { + if route.Namespace == "" || route.Name == "" { + return fmt.Errorf("route identity requires namespace and name") + } + routeKey := route.Namespace + "/" + route.Name + if _, found := routes[routeKey]; found { + return fmt.Errorf("duplicate route %q", routeKey) + } + routes[routeKey] = struct{}{} + + backends := make(map[string]struct{}, len(route.Backends)) + for _, backend := range route.Backends { + if backend.Namespace == "" || backend.Name == "" { + return fmt.Errorf("route %q backend identity requires namespace and name", routeKey) + } + backendKey := fmt.Sprintf("%s/%s:%d", backend.Namespace, backend.Name, backend.Port) + if _, found := backends[backendKey]; found { + return fmt.Errorf("route %q has duplicate backend %q", routeKey, backendKey) + } + backends[backendKey] = struct{}{} + if backend.Port < 1 || backend.Port > 65535 { + return fmt.Errorf("route %q backend %q port %d is outside [1, 65535]", routeKey, backendKey, backend.Port) + } + if backend.RouteWeight < 0 || backend.RouteWeight > maxHTTPRouteBackendWeight { + return fmt.Errorf("route %q backend %q route weight %d is outside [0, %d]", routeKey, backendKey, backend.RouteWeight, maxHTTPRouteBackendWeight) + } + + origins := make(map[string]struct{}, len(backend.Origins)) + for _, origin := range backend.Origins { + if origin.Cluster == "" { + return fmt.Errorf("backend %q origin cluster must not be empty", backendKey) + } + if _, found := origins[origin.Cluster]; found { + return fmt.Errorf("backend %q has duplicate origin cluster %q", backendKey, origin.Cluster) + } + origins[origin.Cluster] = struct{}{} + if origin.Weight < 0 || origin.Weight > maxServiceExportWeight { + return fmt.Errorf("backend %q origin %q weight %d is outside [0, %d]", backendKey, origin.Cluster, origin.Weight, maxServiceExportWeight) + } + } + } + } + return nil +} diff --git a/pkg/controllers/hub/gatewaymodel/model_test.go b/pkg/controllers/hub/gatewaymodel/model_test.go new file mode 100644 index 00000000..248418f0 --- /dev/null +++ b/pkg/controllers/hub/gatewaymodel/model_test.go @@ -0,0 +1,212 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package gatewaymodel + +import ( + "reflect" + "strings" + "testing" +) + +func TestNormalize_SortsNestedResources(t *testing.T) { + model := GlobalGateway{ + Namespace: "store", + Name: "global", + Listeners: []Listener{ + {Name: "https", Port: 443}, + {Name: "http", Port: 80}, + }, + Routes: []Route{ + { + Namespace: "store", + Name: "z-route", + Backends: []Backend{ + { + Namespace: "store", + Name: "z-api", + Port: 8080, + Origins: []Origin{ + {Cluster: "west", Weight: 20}, + {Cluster: "east", Weight: 10}, + }, + }, + {Namespace: "store", Name: "a-api", Port: 8080}, + }, + }, + {Namespace: "store", Name: "a-route"}, + }, + } + + got, err := Normalize(model) + if err != nil { + t.Fatalf("Normalize() error = %v", err) + } + + if got.Listeners[0].Name != "http" || got.Listeners[1].Name != "https" { + t.Errorf("listener order = %v, want http then https", got.Listeners) + } + if got.Routes[0].Name != "a-route" || got.Routes[1].Name != "z-route" { + t.Errorf("route order = %v, want a-route then z-route", got.Routes) + } + if got.Routes[1].Backends[0].Name != "a-api" || got.Routes[1].Backends[1].Name != "z-api" { + t.Errorf("backend order = %v, want a-api then z-api", got.Routes[1].Backends) + } + origins := got.Routes[1].Backends[1].Origins + if origins[0].Cluster != "east" || origins[1].Cluster != "west" { + t.Errorf("origin order = %v, want east then west", origins) + } + + if reflect.DeepEqual(model, got) { + t.Error("Normalize() returned a model equal to unsorted input") + } + if model.Listeners[0].Name != "https" { + t.Error("Normalize() mutated the input model") + } +} + +func TestNormalize_RejectsInvalidModels(t *testing.T) { + tests := []struct { + name string + model GlobalGateway + wantErr string + }{ + { + name: "missing Gateway identity", + model: GlobalGateway{ + Listeners: []Listener{{Name: "http", Port: 80}}, + }, + wantErr: "Gateway identity", + }, + { + name: "duplicate listener", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Listeners: []Listener{ + {Name: "http", Port: 80}, + {Name: "http", Port: 8080}, + }, + }, + wantErr: "duplicate listener", + }, + { + name: "duplicate route", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{ + {Namespace: "store", Name: "api"}, + {Namespace: "store", Name: "api"}, + }, + }, + wantErr: "duplicate route", + }, + { + name: "duplicate backend", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{ + {Namespace: "store", Name: "backend", Port: 8080}, + {Namespace: "store", Name: "backend", Port: 8080}, + }, + }}, + }, + wantErr: "duplicate backend", + }, + { + name: "invalid backend port", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 70000, + }}, + }}, + }, + wantErr: "port", + }, + { + name: "invalid route weight", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 8080, + RouteWeight: 1000001, + }}, + }}, + }, + wantErr: "route weight", + }, + { + name: "duplicate origin cluster", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 8080, + Origins: []Origin{ + {Cluster: "east", Weight: 1}, + {Cluster: "east", Weight: 2}, + }, + }}, + }}, + }, + wantErr: "duplicate origin", + }, + { + name: "invalid origin weight", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 8080, + Origins: []Origin{ + {Cluster: "east", Weight: 1001}, + }, + }}, + }}, + }, + wantErr: "weight", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := Normalize(tt.model) + if err == nil { + t.Fatalf("Normalize() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("Normalize() error = %q, want containing %q", err, tt.wantErr) + } + }) + } +} From eac68a5a697d3210c019f891a8360a48e082a91f Mon Sep 17 00:00:00 2001 From: Ramkumar Chinchani Date: Thu, 27 Aug 2026 20:57:12 -0700 Subject: [PATCH 3/4] test(e2e): add Gateway API contract coverage Install the selected Gateway API CRDs in the E2E hub and verify Gateway, HTTPRoute, ReferenceGrant, and Fleet ServiceImport interoperability. Document the managed-hub prerequisite and upgrade path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../2026-08-17-2356-gep-1748-gateway-api.md | 47 ++++ docs/design/gep-1748-gateway-api.md | 15 +- docs/design/gep-1748-implementation-plan.md | 2 + docs/howtos/gateway-api-afd-configuration.md | 21 +- test/e2e/e2e_test.go | 7 + test/e2e/framework/objects.go | 140 +++++++++++ test/e2e/gateway_api_test.go | 238 ++++++++++++++++++ test/scripts/bootstrap.sh | 20 ++ 8 files changed, 483 insertions(+), 7 deletions(-) create mode 100644 test/e2e/gateway_api_test.go diff --git a/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md b/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md index 011ea9e7..7365c04e 100644 --- a/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md +++ b/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md @@ -17,6 +17,9 @@ - The architecture should align with GKE's multi-cluster Gateway requirements where the concepts apply to Fleet and Azure. - The user approved proceeding with implementation after the design commit. +- The user requested PR-scoped E2E coverage for the Gateway API foundation changes. +- The user selected a managed Fleet hub as a prerequisite for Gateway API integration; hubless + Fleet Manager resources are out of scope for this architecture. ## Plan @@ -70,6 +73,26 @@ - Success criteria: unit tests, build, vet, dependency tidy, and diff checks pass. Race validation is delegated to CI because this Windows host does not have a C compiler. +### Phase 5: Add live-cluster Gateway API contract coverage + +- [x] **Task 5.1: Document the managed-hub prerequisite and install the supported Gateway API CRDs + in the E2E hub.** + - Success criteria: the design and configuration guide state that hubless Fleet Manager is + ARM-only and unsupported by this architecture, identify upgrading to a managed hub as the + supported path, and the E2E bootstrap installs the Gateway API v1.2.1 standard CRDs selected by + this PR before tests run. +- [x] **Task 5.2: Add tests-first Gateway API object builders and scheme registration.** + - Success criteria: the E2E client recognizes GatewayClass, Gateway, HTTPRoute, ReferenceGrant, + and the unchanged Fleet ServiceImport GVK without duplicating production annotation parsing. +- [x] **Task 5.3: Add a live API contract E2E scenario.** + - Success criteria: a real hub API server accepts and round-trips the Fleet GatewayClass, + Gateway annotations, same-namespace and ReferenceGrant-authorized cross-namespace HTTPRoutes, + and exact `networking.fleet.azure.com/v1alpha1 ServiceImport` backend references. +- [ ] **Task 5.4: Validate cleanup and the E2E build path.** + - Success criteria: test resources are removed reliably, E2E-tagged packages compile, focused + unit tests and vet pass, formatting is clean, and no Azure Front Door programming is claimed + before reconcilers exist. + ### Detailed checklist - [x] Phase 1 / Task 1.1 completed. @@ -84,6 +107,10 @@ - [x] Phase 4 / Task 4.4 completed. - [x] Phase 4 / Task 4.5 completed. - [x] Phase 4 / Task 4.6 completed. +- [x] Phase 5 / Task 5.1 completed. +- [x] Phase 5 / Task 5.2 completed. +- [x] Phase 5 / Task 5.3 completed. +- [ ] Phase 5 / Task 5.4 completed. ### Overall success criteria @@ -108,6 +135,11 @@ placement documentation. - Use a separate hub Gateway controller manager to isolate Azure permissions, rollout, failures, and ARM throttling. +- Require Fleet Manager with a managed hub for Gateway API integration. Hubless Fleet Manager is + an ARM grouping and update-orchestration boundary without the Kubernetes configuration plane, + MemberCluster resources, ServiceImport aggregation, or controller placement required by this + design. Supporting a designated member or external ARM-native control plane requires a separate + architecture proposal. - Start with direct per-Service member origins; shared per-cluster ingress gateways are deferred. - Treat the Fleet hub as the configuration cluster and install the multi-cluster GatewayClass there. - Accept only Fleet ServiceImport backends for the multi-cluster GatewayClass, matching GKE's @@ -153,6 +185,21 @@ compliance until Private Link reconciliation, enforcement, and end-to-end evidence are complete. - Added the Gateway API dependency, scheme registration tests, manager skeleton, annotation parser, normalized model, Makefile build target, and focused unit tests. +- Documented the managed-hub prerequisite and one-way hubless upgrade path in the design and + operator guide. +- Updated the E2E bootstrap to install the Gateway API v1.2.1 standard-channel CRDs selected by + `go.mod` on the hub cluster. +- Registered Gateway API v1 and v1beta1 types in the E2E client scheme and added reusable builders + for GatewayClass, Gateway, HTTPRoute, ReferenceGrant, and Fleet ServiceImport resources. +- Added a live hub API-server contract scenario covering same-namespace and ReferenceGrant-approved + cross-namespace Fleet ServiceImport backends, AFD annotation round-tripping, exact backend + group/kind/port/weight preservation, and cleanup. +- The scenario creates backing `InternalServiceExport` resources and waits for the existing hub + controllers to derive each `ServiceImport`; it does not create controller-owned imports directly. +- The E2E-tagged packages compiled successfully before the controller-ownership correction. + Formatting, focused tests, vet, and recompilation after that correction are pending because the + local execution backend fails before process startup while attempting to change the DACL on + `C:\Program Files\nodejs`. - Focused tests pass with 81.8% coverage for the manager, 95.1% for annotation parsing, and 89.7% for the normalized model. Race mode could not run locally because `gcc` is not installed; CI will provide race validation. diff --git a/docs/design/gep-1748-gateway-api.md b/docs/design/gep-1748-gateway-api.md index e6f8aa9a..9e37f214 100644 --- a/docs/design/gep-1748-gateway-api.md +++ b/docs/design/gep-1748-gateway-api.md @@ -187,7 +187,13 @@ group or translate the reference outside this design. ### Hub cluster -The Fleet hub acts as the Gateway API configuration cluster. It contains: +This architecture requires an Azure Kubernetes Fleet Manager resource with a managed hub. A +hubless Fleet Manager provides an ARM management boundary but does not provide the Kubernetes +configuration cluster, hub-side `ServiceImport` aggregation, or controller placement required by +this design. Hubless fleets must be upgraded to a managed hub before enabling Gateway API +integration; managed-hub to hubless downgrade is not supported. + +The managed Fleet hub acts as the Gateway API configuration cluster. It contains: - One platform-managed `GatewayClass` for the AFD implementation. - One or more user-created `Gateway` resources. @@ -231,6 +237,8 @@ requirements. ### Fleet and hub requirements +- The Fleet Manager has a managed hub. Hubless Fleet Manager resources are unsupported by this + architecture. - All target AKS clusters are registered as healthy members of the same Fleet. - Fleet member and hub networking controllers are installed and healthy. - Gateway API CRDs for the selected supported version are installed on the hub. @@ -239,6 +247,11 @@ requirements. - The hub has a durable lifecycle and backup/recovery process because it is the configuration cluster. +To upgrade an existing hubless Fleet Manager, enable its managed hub and then reconcile existing +members as described in the +[Fleet hub upgrade guidance](https://learn.microsoft.com/azure/kubernetes-fleet/upgrade-hub-cluster-type). +The upgrade is one-way: a Fleet Manager with a managed hub cannot be converted back to hubless. + ### Member-cluster requirements - The Fleet member networking agent is installed and can publish internal export state to the hub. diff --git a/docs/design/gep-1748-implementation-plan.md b/docs/design/gep-1748-implementation-plan.md index 12bd6990..b53b9a96 100644 --- a/docs/design/gep-1748-implementation-plan.md +++ b/docs/design/gep-1748-implementation-plan.md @@ -19,6 +19,8 @@ 7. Keep AFD in a separate hub controller manager. 8. Fail explicitly through Gateway API conditions; never silently downgrade requested behavior. 9. Keep each implementation pull request independently reviewable and testable. +10. Require a Fleet Manager managed hub; hubless Fleet Manager resources need a separate + architecture. ## Phase 0: Confirm API and dependency baselines diff --git a/docs/howtos/gateway-api-afd-configuration.md b/docs/howtos/gateway-api-afd-configuration.md index 432a3fd4..d123960b 100644 --- a/docs/howtos/gateway-api-afd-configuration.md +++ b/docs/howtos/gateway-api-afd-configuration.md @@ -25,18 +25,27 @@ later implementation phase. ## Common prerequisites +> [!IMPORTANT] +> This integration requires a Fleet Manager with a managed hub. Hubless Fleet Manager resources +> do not have the Kubernetes configuration plane or hub-side `ServiceImport` aggregation used by +> the Gateway controller. Upgrade a hubless Fleet Manager by enabling its hub and reconciling +> existing members before following this guide. The upgrade cannot be reversed. + Before configuring either topology: -1. Register every workload cluster as a healthy member of the same Fleet. -2. Install the Fleet hub and member networking controllers. -3. Install Gateway API `v1.2.1` CRDs in the Fleet hub. -4. Install the proposed `hub-gateway-controller-manager` in the Fleet hub. -5. Configure the controller with: +1. Confirm that the Fleet Manager has a managed hub. For a hubless Fleet Manager, follow the + [hub upgrade guidance](https://learn.microsoft.com/azure/kubernetes-fleet/upgrade-hub-cluster-type) + and reconcile its existing members. +2. Register every workload cluster as a healthy member of the same Fleet. +3. Install the Fleet hub and member networking controllers. +4. Install Gateway API `v1.2.1` CRDs in the Fleet hub. +5. Install the proposed `hub-gateway-controller-manager` in the Fleet hub. +6. Configure the controller with: - An Azure subscription. - A controller-wide AFD resource group. - An approved workload or managed identity. - Permission to manage AFD resources in the configured resource group. -6. Confirm the platform-installed GatewayClass is accepted: +7. Confirm the platform-installed GatewayClass is accepted: ```bash kubectl get gatewayclass azure-fleet-afd diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index 05ec7b34..d6d6ec4c 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -16,11 +16,14 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" corev1 "k8s.io/api/core/v1" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" utilruntime "k8s.io/apimachinery/pkg/util/runtime" clientgoscheme "k8s.io/client-go/kubernetes/scheme" "sigs.k8s.io/cloud-provider-azure/pkg/azclient/publicipaddressclient" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" fleetv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" @@ -63,9 +66,13 @@ var ( func init() { utilruntime.Must(clientgoscheme.AddToScheme(scheme)) + utilruntime.Must(apiextensionsv1.AddToScheme(scheme)) utilruntime.Must(fleetnetv1alpha1.AddToScheme(scheme)) utilruntime.Must(fleetnetv1beta1.AddToScheme(scheme)) utilruntime.Must(fleetv1beta1.AddToScheme(scheme)) + utilruntime.Must(gatewayv1.Install(scheme)) + // ReferenceGrant remains v1beta1 in Gateway API v1.2.1. + utilruntime.Must(gatewayv1beta1.Install(scheme)) } func TestE2E(t *testing.T) { diff --git a/test/e2e/framework/objects.go b/test/e2e/framework/objects.go index 85ac0ace..2c17c8a3 100644 --- a/test/e2e/framework/objects.go +++ b/test/e2e/framework/objects.go @@ -4,7 +4,10 @@ import ( corev1 "k8s.io/api/core/v1" discoveryv1 "k8s.io/api/discovery/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" fleetnetv1beta1 "go.goms.io/fleet-networking/api/v1beta1" @@ -88,3 +91,140 @@ func MultiClusterService(namespace, name, svcName string) *fleetnetv1alpha1.Mult }, } } + +// GatewayClass returns a GatewayClass object. +func GatewayClass(name string, controllerName gatewayv1.GatewayController) *gatewayv1.GatewayClass { + return &gatewayv1.GatewayClass{ + ObjectMeta: metav1.ObjectMeta{Name: name}, + Spec: gatewayv1.GatewayClassSpec{ + ControllerName: controllerName, + }, + } +} + +// Gateway returns an HTTP Gateway object. +func Gateway(namespace, name, className, hostname string, annotations map[string]string) *gatewayv1.Gateway { + listenerHostname := gatewayv1.Hostname(hostname) + return &gatewayv1.Gateway{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + Annotations: annotations, + }, + Spec: gatewayv1.GatewaySpec{ + GatewayClassName: gatewayv1.ObjectName(className), + Listeners: []gatewayv1.Listener{ + { + Name: "http", + Hostname: &listenerHostname, + Port: 80, + Protocol: gatewayv1.HTTPProtocolType, + }, + }, + }, + } +} + +// InternalServiceExport returns the hub-side export used to derive a Fleet ServiceImport. +func InternalServiceExport(namespace, name, serviceNamespace, serviceName, clusterID string, port int32) *fleetnetv1alpha1.InternalServiceExport { + return &fleetnetv1alpha1.InternalServiceExport{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Spec: fleetnetv1alpha1.InternalServiceExportSpec{ + Ports: []fleetnetv1alpha1.ServicePort{ + { + Name: "http", + Protocol: corev1.ProtocolTCP, + Port: port, + }, + }, + ServiceReference: fleetnetv1alpha1.ExportedObjectReference{ + ClusterID: clusterID, + APIVersion: "v1", + Kind: "Service", + Namespace: serviceNamespace, + Name: serviceName, + ResourceVersion: "1", + Generation: 1, + UID: types.UID(name), + NamespacedName: types.NamespacedName{Namespace: serviceNamespace, Name: serviceName}.String(), + }, + Type: corev1.ServiceTypeClusterIP, + }, + } +} + +// HTTPRouteToFleetServiceImport returns an HTTPRoute with one Fleet ServiceImport backend. +func HTTPRouteToFleetServiceImport( + namespace, name, gatewayName, backendNamespace, backendName string, + port gatewayv1.PortNumber, + weight int32, +) *gatewayv1.HTTPRoute { + fleetGroup := gatewayv1.Group(fleetnetv1alpha1.GroupVersion.Group) + serviceImportKind := gatewayv1.Kind("ServiceImport") + backendRef := gatewayv1.BackendObjectReference{ + Group: &fleetGroup, + Kind: &serviceImportKind, + Name: gatewayv1.ObjectName(backendName), + Port: &port, + } + if backendNamespace != "" { + ns := gatewayv1.Namespace(backendNamespace) + backendRef.Namespace = &ns + } + + return &gatewayv1.HTTPRoute{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Spec: gatewayv1.HTTPRouteSpec{ + CommonRouteSpec: gatewayv1.CommonRouteSpec{ + ParentRefs: []gatewayv1.ParentReference{ + {Name: gatewayv1.ObjectName(gatewayName)}, + }, + }, + Rules: []gatewayv1.HTTPRouteRule{ + { + BackendRefs: []gatewayv1.HTTPBackendRef{ + { + BackendRef: gatewayv1.BackendRef{ + BackendObjectReference: backendRef, + Weight: &weight, + }, + }, + }, + }, + }, + }, + } +} + +// ReferenceGrantForFleetServiceImport permits HTTPRoutes in routeNamespace to reference one Fleet ServiceImport. +func ReferenceGrantForFleetServiceImport(namespace, name, routeNamespace, serviceImportName string) *gatewayv1beta1.ReferenceGrant { + serviceImportObjectName := gatewayv1beta1.ObjectName(serviceImportName) + return &gatewayv1beta1.ReferenceGrant{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Spec: gatewayv1beta1.ReferenceGrantSpec{ + From: []gatewayv1beta1.ReferenceGrantFrom{ + { + Group: gatewayv1beta1.Group(gatewayv1.GroupVersion.Group), + Kind: "HTTPRoute", + Namespace: gatewayv1beta1.Namespace(routeNamespace), + }, + }, + To: []gatewayv1beta1.ReferenceGrantTo{ + { + Group: gatewayv1beta1.Group(fleetnetv1alpha1.GroupVersion.Group), + Kind: "ServiceImport", + Name: &serviceImportObjectName, + }, + }, + }, + } +} diff --git a/test/e2e/gateway_api_test.go b/test/e2e/gateway_api_test.go new file mode 100644 index 00000000..89f0c9f5 --- /dev/null +++ b/test/e2e/gateway_api_test.go @@ -0,0 +1,238 @@ +//go:build e2e + +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package e2e + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/api/errors" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" + "go.goms.io/fleet-networking/pkg/annotations" + "go.goms.io/fleet-networking/test/e2e/framework" +) + +const gatewayControllerName = gatewayv1.GatewayController("networking.fleet.azure.com/afd") + +// This suite validates the live API contract only; AFD reconciliation is not implemented yet. +var _ = Describe("Gateway API contract", Ordered, func() { + var ( + backendNamespace string + gatewayClassName string + gatewayName = "afd-gateway" + ) + + createWithCleanup := func(obj client.Object) { + GinkgoHelper() + Expect(hubCluster.Client().Create(ctx, obj)).To(Succeed()) + key := client.ObjectKeyFromObject(obj) + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubCluster.Client().Delete(ctx, obj))).To(Succeed()) + Eventually(func() bool { + probe := obj.DeepCopyObject().(client.Object) + return errors.IsNotFound(hubCluster.Client().Get(ctx, key, probe)) + }, framework.PollTimeout, framework.PollInterval).Should(BeTrue(), "Failed to delete %T %s", obj, key) + }) + } + + createExportWithCleanup := func(export *fleetnetv1alpha1.InternalServiceExport) { + GinkgoHelper() + Expect(hubCluster.Client().Create(ctx, export)).To(Succeed()) + exportKey := client.ObjectKeyFromObject(export) + serviceImportKey := types.NamespacedName{ + Namespace: export.Spec.ServiceReference.Namespace, + Name: export.Spec.ServiceReference.Name, + } + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubCluster.Client().Delete(ctx, export))).To(Succeed()) + Eventually(func() bool { + probe := &fleetnetv1alpha1.InternalServiceExport{} + return errors.IsNotFound(hubCluster.Client().Get(ctx, exportKey, probe)) + }, framework.PollTimeout, framework.PollInterval).Should(BeTrue(), "Failed to delete InternalServiceExport %s", exportKey) + Eventually(func() bool { + probe := &fleetnetv1alpha1.ServiceImport{} + return errors.IsNotFound(hubCluster.Client().Get(ctx, serviceImportKey, probe)) + }, framework.PollTimeout, framework.PollInterval).Should(BeTrue(), "Failed to delete derived ServiceImport %s", serviceImportKey) + }) + } + + waitForServiceImport := func(key types.NamespacedName) *fleetnetv1alpha1.ServiceImport { + GinkgoHelper() + serviceImport := &fleetnetv1alpha1.ServiceImport{} + Eventually(func() bool { + if err := hubCluster.Client().Get(ctx, key, serviceImport); err != nil { + return false + } + return len(serviceImport.Status.Clusters) != 0 + }, framework.PollTimeout, framework.PollInterval).Should(BeTrue(), "Failed to get resolved ServiceImport %s", key) + return serviceImport + } + + annotateServiceImport := func(key types.NamespacedName, desired map[string]string) { + GinkgoHelper() + Eventually(func() error { + serviceImport := &fleetnetv1alpha1.ServiceImport{} + if err := hubCluster.Client().Get(ctx, key, serviceImport); err != nil { + return err + } + if serviceImport.Annotations == nil { + serviceImport.Annotations = make(map[string]string, len(desired)) + } + for annotation, value := range desired { + serviceImport.Annotations[annotation] = value + } + return hubCluster.Client().Update(ctx, serviceImport) + }, framework.PollTimeout, framework.PollInterval).Should(Succeed(), "Failed to annotate ServiceImport %s", key) + } + + BeforeAll(func() { + backendNamespace = framework.UniqueTestNamespace() + gatewayClassName = "afd-" + backendNamespace + createWithCleanup(framework.Namespace(backendNamespace)) + }) + + It("round-trips Gateway API resources with Fleet ServiceImport backends", func() { + gatewayClass := framework.GatewayClass(gatewayClassName, gatewayControllerName) + createWithCleanup(gatewayClass) + + gatewayAnnotations := map[string]string{ + annotations.AFDSKUAnnotation: string(annotations.SKUPremium), + } + gateway := framework.Gateway(testNamespace, gatewayName, gatewayClassName, "store.example.com", gatewayAnnotations) + createWithCleanup(gateway) + + serviceImportAnnotations := map[string]string{ + annotations.AFDOriginConnectivityAnnotation: string(annotations.ConnectivityPrivateLink), + annotations.AFDHealthProbePathAnnotation: "/healthz", + annotations.AFDOriginHostHeaderAnnotation: "store.internal.example.com", + } + sameNamespaceExport := framework.InternalServiceExport( + testNamespace, "store-same-export", testNamespace, "store-same", memberClusterNames[0], 8080, + ) + createExportWithCleanup(sameNamespaceExport) + sameNamespaceServiceImportKey := types.NamespacedName{Namespace: testNamespace, Name: "store-same"} + sameNamespaceServiceImport := waitForServiceImport(sameNamespaceServiceImportKey) + annotateServiceImport(sameNamespaceServiceImportKey, serviceImportAnnotations) + + crossNamespaceExport := framework.InternalServiceExport( + backendNamespace, "store-cross-export", backendNamespace, "store-cross", memberClusterNames[1], 8080, + ) + createExportWithCleanup(crossNamespaceExport) + crossNamespaceServiceImportKey := types.NamespacedName{Namespace: backendNamespace, Name: "store-cross"} + crossNamespaceServiceImport := waitForServiceImport(crossNamespaceServiceImportKey) + annotateServiceImport(crossNamespaceServiceImportKey, serviceImportAnnotations) + + sameNamespaceRoute := framework.HTTPRouteToFleetServiceImport( + testNamespace, "store-same", gatewayName, "", sameNamespaceServiceImport.Name, 8080, 70, + ) + createWithCleanup(sameNamespaceRoute) + + referenceGrant := framework.ReferenceGrantForFleetServiceImport( + backendNamespace, "allow-store-route", testNamespace, crossNamespaceServiceImport.Name, + ) + createWithCleanup(referenceGrant) + crossNamespaceRoute := framework.HTTPRouteToFleetServiceImport( + testNamespace, "store-cross", gatewayName, backendNamespace, crossNamespaceServiceImport.Name, 8080, 30, + ) + createWithCleanup(crossNamespaceRoute) + + By("validating the GatewayClass and Gateway contract") + var gotGatewayClass gatewayv1.GatewayClass + Expect(hubCluster.Client().Get(ctx, types.NamespacedName{Name: gatewayClassName}, &gotGatewayClass)).To(Succeed()) + Expect(gotGatewayClass.Spec.ControllerName).To(Equal(gatewayControllerName)) + + var gotGateway gatewayv1.Gateway + Expect(hubCluster.Client().Get(ctx, types.NamespacedName{Namespace: testNamespace, Name: gatewayName}, &gotGateway)).To(Succeed()) + Expect(gotGateway.Spec.GatewayClassName).To(Equal(gatewayv1.ObjectName(gatewayClassName))) + Expect(gotGateway.Annotations).To(HaveKeyWithValue(annotations.AFDSKUAnnotation, string(annotations.SKUPremium))) + Expect(gotGateway.Spec.Listeners).To(HaveLen(1)) + Expect(gotGateway.Spec.Listeners[0].Protocol).To(Equal(gatewayv1.HTTPProtocolType)) + + By("validating the same-namespace Fleet ServiceImport backend reference") + var gotSameNamespaceRoute gatewayv1.HTTPRoute + Expect(hubCluster.Client().Get( + ctx, + types.NamespacedName{Namespace: sameNamespaceRoute.Namespace, Name: sameNamespaceRoute.Name}, + &gotSameNamespaceRoute, + )).To(Succeed()) + assertFleetServiceImportBackend(gotSameNamespaceRoute, gatewayName, "", sameNamespaceServiceImport.Name, 8080, 70) + + By("validating the cross-namespace backend and ReferenceGrant") + var gotReferenceGrant gatewayv1beta1.ReferenceGrant + Expect(hubCluster.Client().Get( + ctx, + types.NamespacedName{Namespace: referenceGrant.Namespace, Name: referenceGrant.Name}, + &gotReferenceGrant, + )).To(Succeed()) + Expect(gotReferenceGrant.Spec.From).To(Equal(referenceGrant.Spec.From)) + Expect(gotReferenceGrant.Spec.To).To(Equal(referenceGrant.Spec.To)) + + var gotCrossNamespaceRoute gatewayv1.HTTPRoute + Expect(hubCluster.Client().Get( + ctx, + types.NamespacedName{Namespace: crossNamespaceRoute.Namespace, Name: crossNamespaceRoute.Name}, + &gotCrossNamespaceRoute, + )).To(Succeed()) + assertFleetServiceImportBackend(gotCrossNamespaceRoute, gatewayName, backendNamespace, crossNamespaceServiceImport.Name, 8080, 30) + + By("validating the Fleet ServiceImport API version and annotations") + var serviceImportCRD apiextensionsv1.CustomResourceDefinition + Expect(hubCluster.Client().Get( + ctx, + types.NamespacedName{Name: "serviceimports.networking.fleet.azure.com"}, + &serviceImportCRD, + )).To(Succeed()) + Expect(serviceImportCRD.Spec.Group).To(Equal(fleetnetv1alpha1.GroupVersion.Group)) + Expect(serviceImportCRD.Spec.Versions).To(ContainElement(And( + HaveField("Name", fleetnetv1alpha1.GroupVersion.Version), + HaveField("Served", true), + HaveField("Storage", true), + ))) + + var gotServiceImport fleetnetv1alpha1.ServiceImport + Expect(hubCluster.Client().Get(ctx, crossNamespaceServiceImportKey, &gotServiceImport)).To(Succeed()) + for annotation, value := range serviceImportAnnotations { + Expect(gotServiceImport.Annotations).To(HaveKeyWithValue(annotation, value)) + } + }) +}) + +func assertFleetServiceImportBackend( + route gatewayv1.HTTPRoute, + gatewayName, namespace, name string, + port gatewayv1.PortNumber, + weight int32, +) { + GinkgoHelper() + Expect(route.Spec.ParentRefs).To(HaveLen(1)) + Expect(route.Spec.ParentRefs[0].Name).To(Equal(gatewayv1.ObjectName(gatewayName))) + Expect(route.Spec.Rules).To(HaveLen(1)) + Expect(route.Spec.Rules[0].BackendRefs).To(HaveLen(1)) + + backendRef := route.Spec.Rules[0].BackendRefs[0].BackendRef + Expect(backendRef.Group).NotTo(BeNil()) + Expect(string(*backendRef.Group)).To(Equal(fleetnetv1alpha1.GroupVersion.Group)) + Expect(backendRef.Kind).NotTo(BeNil()) + Expect(string(*backendRef.Kind)).To(Equal("ServiceImport")) + Expect(string(backendRef.Name)).To(Equal(name)) + Expect(backendRef.Port).NotTo(BeNil()) + Expect(*backendRef.Port).To(Equal(port)) + Expect(backendRef.Weight).NotTo(BeNil()) + Expect(*backendRef.Weight).To(Equal(weight)) + if namespace == "" { + Expect(backendRef.Namespace).To(BeNil()) + return + } + Expect(backendRef.Namespace).NotTo(BeNil()) + Expect(string(*backendRef.Namespace)).To(Equal(namespace)) +} diff --git a/test/scripts/bootstrap.sh b/test/scripts/bootstrap.sh index f0bf67a2..0a0c6e29 100644 --- a/test/scripts/bootstrap.sh +++ b/test/scripts/bootstrap.sh @@ -317,6 +317,26 @@ fi # Helm install charts for hub cluster. kubectl config use-context $HUB_CLUSTER-admin +# Install the Gateway API standard channel CRDs at the version selected in go.mod. +GATEWAY_API_CRD_DIR="$(go list -m -f '{{.Dir}}' sigs.k8s.io/gateway-api)/config/crd/standard" +if [ ! -d "$GATEWAY_API_CRD_DIR" ]; then + echo "error: Gateway API standard CRD directory not found: $GATEWAY_API_CRD_DIR" + exit 1 +fi +GATEWAY_API_CRDS=( + gateway.networking.k8s.io_gatewayclasses.yaml + gateway.networking.k8s.io_gateways.yaml + gateway.networking.k8s.io_httproutes.yaml + gateway.networking.k8s.io_referencegrants.yaml +) +for crd in "${GATEWAY_API_CRDS[@]}"; do + kubectl apply --server-side -f "$GATEWAY_API_CRD_DIR/$crd" +done +kubectl wait --for=condition=Established --timeout=60s \ + crd/gatewayclasses.gateway.networking.k8s.io \ + crd/gateways.gateway.networking.k8s.io \ + crd/httproutes.gateway.networking.k8s.io \ + crd/referencegrants.gateway.networking.k8s.io # need to make sure the version matches the one in the go.mod # workaround mentioned in https://github.com/kubernetes-sigs/controller-runtime/issues/1191 kubectl apply -f `go env GOPATH`/pkg/mod/go.goms.io/fleet@v0.14.0/config/crd/bases/cluster.kubernetes-fleet.io_internalmemberclusters.yaml From 4da555af91c1f824d3286f1abfb2183e52cd5212 Mon Sep 17 00:00:00 2001 From: Ramkumar Chinchani Date: Mon, 31 Aug 2026 13:18:55 -0700 Subject: [PATCH 4/4] docs: clarify feature-specific e2e setup Separate baseline networking, Traffic Manager, and current Gateway API contract coverage. Document the additional wiring required for future Azure Front Door reconciliation tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-08-31-1959-e2e-readme-feature-guidance.md | 101 +++++++++++++ test/README.md | 137 +++++++++++++++--- 2 files changed, 218 insertions(+), 20 deletions(-) create mode 100644 .github/.copilot/breadcrumbs/2026-08-31-1959-e2e-readme-feature-guidance.md diff --git a/.github/.copilot/breadcrumbs/2026-08-31-1959-e2e-readme-feature-guidance.md b/.github/.copilot/breadcrumbs/2026-08-31-1959-e2e-readme-feature-guidance.md new file mode 100644 index 00000000..71fb7089 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-31-1959-e2e-readme-feature-guidance.md @@ -0,0 +1,101 @@ +# E2E README Feature Guidance + +## Requirements + +- Update `test/README.md` so local E2E setup is clear and feature-specific. +- Separate the baseline networking, Azure Traffic Manager (ATM), and Azure Front Door (AFD) + requirements and commands. +- Accurately describe which AFD behavior is testable today. +- Preserve the existing setup, test, and cleanup workflow. + +## Additional comments from user + +- The user asked whether the README can be updated separately for ATM and AFD. + +## Plan + +### Phase 1: Verify the current E2E wiring + +- [x] **Task 1.1: Review the E2E README and Makefile entrypoints.** + - Success criteria: the guide uses the repository's actual `e2e-setup`, `e2e-tests`, + `e2e-collect-logs`, and `e2e-cleanup` targets. +- [x] **Task 1.2: Review feature gates and CI coverage.** + - Success criteria: the guide accurately reflects `AZURE_NETWORK_SETTING`, + `ENABLE_TRAFFIC_MANAGER`, and the current AFD contract-test behavior. + +### Phase 2: Restructure the local developer guide + +- [x] **Task 2.1: Document shared prerequisites and lifecycle.** + - Success criteria: common Azure, tool, cluster, test, and cleanup instructions are stated once. +- [x] **Task 2.2: Add baseline networking and ATM scenarios.** + - Success criteria: each scenario provides the exact environment variables, supported network + settings, setup command, and test-selection behavior. +- [x] **Task 2.3: Add the AFD scenario and limitation.** + - Success criteria: the guide explains that Gateway API CRDs and the live Kubernetes API contract + are covered, while AFD resource reconciliation is not yet implemented or deployed by E2E. + +### Phase 3: Review the documentation + +- [x] **Task 3.1: Check commands, links, terminology, and diff.** + - Success criteria: commands match the scripts and Makefile, ATM and AFD are not conflated, and + the resulting diff contains only the README and breadcrumb. + +### Detailed checklist + +- [x] Phase 1 / Task 1.1 completed. +- [x] Phase 1 / Task 1.2 completed. +- [x] Phase 2 / Task 2.1 completed. +- [x] Phase 2 / Task 2.2 completed. +- [x] Phase 2 / Task 2.3 completed. +- [x] Phase 3 / Task 3.1 completed. + +### Overall success criteria + +- A developer can choose a baseline networking, ATM, or AFD-oriented E2E workflow without guessing + which flags apply. +- The guide does not claim that the current AFD E2E creates or validates Azure Front Door resources. +- Existing Makefile and script behavior remains unchanged. + +## Decisions + +- Keep one shared lifecycle and use scenario sections only for differing environment variables and + behavior. +- Describe the current AFD suite as a Gateway API contract scenario because the Gateway manager has + no reconcilers yet. +- Do not introduce new scripts or feature flags in this documentation-only change. + +## Implementation Details + +- `test/README.md` now has one common lifecycle with separate baseline networking, Traffic Manager, + and AFD/Gateway API scenario sections. +- The AFD section identifies the exact live API resources and backend-reference behavior exercised + today. +- The guide states that real AFD reconciliation requires controller deployment, Azure + configuration, validators, status assertions, and cleanup that do not exist yet. + +## Changes Made + +- Added this breadcrumb as the documentation and decision trail. +- Expanded prerequisites to cover the tools invoked by the E2E setup. +- Added feature-specific environment examples and behavior. +- Added focused Gateway API contract-test execution and log-collection guidance. +- Reviewed the final documentation diff and confirmed it has no whitespace errors. + +## Before/After Comparison + +- **Before:** The E2E guide presents Traffic Manager variables as if they apply to every local run + and does not explain Gateway API or AFD coverage. +- **After:** The guide separates shared setup from baseline, ATM, and current AFD contract coverage + and explicitly identifies the missing real-AFD reconciliation wiring. + +## References + +- `test/README.md`: Existing local E2E guide. +- `Makefile`: E2E setup, execution, log collection, and cleanup targets. +- `test/scripts/bootstrap.sh`: Network-setting and Traffic Manager feature setup. +- `test/e2e/e2e_test.go`: Shared E2E suite and Gateway API scheme registration. +- `test/e2e/traffic_manager_test.go`: Traffic Manager feature gate and Azure validation. +- `test/e2e/gateway_api_test.go`: Current Gateway API live-cluster contract coverage. +- `.github/workflows/e2e-tests.yml`: CI scenario matrix. +- Repository domain knowledge: no files are present under `.github/.copilot/domain_knowledge`. +- Repository specifications: no files are present under `.github/.copilot/specifications`. diff --git a/test/README.md b/test/README.md index 81c71e1b..b4a7d3e1 100644 --- a/test/README.md +++ b/test/README.md @@ -1,57 +1,154 @@ # Local Developer E2E Guide +The E2E suite provisions one hub AKS cluster and two member AKS clusters, installs the Fleet +networking components, and runs the Ginkgo tests under `test/e2e`. + ## Prerequisites -- [`go`](https://golang.org/dl) 1.18.0 or later -- [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/) -- [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) -- An [Azure](https://azure.microsoft.com/en-us/) subscription +- [Go](https://go.dev/dl/) at the version declared in `go.mod` +- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) +- [Docker](https://docs.docker.com/engine/install/) +- [Helm](https://helm.sh/docs/intro/install/) +- [kubectl](https://kubernetes.io/docs/tasks/tools/) +- [`jq`](https://jqlang.github.io/jq/download/) +- [`yq`](https://github.com/mikefarah/yq#install) +- Bash and GNU Make +- An Azure subscription in which the signed-in identity has the Owner role + +Sign in and select the subscription before starting: + +```bash +az login +export AZURE_SUBSCRIPTION_ID= +az account set --subscription "${AZURE_SUBSCRIPTION_ID}" +``` + +The setup builds and pushes test images, creates an Azure Container Registry and AKS clusters, and +assigns Azure roles. Use a test subscription rather than a production subscription. + +## Choose an E2E scenario + +All scenarios use the same setup, test, and cleanup targets. The environment variables determine +the cluster network topology and whether Traffic Manager reconciliation is enabled. + +### Baseline Fleet networking + +Use this scenario for ServiceExport, ServiceImport, MultiClusterService, and networking behavior +without Azure Traffic Manager: + +```bash +export AZURE_RESOURCE_GROUP= +export AZURE_SUBSCRIPTION_ID= +export AZURE_NETWORK_SETTING=shared-vnet +export ENABLE_TRAFFIC_MANAGER=false +``` + +`AZURE_NETWORK_SETTING` supports: -## Run E2E tests +- `shared-vnet` +- `dynamic-ip-allocation` +- `peered-vnet` +- `unsupported` -Make sure you have owner role to the e2e test subscription. +The scripts under `test/scripts` define each topology. `unsupported` verifies behavior when member +clusters do not have supported network connectivity. -### Set up e2e environment +### Azure Traffic Manager -Currently e2e resources are built on top of Azure, so provide your Azure resource setting: +Traffic Manager E2E provisions and validates real Azure Traffic Manager resources. It is supported +only with the shared-VNet topology: ```bash export AZURE_RESOURCE_GROUP= export AZURE_SUBSCRIPTION_ID= -# Available values for AZURE_NETWORK_SETTING are shared-vnet, dynamic-ip-allocation and peered-vnet, -# and detailed explanations for each network setting are provided in the scripts under folder "test/scripts". export AZURE_NETWORK_SETTING=shared-vnet export ENABLE_TRAFFIC_MANAGER=true ``` -Run Makefile Target to setup e2e environment: +The setup configures the additional AKS identities, Azure role assignments, and controller feature +flags required by `test/e2e/traffic_manager_test.go`. Tests in that file skip when +`ENABLE_TRAFFIC_MANAGER` is not `true`. + +### Azure Front Door and Gateway API + +The current AFD-oriented E2E coverage is a live Kubernetes API contract test. Setup installs the +Gateway API CRDs in the hub, and `test/e2e/gateway_api_test.go` verifies that the API server accepts +and round-trips: + +- `GatewayClass`, `Gateway`, and `HTTPRoute` +- Same-namespace Fleet `ServiceImport` backends +- Cross-namespace Fleet `ServiceImport` backends authorized by `ReferenceGrant` +- AFD annotations and the exact backend group, kind, port, and weight + +Use the baseline environment: + +```bash +export AZURE_RESOURCE_GROUP= +export AZURE_SUBSCRIPTION_ID= +export AZURE_NETWORK_SETTING=shared-vnet +export ENABLE_TRAFFIC_MANAGER=false +``` + +There is no `ENABLE_AFD` E2E flag yet. The Gateway controller manager currently has no AFD +reconcilers, and the E2E setup does not deploy it or create and validate Azure Front Door resources. +Real AFD reconciliation E2E will additionally need controller deployment, Azure identity and RBAC, +AFD configuration, Azure SDK validators, Gateway API status assertions, and Azure resource cleanup. + +## Set up the E2E environment + +After exporting the variables for one scenario, run: ```bash make e2e-setup ``` -Until now, you will have one hub cluster and two member clusters according to your setting, and you may switch between clusters using the following commands: +The default cluster names are: ```bash export HUB_CLUSTER=hub export MEMBER_CLUSTER_1=member-1 export MEMBER_CLUSTER_2=member-2 -# Use hub cluster kubeconfig context -kubectl config use-context $HUB_CLUSTER-admin -# Use member cluster $MEMBER_CLUSTER_1 kubeconfig context -kubectl config use-context $MEMBER_CLUSTER_1-admin -# Use member cluster $MEMBER_CLUSTER_2 kubeconfig context -kubectl config use-context $MEMBER_CLUSTER_2-admin ``` -Run e2e tests with the following command, and you may reuse the test environment to run e2e tests multiple times to validate your new tests or debug test issues. +Switch kubeconfig contexts with: + +```bash +kubectl config use-context "${HUB_CLUSTER}-admin" +kubectl config use-context "${MEMBER_CLUSTER_1}-admin" +kubectl config use-context "${MEMBER_CLUSTER_2}-admin" +``` + +## Run the tests + +Run the complete E2E suite: ```bash make e2e-tests ``` -Clean up e2e tests resources: +The environment can be reused for repeated test runs. To run only the current Gateway API contract +scenario: + +```bash +go test -timeout 50m -tags=e2e -v ./test/e2e \ + -args -ginkgo.v -ginkgo.focus='Gateway API contract' +``` + +## Collect logs + +Collect controller and agent logs before cleanup when investigating a failure: + +```bash +export LOG_DIR=agent-logs +make e2e-collect-logs +``` + +## Clean up + +Delete the Azure resource group created for the E2E environment: ```bash make e2e-cleanup ``` + +Cleanup requires `AZURE_RESOURCE_GROUP` to still identify the test resource group.