Make InferenceGateway an AI gateway - #425
Draft
negz wants to merge 13 commits into
Draft
Conversation
Making the fleet gateway an AI gateway needs AIGatewayRoute's streamIdleTimeout, which resets a backend that hangs before the first token and fails over to the next priority. That field arrived in Envoy AI Gateway v1.1.0, so the v0.7.0 pin blocks the rest of this branch. Envoy Gateway goes to v1.8.4 rather than v1.9.x, which AI Gateway v1.1.x isn't tested against yet, and GAIE to v1.0.2, what AI Gateway v1.1.0 builds against. The endpoint picker Modelplane deploys is llm-d's fork, so its CRDs move on llm-d's cadence rather than upstream's; that's a separate change. logRequestHeaderAttributes maps the caller header into request metadata. The gateway strips that header before a request reaches a third-party backend, so an access log reading the header would lose the caller on exactly the records that attribute provider spend. Reading metadata instead re-enables the path behind envoyproxy/ai-gateway#2600, whose failure is loud and has a fix in flight; losing the caller is silent. Signed-off-by: Nic Cope <nicc@rk0n.org>
The fleet gateway was a generic HTTP router that understood nothing about the requests it forwarded: a ModelService got a URL, a caller reached one by path having already worked out what the engine called itself, nothing authenticated or counted tokens, and the gateway was a cluster-scoped singleton on the control plane, so every prompt crossed whatever region that happened to be. These are the APIs for the gateway the design describes: one that speaks the inference protocol, resolves the model a request names, and translates the request for the endpoint that serves it. InferenceGateway stops being a singleton and names the InferenceCluster it runs on, gaining a hostname, TLS, caller-key auth, and a serviceSelector that scopes which services it serves. ModelEndpoint stops being a URL and becomes a backend description. ModelService gains a priority order, separating failover from deliberate traffic shifting, and loses its address, since a caller now names it as the model at any gateway serving it. An endpoint's origin must be a hostname, not an address: Envoy AI Gateway applies per-backend model rewriting, credentials and priority failover only when every backend in a route is addressed by hostname, and silently stops applying all three otherwise. So InferenceCluster gains spec.gateway.hostname. The example manifests and the validator that checks them against the generated models move with the APIs; the composition functions still expect the old fields and are updated next. Signed-off-by: Nic Cope <nicc@rk0n.org>
compose-inference-gateway installed Traefik, MetalLB and a megabyte of vendored Gateway API CRDs on the control plane, because nothing else on a fresh control plane provided them, and composed a single Gateway that ModelServices hung path-prefixed HTTPRoutes off. The gateway now runs on the InferenceCluster the XR names, which already runs Envoy Gateway and Envoy AI Gateway, so this function installs nothing: it composes the gateway-scoped objects onto that cluster through its ClusterProviderConfig, and the control plane stops owning a gateway or the Gateway API CRDs. It owns the Gateway and its listeners, an EnvoyProxy carrying the usage-record access log, a SecurityPolicy authenticating callers, a BackendTrafficPolicy that makes priorities mean something, and a /healthz route with its own allow-all policy. The per-service routes and backends come from compose-model-service. Composing nothing is a valid outcome, and the conditions say why. Three things are load-bearing and were only found running it: a cluster-scoped composite doesn't default its composed objects' namespace, so they need it set explicitly; ndots:5 makes the proxy stall resolving every short backend hostname through the search list, so its dnsConfig sets ndots:1; and without retry.numAttemptsPerPriority the priority failover silently never happens. Signed-off-by: Nic Cope <nicc@rk0n.org>
compose-model-endpoint parsed spec.url and composed a selectorless Service plus a hand-written EndpointSlice on the control plane, with two Traefik-specific workarounds baked in. With gateways plural, the objects a gateway needs to reach an endpoint are per-gateway, and an endpoint can't know which gateways serve it without reading the ModelServices that select it. So compose-model-service composes them and this function composes nothing. What's worth keeping is deciding whether an endpoint is usable, which belongs here rather than in every route that references it. An endpoint naming a credential Secret that doesn't exist, or one without the key it names, would otherwise be composed into each gateway's route and fail at request time. EndpointReady puts that on the endpoint, so compose-model-service can leave a broken endpoint out of a route. Whether a backend actually answers is a question only a request can settle, so the gateway's outlier detection handles that. Signed-off-by: Nic Cope <nicc@rk0n.org>
compose-model-service composed one HTTPRoute on the control plane's singleton gateway, fanning over backendRefs that each carried a URLRewrite filter, which was the only reason Modelplane ran Traefik. A ModelService now becomes an AIGatewayRoute matching the model a caller names, plus, per endpoint, the objects a gateway needs to reach it and translate for it: a Backend, an AIServiceBackend, and where the endpoint has a credential a BackendSecurityPolicy and propagated Secret. Which gateways serve a service is the gateway's choice: this reads every InferenceGateway and keeps those whose serviceSelector matches, which is what makes residency fall out of labels rather than needing a feature. The objects are named per service, so two services selecting one endpoint compose their own copies rather than fighting over one. Three details are load-bearing and each has a test that fails when it's undone: every backend is addressed by hostname, or Envoy stops applying the model rewrite, host rewrite and credential while still passing traffic; llmRequestCosts is what makes a streamed response report tokens; and a caller header is stripped from an endpoint Modelplane doesn't operate, ours kept. Weights compete only within a priority tier, and the schema requires at least 1, since a weight of 0 drops a backend from the load assignment rather than parking it. Signed-off-by: Nic Cope <nicc@rk0n.org>
…rves A composed ModelEndpoint pointed at its cluster gateway's IP and carried a rewritePath for the control-plane gateway. Neither survives the reshape. An address doesn't work: Envoy AI Gateway emits a STRICT_DNS cluster for a route whose backends are all hostnames, which carries the per-priority failover, and an EDS cluster for an address, where model rewriting, host rewriting and the credential all stop applying while traffic keeps flowing. So an endpoint's origin is its cluster gateway's hostname, and a cluster is schedulable only once spec.gateway.hostname is set. Publishing DNS for a cluster gateway becomes a prerequisite for placing models on it. A caller names a ModelService but an engine only answers to the name it was started with, so Modelplane injects MODELPLANE_SERVED_MODEL_NAME into every engine container and expects args to reference it. The composed endpoint carries the same value as spec.model, so a service rewrites one name for a whole deployment. A user override of the same name is dropped rather than honoured, since it would let the engine answer to a name nothing routes to. The convention is documented on the args field; a CEL rule enforcing it exceeds the schema's cost budget. This also finishes endpoint residency. An InferenceCluster gains spec.placement.metadata.labels, stamped onto every ModelReplica and ModelEndpoint composed there. A ModelService selects endpoints by label, so without it a region-scoped service couldn't select its own replicas, and nobody could label them by hand because Modelplane owns them. Cluster labels beat the deployment's template on a collision, since the cluster is the authority on where it is. Signed-off-by: Nic Cope <nicc@rk0n.org>
The e2e asserted a 200 from a path-addressed ModelService on the control plane, which no longer exists. It now drives the gateway the way a caller does and asserts the mechanisms the design turns on. The InferenceGateway runs on the workload cluster beside the models, so the control plane installs no gateway; two clusters are kept as the realistic topology. The mock engine now behaves like a real one: it takes --served-model-name=$(MODELPLANE_SERVED_MODEL_NAME) and 404s anything else, so a 200 through the gateway proves the caller's ModelService name was rewritten to the deployment's. It also reads a chunked request body, since the gateway's ext_proc rewrites the body and Envoy forwards it without a content-length. A wired-in DNS step stands in for the DNS a platform publishes per cluster gateway, which an InferenceGateway needs to address a cluster by name. Beyond a 200 it asserts that an unclaimed model routes nowhere, that /v1/models lists the service, that the Anthropic Messages API is translated, and that the gateway emits a usage record with the engine's token counts. Verify gates on the ModelService reporting RoutingReady, so it doesn't curl while the engine is still rolling out. Signed-off-by: Nic Cope <nicc@rk0n.org>
The hop from a fleet gateway to a cluster gateway crossed whatever network separates two clusters as plain HTTP. The design calls for TLS in both directions, with each cluster gateway holding a certificate, none of it optional. Where the CA lives is a decision the design leaves open and it shapes everything. A composition function can't generate key material, and cert-manager exists only on InferenceClusters, so a single shared CA would mean one private key transiting every cluster. This diverges from "a CA", singular: each cluster runs a CA for its gateway's serving certificate and each gateway one for the client certificate it presents, both issued by cert-manager on the cluster they belong to, so no private key leaves it. A gateway publishes its client CA, an InferenceCluster collects every gateway's into a ClientTrafficPolicy on its HTTPS listener, and compose-model-service points each Backend at the target cluster's CA with the gateway's client certificate attached. The listener fails closed rather than open. The HTTPS listener replaces the HTTP one rather than joining it, so port 80 can't serve the engines in the clear; it is served only once there is a client CA to enforce with; and a cluster publishes its hostname, which is what makes it schedulable, only once it has also published its CA, so no endpoint exists before the hop it describes can be mutually authenticated. Verified on a live cluster: a client presenting no certificate is refused during the handshake. Signed-off-by: Nic Cope <nicc@rk0n.org>
A cluster gateway's serving CA and a fleet gateway's client CA are issued by cert-manager on the cluster they belong to, and the certificate has to reach the control plane so the other end can trust it. Reading it from cert-manager's Secret through provider-kubernetes was insecure: provider-kubernetes copies the whole observed object into status with --sanitize-secrets defaulting off, so every CA private key was readable by anyone who could get objects. That removed the whole basis for the per-cluster CA design, whose point was that no key leaves its cluster. trust-manager publishes each CA certificate into a ConfigMap, which the control plane observes instead. A Bundle copies one named key out of a Secret in-cluster and rejects any PEM that isn't a CERTIFICATE, so the private key never crosses a trust boundary and can't be made to by misconfiguration. With nothing reading Secret data back, --sanitize-secrets is now safe to turn on as well. Three constraints decide how trust-manager installs. Its own webhook Certificate races cert-manager's admission, so it waits for cert-manager to report Ready; its default public-CA trust package blocks the pod on an image pull no Bundle here needs, so it's disabled; and it's gated on the self-signed Issuer, composed on every cluster, rather than a CA that fleet-only clusters never compose. Both charts keep their CRDs on uninstall, since provider-kubernetes can't observe an object whose CRD has gone. The e2e now asserts an uncertified caller is refused mid-handshake and that nothing answers on port 80, so a silently disabled mTLS policy fails it. Signed-off-by: Nic Cope <nicc@rk0n.org>
The manifests under docs/manifests were reshaped with the APIs, but the prose around them wasn't, so the docs contradicted their own examples. Every copy-pasteable request read ModelService.status.address, which no longer exists, and built a URL from a namespace and service path, which isn't how the gateway routes: a caller names the model and the gateway rewrites it. The request examples now read the base URL from the gateway that serves the service and name the model. The platform gateway guide describes an InferenceGateway that runs on a cluster and scopes itself with serviceSelector, rather than a Traefik-and-MetalLB singleton, and getting started's gateway section moved after registering the cluster, since a gateway now needs one to run on. The e2e README explains the two-cluster split as the shape Modelplane is for rather than a CRD race. Also drops control-plane RBAC for resources that no longer exist or now land on a workload cluster. Signed-off-by: Nic Cope <nicc@rk0n.org>
A fleet gateway addresses a cluster gateway by name, because Envoy AI Gateway only applies per-backend model rewriting, credentials and failover when a route's backends are hostnames. The branch met that by making a platform publish a DNS name per InferenceCluster in spec.gateway.hostname. The name only has to resolve where the fleet gateway's Envoy runs, and Modelplane controls that cluster, so the burden is unnecessary. This commit drops spec.gateway.hostname and derives the name from the cluster's own, which compose-inference-gateway resolves by composing a Service per cluster gateway on each cluster that runs a fleet gateway. An IP address is served by a headless Service, a load balancer's own DNS name by an ExternalName, chosen by parsing the address rather than by cloud. The derived name is a full Service FQDN, longer than a name a person would have written, so the cluster gateway's CA certificate commonName is bounded to the 64-byte X.509 limit. Dropping the field also renamed the generated status model from GatewayModel to Gateway, which compose-model-deployment tracks. Signed-off-by: Nic Cope <nicc@rk0n.org>
A fleet gateway's CA and client certificate both put the InferenceGateway's name in their commonName, and that name is a cluster-scoped resource name with no short bound, so a long one produces a commonName cert-manager rejects for exceeding 64 bytes. Both are now truncated to fit. The commonName is cosmetic here: a cluster gateway trusts the CA by its certificate and identifies a caller as a fleet gateway by that CA, not by either name. Signed-off-by: Nic Cope <nicc@rk0n.org>
The e2e stood in for per-cluster DNS by writing a CoreDNS hosts entry for a hardcoded name and pointing the mTLS verify curls at it. Modelplane now derives the name and composes the Service that resolves it, so the hosts entry is gone and the curls read the derived name from the InferenceCluster's status instead. The InferenceCluster manifest no longer sets a gateway hostname. Signed-off-by: Nic Cope <nicc@rk0n.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Prototype, not for merge. It reshapes
InferenceGatewayinto an AI gateway.Today the fleet gateway, the one on the control plane, is a Traefik HTTP router that understands nothing about the requests it forwards. A caller reaches a
ModelServiceby a path prefix, nothing authenticates, and the hop to each cluster gateway is plain HTTP across the public internet. This PR makes it an Envoy AI Gateway, the same one everyInferenceClusteralready runs at its edge. It authenticates callers and resolves theModelServicenamed in a request's body. For the backend it picks, it rewrites the model name, credential and path. Weight splits traffic between backends at one priority, and priority fails it over to the next when they go unhealthy. Every request meters a token count per caller, streams included.InferenceGatewaystops being a singleton and names the cluster it runs on, so a Modelplane can run several, one per region for residency or two in a region for availability:A
ModelEndpointis no longer a URL. It now names the backend'soriginand the API it speaks, plus the model name it knows and the credential to present.ModelServicegainspriorityalongsideweight. A per-cluster PKI now authenticates the hop from a fleet gateway to a cluster gateway in both directions. cert-manager issues it and trust-manager distributes it.I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.