Skip to content

Make InferenceGateway an AI gateway #425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
negz wants to merge 13 commits into
base: main
Choose a base branch
from
Draft

Make InferenceGateway an AI gateway #425

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
28b32ba
Update the serving stack's gateway components to current releases
negz Sep 1, 2026
f2230db
Reshape the gateway APIs around an AI gateway front door
negz Sep 1, 2026
12a0f26
Run the fleet gateway on an InferenceCluster as an AI gateway
negz Sep 1, 2026
241e291
Make a ModelEndpoint report whether it can carry traffic
negz Sep 1, 2026
02c575e
Compose a ModelService's route and backends on every gateway serving it
negz Sep 1, 2026
020c6aa
Address a replica's cluster gateway by name, and name the model it se…
negz Sep 1, 2026
6794289
Exercise the AI gateway path end to end in the local e2e
negz Sep 2, 2026
3f329f8
Authenticate the hop between a fleet gateway and a cluster gateway
negz Sep 2, 2026
676e7c7
Distribute the gateway CAs with trust-manager, not by reading Secrets
negz Sep 2, 2026
2613e48
Update the prose to the gateway the code now composes
negz Sep 2, 2026
15c2af7
Derive the cluster gateway name instead of requiring DNS per cluster
negz Sep 3, 2026
c91b0fd
Bound the fleet gateway certificate commonNames to the X.509 limit
negz Sep 3, 2026
6c88c3e
Resolve the cluster gateway name in the e2e via the composed Service
negz Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 55 additions & 2 deletions apis/inferenceclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,38 @@ spec:
enum:
- Standard
- Dynamo
placement:
type: object
description: >-
Facts about where this cluster is, projected onto everything
Modelplane composes here.
properties:
metadata:
type: object
description: Metadata to project.
properties:
labels:
type: object
description: >-
Labels stamped onto every ModelReplica and
ModelEndpoint composed on this cluster, so a fact about
the cluster is declared once here rather than repeated
on each of them.

This is how a self-hosted endpoint gets its region: a
ModelService selects endpoints by label, so a service
scoped to a region selects only the endpoints in it.
These are your labels, under your own prefix.
Modelplane carries and matches them, and never
interprets them, so "eu" means no more to it than
"prod".
additionalProperties:
type: string
maxLength: 63
maxProperties: 16
x-kubernetes-validations:
- rule: "self.all(k, !k.startsWith('modelplane.ai/'))"
message: spec.placement.metadata.labels must not use the reserved modelplane.ai/ prefix.
taints:
type: array
description: >-
Expand Down Expand Up @@ -499,8 +531,29 @@ spec:
address:
type: string
description: >-
External IP of the inference gateway on the remote cluster.
Used by ModelDeployment for unified endpoint routing.
External address of the inference gateway on the remote
cluster. Modelplane resolves status.gateway.hostname to
this itself, on each InferenceGateway's cluster, so a
platform publishes no DNS for it.
hostname:
type: string
description: >-
The internal name an InferenceGateway addresses this
cluster's gateway by, derived by Modelplane and resolved to
status.gateway.address on each gateway's cluster. Published
once the gateway has an address and traffic to it is
mutually authenticated. ModelDeployment composes a
ModelEndpoint origin from it, and withholds the endpoint
while it's unset.
caCertificate:
type: string
maxLength: 16384
description: >-
PEM certificate of the CA that signed this gateway's
serving certificate. An InferenceGateway validates against
it, so it reaches the cluster it meant to rather than
whatever else answers on that address. Written once
cert-manager on the cluster has issued.
cache:
type: object
description: >-
Expand Down
205 changes: 169 additions & 36 deletions apis/inferencegateways/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,66 +15,199 @@ spec:
served: true
referenceable: true
additionalPrinterColumns:
- name: GATEWAY
- name: CLUSTER
type: string
jsonPath: .spec.backend
jsonPath: .spec.clusterName
- name: HOSTNAME
type: string
jsonPath: .spec.hostname
- name: ADDRESS
type: string
jsonPath: .status.address
schema:
openAPIV3Schema:
description: >-
An InferenceGateway is the front door for inference requests: the only
address a caller sees. It speaks the OpenAI and Anthropic APIs,
authenticates callers, resolves the model a request names to a
ModelService, and forwards to whichever of that service's endpoints
should serve it, translating the request for the backend that won.

A Modelplane can run several, each on an InferenceCluster of its own.
Run one per region to keep a caller's traffic in its jurisdiction, or
two in a region to survive losing a cluster. Modelplane runs no global
load balancer: distributing callers across gateways is yours to
configure, whether by geo DNS, an anycast address, or an edge of your
own with these gateways as origins.
type: object
required: [spec]
properties:
spec:
type: object
required: [backend]
required: [clusterName]
x-kubernetes-validations:
- rule: "self.backend != 'Traefik' || has(self.traefik)"
message: spec.traefik is required when spec.backend is Traefik.
# Serving HTTPS needs somewhere to get a certificate, and a
# certificate is only meaningful for a name.
- rule: "!has(self.tls) || has(self.hostname)"
message: spec.hostname is required when spec.tls is set.
properties:
backend:
clusterName:
type: string
description: >-
The InferenceCluster this gateway runs on, which decides its
region and its address. A gateway doesn't move: unlike a
ModelDeployment, whose replicas re-place when their cluster
goes away, a gateway stays where it was put. Availability
comes from running more of them, because failing over would
change the address callers use and could move traffic out of
the jurisdiction the gateway exists to hold.

The cluster needs no GPU pools. A cluster with none is a
gateway and nothing else, which is what a region with callers
but no accelerators wants. A cluster that serves models can
host a gateway too, and does so at most once.
minLength: 1
maxLength: 253
hostname:
type: string
description: Gateway implementation.
enum: [Traefik]
traefik:
type: object
description: >-
Traefik Proxy configuration. Required when backend is
Traefik.
required: [version]
The name this gateway answers on. Point it at
status.address once the gateway has one.

Omit it and the gateway answers on its address alone, over
plain HTTP. That is the getting-started shape, and also the
shape for anyone terminating TLS on an edge of their own in
front of the gateway.
minLength: 1
maxLength: 253
x-kubernetes-validations:
- rule: "!has(self.loadBalancer) || self.loadBalancer != 'MetalLB' || has(self.metallb)"
message: spec.traefik.metallb is required when spec.traefik.loadBalancer is MetalLB.
- rule: "!self.contains('/') && !self.contains(':')"
message: spec.hostname must be a bare DNS name, with no scheme, port or path.
tls:
type: object
description: >-
Serves callers over HTTPS. Without it the caller's hop is
unencrypted, so anything reachable from an untrusted network
wants this or an edge that terminates TLS in front.
required: [certificateRefs]
properties:
version:
type: string
description: Traefik Helm chart version.
loadBalancer:
type: string
certificateRefs:
type: array
description: >-
Load balancer implementation for the gateway Service.
Omit for cloud environments where a native LB
controller is available.
enum: [MetalLB]
metallb:
Secrets holding the gateway's certificate, of type
kubernetes.io/tls, in the same namespace as this
Modelplane's other gateway Secrets. Modelplane copies them
to the gateway's cluster.
minItems: 1
maxItems: 8
x-kubernetes-list-type: map
x-kubernetes-list-map-keys: [name]
items:
type: object
required: [name]
properties:
name:
type: string
minLength: 1
maxLength: 253
auth:
type: object
description: >-
Authenticates callers against keys this gateway holds. Omit it
and the gateway authenticates nobody, so anything that can
reach the address can invoke any ModelService it serves. That
is only appropriate behind something that has already
established who is calling.

Modelplane authenticates callers; it does not authorize them.
Every accepted key can reach every ModelService this gateway
serves, and /v1/models lists them all regardless of key. To
narrow what a key can reach, narrow the gateway with
serviceSelector or run a separate gateway with its own keys.
required: [secretSelector]
properties:
secretSelector:
type: object
description: >-
MetalLB configuration. Required when loadBalancer is
MetalLB. Use for kind or bare-metal clusters.
required: [addressPool]
Selects Secrets holding caller API keys. Each key in a
selected Secret is one caller: the entry's name is the
caller's identity and its value is the key. So adding a
caller means writing a Secret, not editing this gateway.

The gateway stamps the resolved identity onto every
request and every usage record, and never forwards the
caller's key. Ranking one caller above another is not
Modelplane's decision to make, so it publishes the
identity and leaves acting on it to whatever does decide.
required: [matchLabels]
properties:
addressPool:
type: string
description: >-
IP address range for the MetalLB pool
(e.g. "172.18.255.200-172.18.255.250").
Must be within the cluster's network CIDR.
matchLabels:
type: object
additionalProperties:
type: string
maxLength: 63
minProperties: 1
maxProperties: 16
serviceSelector:
type: object
description: >-
Selects the ModelServices this gateway serves, by their
labels. Absent, it serves every one.

This is how a gateway is scoped: to a region, so an EU service
is only reachable through EU gateways; to your public services
on an internet-facing front door; or to a named set on a
dedicated gateway. These are your labels, under your own
prefix. Modelplane matches them and never interprets them, so
a region means no more to it than any other label.
required: [matchLabels]
properties:
matchLabels:
type: object
additionalProperties:
type: string
maxLength: 63
minProperties: 1
maxProperties: 16
status:
type: object
properties:
address:
type: string
description: >-
External address of the control plane gateway.
Backend-agnostic — works for any routing implementation.
The address this gateway answers on, and what spec.hostname
should point at. It is also the target to health check, at
/healthz, to decide whether this gateway is in rotation.

/healthz answers 200 whenever this gateway's proxy is running
and serving. It says nothing about whether any ModelService is
reachable through it, so a gateway with no healthy backend
stays in rotation and answers requests with a 503. Read each
ModelService's RoutingReady for that.
clientCACertificate:
type: string
maxLength: 16384
description: >-
PEM certificate of the CA that signs this gateway's client
certificate. Every InferenceCluster accepts client
certificates from it, which is how this gateway proves itself
to a cluster gateway and how anything else is refused.

One CA per gateway rather than one per Modelplane, so that no
private key has to be distributed: each is generated on the
cluster that uses it and only its certificate travels. Note
that every cluster gateway trusts every fleet gateway's CA and
checks the signing CA rather than the subject, so this bounds
where the keys live, not what one of them can reach.
endpoints:
type: object
description: The paths this gateway serves.
properties:
openAI:
type: string
description: >-
Base URL for the OpenAI API. A caller sets its SDK's
base_url to this and names a ModelService as the model.
anthropic:
type: string
description: Base URL for Anthropic's Messages API.
16 changes: 15 additions & 1 deletion apis/modeldeployments/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,7 @@ spec:
semver() helpers, e.g.
device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("141Gi")) >= 0.
minLength: 1
maxLength: 10240
maxLength: 8192
# This object has no schema default on purpose: the
# apiserver applies defaults before CEL validation, so
# defaulting it would inject it onto Standalone and
Expand Down Expand Up @@ -500,6 +500,20 @@ spec:
serving engine. Includes the model
identifier (e.g. --model=...) and any
parallelism flags.

Pass --served-model-name
$(MODELPLANE_SERVED_MODEL_NAME), the
variable Modelplane injects, so the
engine answers to the name a gateway
routes to. A caller names a
ModelService and the gateway rewrites
the request's model to the
deployment's, so an engine started
under a literal name returns 404 for
every request. Nothing enforces this:
a CEL rule requiring the reference
exceeds the schema's rule cost budget
however tightly args is bounded.
items:
type: string
env:
Expand Down
Loading
Loading