Design: generating the serving stack at build time - #422
Conversation
…nerate poc Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
haarchri
changed the title
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
| - **No overrides.** No version field, no values override, no way to add, | ||
| remove or substitute a component on one cluster. A platform team that needs | ||
| a different stack takes the break-glass `Composition` and owns the result. | ||
| - **No registry override.** Every cluster pulls charts and images from the | ||
| upstream repositories and registries the generated lists name. A cluster | ||
| that must pull from a private mirror, has no field for that either. |
There was a problem hiding this comment.
Coming from mostly insurance/banking companies this part would be a real struggle to work around for most platform teams I've worked with. We've usually had a requirement that all helm charts/oci artifacts needs to come from a internal registry mirror and be signed and cve scanned by us, which this effectively blocks. Also usually helm charts don't come with all the security requirements/settings enabled by default like strict non-root security contexts, PDBs etc. and lack of overrides would force us to build a break-glass Composition even if the only goal would be to set the pdb.enabled value or update the security contexts. Platforms in highly-regulated environments are also usually network-restricted which means that without ability to override the image registry, we would not be able to run the modelplane serving stack at all without building the break-glass Composition.
In my personal opinion it's not a reasonable requirement to ask users to build their own compositions just to ensure the correct security posture of the serving stack if they could easily do it just by tweaking a few values
| - **Upgrades ride releases.** Updating Modelplane updates the stack on every | ||
| cluster its control plane manages, at once, with no way to stage it. A | ||
| cluster cannot hold a component back, and a component cannot move without a | ||
| release, a patch release for a fix, a minor for everything else. The one | ||
| exception is the GPU driver on clouds whose node image supplies it: that | ||
| version follows the image, so it can move as nodes and pools cycle, with no | ||
| release involved. |
There was a problem hiding this comment.
In regulated environments that will mean a Modelplane deployment per environment as usually critical component updates (such as serving stack) need to be tested in dev/test environments before they can be officially promoted to the production, this should be explicitly documented as it's a driving cost and effort factor. With serving stacks being disconnected from Modelplane version we could gradually rollout new versions of the serving stack across the environments using only one Modelplane instance
| - **No overrides.** No version field, no values override, no way to add, | ||
| remove or substitute a component on one cluster. A platform team that needs | ||
| a different stack takes the break-glass `Composition` and owns the result. |
There was a problem hiding this comment.
This part also has a side-effect on BYO clusters in regulated environments. They may already come pre-provisioned with all the required compliance components such as cert-manager, prometheus stack connected to central observability system (e.g. Grafana cloud) and the default serving stack would try to pull in cert-manager and prometheus-stack with default values and lead to CRD ownership conflict, again forcing BYO cluster users to use break-glass Composition and lose the benefit of using the upstream
| Modelplane API: a mutating admission webhook can set the composition reference | ||
| on the `ServingStack` as Modelplane composes it, or a Crossplane `ImageConfig` | ||
| can rewrite the `compose-serving-stack` function image to one the team owns, so | ||
| the stock `Composition` runs their function. Both are advanced mechanisms, and | ||
| deliberately so, this is an escape hatch to be reached for knowingly, not a | ||
| feature to be configured. |
There was a problem hiding this comment.
compose-serving-stack function should probably be reusable as a python library if that's the case so that end-users can import it and only do some minor changes instead of having to write their own full function from scratch if they only want to change the registry url or security contexts
| - **No GPU driver setting.** Which kernel driver a GPU node runs, and whether | ||
| the node image or the GPU Operator provides it, is a generated value per | ||
| cloud. No API field reads or writes it, and changing it means a Modelplane | ||
| release. One driver version per cluster, whatever its pools run. |
There was a problem hiding this comment.
This turns a CUDA-major or GPU-generation boundary into "day of the next Modelplane release plus a full fleet upgrade" event, which may block a day 0 support for new models if they require a vLLM (or similar) version, which only runs on a new CUDA-major/GPU-gen
| Everything is fixed at build time. Every version, every values block, every | ||
| component membership decision is resolved where a human reviews a diff, and the | ||
| reconcile path only renders. This is the one commitment the design treats as | ||
| settled. Everything downstream of it, the intermediate format, the | ||
| one-driver-per-cluster choice, is a consequence that can be revisited without | ||
| touching it. |
There was a problem hiding this comment.
This blocks day-0 support for new hardware, which would require an AICR version update, which has to come together with Modelplane update and a full serving stack update rolled-out across a full fleet
| `v0.1.0-alpha.12-rc2`. Neither an alpha API nor a pre-release chart promises | ||
| compatibility across versions. | ||
|
|
||
| The coupling is growing. Grove and ModelExpress arrived with the Dynamo serving |
There was a problem hiding this comment.
NVIDIA's own installation guide says Grove must be upgraded in lockstep with Dynamo while Grove APIs are unstable. A model whose day-0 support lands in a new Dynamo release therefore implies a Grove bump, a Modelplane release, and a fleet-wide upgrade. Which again means issues with no-effort day-0 model support, which we've stated in a blog post for Nemotron "without a line of new Modelplane code, because day-zero model support is built into the design"
|
General opinion |
3 participants
Description of your changes
Towards #279.
This adds a design proposing to generate the per-cloud component lists at build time instead of maintaining them by hand. Generated and hand-written sources feed the same shape: NVIDIA AICR recipes cover the parts NVIDIA maintains, and Modelplane's own components resolve alongside them for everything else, including clouds and accelerators AICR doesn't cover. A release then installs one tested stack per cloud, the same on every cluster, with nothing to override. spec.versions, spec.standard, spec.dynamo and nvidiaDriverRoot go away, replaced by a single field naming the cloud.
A prototype lives beside the document as evidence the pipeline works: generate.py resolves recipes into the module shape compose-serving-stack iterates, and stacks/ carries the hand-written halves (Nebius, Vultr, AMD).
Fixes #
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.