From 4feb19b6bd252aac28764ae95578258e3bd98a89 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 21 Jul 2026 09:54:57 -0700 Subject: [PATCH 01/11] Add design for decomposing ModelCache into ModelCacheHydration ModelCache runs every matched cluster's hydration lifecycle inline in one fan-out function, and its footprint (clusterSelector) can silently diverge from where replicas are scheduled, so a replica can land on a cluster the cache never staged to and fail to mount at runtime. This design proposes a per-cluster ModelCacheHydration child, mirroring ModelReplica, and makes the cache footprint follow placement. It covers issues #210 and #186, with architecture and lifecycle diagrams. Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 295 +++++++++++++++++++++++++++++++++ 1 file changed, 295 insertions(+) create mode 100644 design/modelcache-hydration.md diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md new file mode 100644 index 000000000..0420d74a8 --- /dev/null +++ b/design/modelcache-hydration.md @@ -0,0 +1,295 @@ +# ModelCacheHydration + +**Status:** Draft +**Date:** July 2026 +**Author:** Dennis Ramdass + +This document proposes `ModelCacheHydration`, a per-cluster child of `ModelCache`, +and uses it to make a cache's footprint follow where replicas are scheduled. It +builds on [modelcache.md](./modelcache.md) and [design.md](./design.md), and +addresses [#210](https://github.com/modelplaneai/modelplane/issues/210) (decompose +`ModelCache`) and [#186](https://github.com/modelplaneai/modelplane/issues/186) +(footprint diverges from placement) together. + +## Summary + +`ModelCache` today does two jobs in one function: it fans a cache out to matched +clusters, and it runs each cluster's hydration lifecycle inline (the PVC, the +hydration `Job`, the token `Secret`, a four-phase machine, and the +drop-Job-after-Ready cleanup), threading a `cluster_name` through every method. +It's also the only fan-out that doesn't follow the `ModelDeployment` → +`ModelReplica` pattern. + +Two changes: + +1. **Split out `ModelCacheHydration`**, a per-cluster child that owns one + cluster's hydration. Pinned by `spec.clusterName`, it resolves its + `InferenceCluster` and auth `Secret`, composes the PVC, token `Secret`, and + `Job`, runs the phase machine, and drops the `Job`/`Secret` once Ready. +2. **Derive the footprint from placement.** `ModelCache` stages onto the + clusters where its referencing replicas are placed, not onto a hand-maintained + `clusterSelector`. A selector, if set, is an opt-in pre-warm set, not the whole + footprint. + +A cache with no selector stages exactly where its consumers run: + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: ModelCache +metadata: + name: kimi-k2 + namespace: ml-team +spec: + source: HuggingFace + huggingFace: + repo: moonshotai/Kimi-K2-Instruct + authSecret: + name: hf-token + sizeGiB: 1500 + # No clusterSelector: footprint follows the ModelDeployments that + # reference this cache. Set one only to pre-warm ahead of placement. +``` + +The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. +`ModelCacheHydration` is composed, never authored, the same as `ModelReplica`. + +## Architecture + +Both fan-outs are per-cluster. The new link is `ModelReplica → ModelCache` (bold): +the cache's footprint is the set of clusters its referencing replicas are placed +on. + +```mermaid +flowchart TD + subgraph ml["ML team creates"] + MD["ModelDeployment"] + MC["ModelCache"] + end + subgraph cp["Composed on the control cluster"] + MR["ModelReplica\n(one per placed cluster)"] + MCH["ModelCacheHydration\n(one per cluster in footprint)"] + end + subgraph wc["Workload cluster"] + WL["serving workload\n(Deployment / LeaderWorkerSet)"] + JOB["hydration Job"] + SEC["token Secret"] + PVC["PVC (weights)"] + end + MD -->|schedules| MR + MC -->|"fan out per cluster"| MCH + MD -. modelCacheRef .-> MC + MR == "placement drives footprint" ==> MC + MR --> WL + MCH --> JOB + MCH --> SEC + MCH --> PVC + JOB -->|writes| PVC + WL -->|"mounts /mnt/models"| PVC + classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; + class MCH new +``` + +## The per-cluster child + +`ModelCacheHydration` is a namespaced composite pinned to one cluster, the cache +analogue of `ModelReplica`. Its spec carries what one cluster's hydration needs +and nothing about fan-out: + +- **`clusterName`**, the cluster it stages onto. Pinned at creation; the parent + re-places only if the cluster disappears. +- **`huggingFace`**, the source, copied down verbatim (`repo`, `revision`, + `sizeGiB`). +- **`authSecret`**, the reference only (`name`, `key`, the cache's `namespace`), + never the token value. The child resolves it and propagates it to the workload + cluster itself, so a credential never lives in a CR spec. Same trust boundary as + today. +- **`cacheName`**, the parent `ModelCache` name, so the child reproduces the + stable PVC/Job/Secret names the serving side mounts by. + +Its status carries a `phase` (Pending/Hydrating/Ready/Failed), so the parent reads +`status.clusters[].phase` straight from the child. This is richer than +`ModelReplica`'s conditions-only status, to preserve the per-cluster phase +`ModelCache` shows today. + +The child is the current per-cluster body of `compose-model-cache` lifted out +whole: the PVC/Job/Secret manifests, the completion-marker skip, the +`_JOB_MANAGEMENT` cleanup, and the Ready-latch (now read from its own prior +`status.phase`). The hydration mechanism is unchanged. It just lives in one place. + +### Naming continuity + +The child keeps the names the monolith produced: PVC +`child_name("modelcache", , )`, Job `…,"hydrate"`, Secret `…,"auth"`. +Two things depend on this: + +1. The serving side mounts the PVC by that name (`base.cache_pvc_name` in + `compose-model-replica` is the same function), so the mount contract is + untouched. +2. On upgrade the child adopts the existing, already-Bound PVC instead of + provisioning a new one and re-downloading the weights. + +## Placement-driven footprint + +The problem (#186): a cache's `clusterSelector` and a deployment's placement are +set independently. The scheduler places a replica on whichever matching cluster +has capacity; if the cache didn't stage there, the PVC is missing and the pod +fails to mount at runtime, with nothing visible at apply time. Staging everywhere +is the only safe workaround, and it wastes storage and pushes the download token +and private weights onto clusters running none of the team's deployments. + +Instead, derive the footprint from placement. Each reconcile, `ModelCache`: + +1. requires every `ModelReplica` in the fleet (the scheduler already does this for + capacity accounting); +2. keeps those that reference it (`spec.modelCacheRef.name` and namespace); +3. takes the clusters they're placed on. That set is the footprint, one + `ModelCacheHydration` each. + +A replica is then never placed on a cluster the model isn't staged to, and +credentials and weights reach only the clusters that run the model. + +`clusterSelector` stays, as an opt-in pre-warm set unioned with the placed set. It +covers the one thing pure placement can't do: warm a tier before any deployment +exists. The default (no selector) follows placement. + +### Reference counting and reclaim + +The placed set is the reference count: + +- Two deployments sharing the cache on cluster X contribute one entry; either one + tearing down leaves the other's, so X's hydration stays. +- When the last referencing replica leaves X (and X isn't pre-warmed), X drops + out, the parent stops composing that child, and the child and its PVC are + reclaimed. + +No hydration is retracted out from under a live replica, and an unreferenced cache +reclaims itself per cluster with no manual cleanup. + +### Hydrating before ready + +A replica can be scheduled onto a cluster new to the model before its PVC exists. +Rather than fail the mount, the replica gates readiness on its cluster's +`ModelCacheHydration`: it reports `Hydrating` until the PVC is Bound, then +proceeds. The first replica on a new cluster pays the download once, as a visible +state rather than a mount error. + +### Lifecycle + +Placement adds a cluster to the footprint. The last replica leaving removes it. + +```mermaid +flowchart TD + A["Replica scheduled onto cluster C"] --> B["ModelCache adds C to the placed set"] + B --> C["Stamp ModelCacheHydration for (cache, C)"] + C --> D["Child composes PVC + hydration Job\nphase: Hydrating"] + D --> E{"PVC Bound and\nJob complete?"} + E -- no --> F["Replica gates on the hydration\nreports Hydrating, holds"] + F --> E + E -- yes --> G["Child phase: Ready"] + G --> H["Replica mounts /mnt/models, becomes Ready"] + H --> I{"Last referencing\nreplica leaves C?"} + I -- no --> H + I -- yes --> J["C drops from the placed set\nchild + PVC reclaimed"] +``` + +## What the parent composes + +`compose-model-cache` becomes fan-out plus roll-up: + +- **Resolve** the referencing `ModelReplica`s (the placed set) and, for pre-warm, + the `clusterSelector`. +- **Stamp** one child per cluster in `placed ∪ preWarm`, named + `child_name("modelcache", ns, cache-name, cluster)`, copying down `huggingFace`, + `clusterName`, the `authSecret` reference, and `cacheName`. +- **Roll up** each child's `status.phase`/`Ready` into `status.clusters[]`, + `status.summary`, and the `ClustersMatched`/`ArtifactReady` conditions, marking a + child `READY_TRUE` only when observed Ready. +- **Shed** the per-cluster machinery to the child: `_wrap_remote`, the + PVC/Job/Secret builders, `derive_cluster_phase`, `_observed_status`, + `_resolve_auth_data`, the phase constants, and the hydration image. + +Registering the new kind takes the usual four touches, plus a schema regen: + +- `apis/modelcachehydrations/` with its XRD and Composition; +- a `functionNames` entry in `flake.nix`; +- a Tarball entry in `crossplane-project.yaml`; +- `nix run .#build` to regenerate the model. + +No `mrap.yaml` change is needed, because it composes +`objects.kubernetes.m.crossplane.io`, already activated. + +## Alternatives considered + +### Keep `clusterSelector` as the whole footprint + +The status quo, and the divergence #186 is about: the selector and placement +drift, and the failure is a runtime mount error with nothing at apply time. +Deriving the footprint from placement removes the class of bug by construction. +The selector survives only for the case placement can't serve, pre-warming ahead +of a deployment. + +### Drop `modelCacheRef`; derive the model from the deployment + +#186 floats going further: drop the explicit `ModelCache` and `modelCacheRef`, and +derive caching from the model the deployment declares. Nothing structured declares +it today. The model lives in opaque engine args (`--model=…`), with the source +structured only on `ModelCache`. Fully automatic caching would need: + +- a structured model source on the deployment (`spec.template.spec.model`), and +- hydration keyed by model identity (a content hash) rather than cache name, so + two deployments of one model share a copy per cluster, which also changes the + serving mount contract. + +That is a larger, user-facing change with its own migration. This proposal keeps +`modelCacheRef` and solves the divergence, waste, sprawl, and lifecycle problems +without it. Identity-keyed sharing can be added on top later, since the child +already carries a source independent of how it was requested. + +### A separate fleet reconciler kind for reference counting + +A single hydration per cluster serves replicas across deployments and outlives any +one of them, and Crossplane composed resources are single-owner, so a +`ModelReplica` can't compose a shared hydration directly. A new fleet-scoped +reconciler kind could own all hydrations. Reusing `ModelCache` is simpler: it is +already the per-model, per-namespace resource, it already fans out per cluster, +and computing the placed set keeps ownership and the namespace security boundary +where they are. A dedicated reconciler is worth revisiting only if caching becomes +fully deployment-derived (the alternative above). + +### Conditions-only child status, like `ModelReplica` + +Mirroring `ModelReplica` exactly would collapse `ModelCache`'s per-cluster `phase` +(Pending/Hydrating/Ready/Failed) to a boolean, losing detail that +`kubectl get modelcache` shows today. The child carries a structured `status.phase` +so the parent preserves it. + +### Deliver the decomposition and the footprint change separately + +The decomposition (#210) is a pure refactor and could merge first. They are kept +together because the placed-set footprint only works once the per-cluster +hydration is its own reconciled unit. Splitting them would mean designing the +child's ownership twice. + +## Open questions + +- **Child kind name:** `ModelCacheHydration` (chosen) versus `ModelCacheReplica` + for symmetry with `ModelReplica`. `Hydration` names the lifecycle the child + owns; `Replica` implies a copy. +- **Readiness gating:** whether a replica gates its readiness on its cluster's + hydration (the `Hydrating` state above), or lets the mount block until the PVC + appears (less coupling, worse failure surface). Leaning toward the gate. + +## Interaction with related issues + +- **#115 (Modelplane-owned hydration image):** the Job builder moves to the child; + #115 becomes a localized image swap there. +- **#281 (multiple models per deployment):** the child stays single-source (one + repo, one PVC). #281 fans out per source later. +- **#204 (EFS load slow), #72 (KVOffloadTier), #71 (routing affinity):** + orthogonal. These are storage read performance and KV or prefix cache, not + model-weight staging. +- **#341 (ImageCache):** a different problem, pre-pulling the container image into + each node's local image store, which a PVC-based cache cannot do (the Kubelet + starts containers from node-local image layers, not a mounted volume). It shares + only the per-cluster fan-out shape, not the mechanism. + From 908b6de5a63eeb881578affe88008384cf09deeb Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 21 Jul 2026 13:30:44 -0700 Subject: [PATCH 02/11] Address hydration failure gating and 63-char naming in design The hydration gate had no failure exit: a bad token or bad revision would hold the replica in Hydrating forever. Add a Failed path that surfaces ArtifactReady=False and fails the replica, bounded by the Job's backoffLimit. Note that child_name keeps every composed name a valid 63-char DNS label. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 0420d74a8..e1328879b 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -128,6 +128,10 @@ Two things depend on this: 2. On upgrade the child adopts the existing, already-Bound PVC instead of provisioning a new one and re-downloading the weights. +`child_name` truncates the prefix and appends a hash, so every composed name stays +a valid DNS label at or under 63 characters, including the child's own name with +the cluster folded in. + ## Placement-driven footprint The problem (#186): a cache's `clusterSelector` and a deployment's placement are @@ -173,6 +177,13 @@ Rather than fail the mount, the replica gates readiness on its cluster's proceeds. The first replica on a new cluster pays the download once, as a visible state rather than a mount error. +Gating can't be open-ended. If hydration fails, a bad token, a bad revision, or +exhausted storage, the child reports `Failed`, the parent surfaces +`ArtifactReady=False` with reason `HydrationFailed`, and the gated replica fails +with that reason instead of sitting in `Hydrating`. The hydration `Job`'s +`backoffLimit` bounds retries, so a permanent failure stops and is reported rather +than retried forever. + ### Lifecycle Placement adds a cluster to the footprint. The last replica leaving removes it. From ab3609543a1749e26357d518093e9f5c124fb501 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 22 Jul 2026 09:18:27 -0700 Subject: [PATCH 03/11] Rework footprint to pre-warm-authoritative per review Adopt Nic's direction: the cache's clusterSelector is the authoritative footprint the platform team pre-warms, rather than deriving the footprint from replica placement. A deployment loads from the cache where staged and from the source elsewhere, decided at compose time via the footprint resolution that already exists (resolve_cache_footprint). This removes the double-download tax and the crossplane#7572 dependency, and relaxes the #189 placement constraint. Keep the ModelCacheHydration decomposition. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 234 +++++++++++++++++++-------------- 1 file changed, 132 insertions(+), 102 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index e1328879b..d4861a1e7 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -5,9 +5,10 @@ **Author:** Dennis Ramdass This document proposes `ModelCacheHydration`, a per-cluster child of `ModelCache`, -and uses it to make a cache's footprint follow where replicas are scheduled. It -builds on [modelcache.md](./modelcache.md) and [design.md](./design.md), and -addresses [#210](https://github.com/modelplaneai/modelplane/issues/210) (decompose +and makes a cache's `clusterSelector` the authoritative footprint that Modelplane +pre-warms ahead of deployment. It builds on [modelcache.md](./modelcache.md) and +[design.md](./design.md), and addresses +[#210](https://github.com/modelplaneai/modelplane/issues/210) (decompose `ModelCache`) and [#186](https://github.com/modelplaneai/modelplane/issues/186) (footprint diverges from placement) together. @@ -26,12 +27,11 @@ Two changes: cluster's hydration. Pinned by `spec.clusterName`, it resolves its `InferenceCluster` and auth `Secret`, composes the PVC, token `Secret`, and `Job`, runs the phase machine, and drops the `Job`/`Secret` once Ready. -2. **Derive the footprint from placement.** `ModelCache` stages onto the - clusters where its referencing replicas are placed, not onto a hand-maintained - `clusterSelector`. A selector, if set, is an opt-in pre-warm set, not the whole - footprint. - -A cache with no selector stages exactly where its consumers run: +2. **Pre-warm from the selector.** A cache's `clusterSelector` is the authoritative + footprint. The platform team declares which clusters hold the weights, and + Modelplane hydrates them ahead of any deployment. A deployment schedules on its + own selector and loads from the cache where the cache is staged, or from the + source where it isn't. No deployment pays the download twice. ```yaml apiVersion: modelplane.ai/v1alpha1 @@ -46,8 +46,9 @@ spec: authSecret: name: hf-token sizeGiB: 1500 - # No clusterSelector: footprint follows the ModelDeployments that - # reference this cache. Set one only to pre-warm ahead of placement. + clusterSelector: # the footprint: pre-warm these clusters + matchLabels: + modelplane.ai/tier: a100 ``` The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. @@ -55,9 +56,9 @@ The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. ## Architecture -Both fan-outs are per-cluster. The new link is `ModelReplica → ModelCache` (bold): -the cache's footprint is the set of clusters its referencing replicas are placed -on. +Both fan-outs are per-cluster. `ModelCache` stages onto the clusters its +`clusterSelector` matches. `ModelReplica` mounts the cache PVC where it exists and +loads from the source where it doesn't. The two are placed independently. ```mermaid flowchart TD @@ -67,7 +68,7 @@ flowchart TD end subgraph cp["Composed on the control cluster"] MR["ModelReplica\n(one per placed cluster)"] - MCH["ModelCacheHydration\n(one per cluster in footprint)"] + MCH["ModelCacheHydration\n(one per selected cluster)"] end subgraph wc["Workload cluster"] WL["serving workload\n(Deployment / LeaderWorkerSet)"] @@ -76,15 +77,14 @@ flowchart TD PVC["PVC (weights)"] end MD -->|schedules| MR - MC -->|"fan out per cluster"| MCH + MC -->|"fan out per selected cluster"| MCH MD -. modelCacheRef .-> MC - MR == "placement drives footprint" ==> MC MR --> WL MCH --> JOB MCH --> SEC MCH --> PVC JOB -->|writes| PVC - WL -->|"mounts /mnt/models"| PVC + WL -->|"mounts /mnt/models where staged"| PVC classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; class MCH new ``` @@ -132,50 +132,52 @@ Two things depend on this: a valid DNS label at or under 63 characters, including the child's own name with the cluster folded in. -## Placement-driven footprint - -The problem (#186): a cache's `clusterSelector` and a deployment's placement are -set independently. The scheduler places a replica on whichever matching cluster -has capacity; if the cache didn't stage there, the PVC is missing and the pod -fails to mount at runtime, with nothing visible at apply time. Staging everywhere -is the only safe workaround, and it wastes storage and pushes the download token -and private weights onto clusters running none of the team's deployments. - -Instead, derive the footprint from placement. Each reconcile, `ModelCache`: - -1. requires every `ModelReplica` in the fleet (the scheduler already does this for - capacity accounting); -2. keeps those that reference it (`spec.modelCacheRef.name` and namespace); -3. takes the clusters they're placed on. That set is the footprint, one - `ModelCacheHydration` each. - -A replica is then never placed on a cluster the model isn't staged to, and -credentials and weights reach only the clusters that run the model. - -`clusterSelector` stays, as an opt-in pre-warm set unioned with the placed set. It -covers the one thing pure placement can't do: warm a tier before any deployment -exists. The default (no selector) follows placement. - -### Reference counting and reclaim - -The placed set is the reference count: - -- Two deployments sharing the cache on cluster X contribute one entry; either one - tearing down leaves the other's, so X's hydration stays. -- When the last referencing replica leaves X (and X isn't pre-warmed), X drops - out, the parent stops composing that child, and the child and its PVC are - reclaimed. - -No hydration is retracted out from under a live replica, and an unreferenced cache -reclaims itself per cluster with no manual cleanup. +## Pre-warm-authoritative footprint + +The divergence (#186): a cache's `clusterSelector` and a deployment's placement are +set independently. If a replica is scheduled onto a cluster the cache didn't stage +to, the PVC is missing and the pod fails to mount at runtime, with nothing visible +at apply time. + +There are two ways to close that gap. Make the cache chase placement (hydrate +on-demand wherever a replica lands), or make placement tolerate a missing cache +(load from the source there). This proposal takes the second. The cache's +`clusterSelector` is the footprint the platform team declares, Modelplane pre-warms +those clusters ahead of any deployment, and a deployment that schedules elsewhere +loads from the source. + +Pre-warm is what makes a large model usable. Hydrating a 1.5TB cache PVC and +loading the model into a replica are two sequential copies. Paid on-demand, the +first deployment onto a new cluster must wait for both, which was roughly an hour +for a model the size of Kimi in testing. Pre-warm moves that hydration ahead of +time. The platform team pays it once. An author onto a warmed cluster then waits +only for the load of roughly fifteen minutes. No deployment ever pays the two +copies in series. + +Because the footprint is the static selector, `ModelCache` does not watch replica +placement and does not recompose when replicas move. It needs no cross-resource +watch to stay correct. + +### Cache or source, per replica + +A deployment keeps `modelCacheRef`, and the resolution already exists. +`compose-model-deployment` resolves the referenced cache's footprint today +(`resolve_cache_footprint`), which is how #189 constrains placement. Here it serves +a different purpose. For each replica, the function knows whether that replica's +cluster is in the footprint and passes the answer down. On a footprint cluster the +replica mounts the PVC and `base.cache_mount` injects `--model=`. Off the +footprint it injects nothing and the engine loads from the source, the existing +no-cache path. The cache-versus-source decision is known at compose time, so +forming engine args needs no runtime lookup. ### Hydrating before ready -A replica can be scheduled onto a cluster new to the model before its PVC exists. -Rather than fail the mount, the replica gates readiness on its cluster's -`ModelCacheHydration`: it reports `Hydrating` until the PVC is Bound, then -proceeds. The first replica on a new cluster pays the download once, as a visible -state rather than a mount error. +A replica can be scheduled onto a footprint cluster while its pre-warm is still +hydrating. Rather than fail the mount, the replica gates readiness on that cluster's +`ModelCacheHydration`, holding at `Hydrating` until the PVC is Bound. A replica +loading from the source doesn't gate at all. The replica watches the +`ModelCacheHydration` object, not the PVC directly, so a future cache that doesn't +use a PVC keeps the same readiness contract. Gating can't be open-ended. If hydration fails, a bad token, a bad revision, or exhausted storage, the child reports `Failed`, the parent surfaces @@ -186,30 +188,31 @@ than retried forever. ### Lifecycle -Placement adds a cluster to the footprint. The last replica leaving removes it. +The selector is the footprint. A cluster enters when it starts matching and leaves +when it stops (or when the `ModelCache` is deleted). The parent stamps a child per +matched cluster and reclaims the child and its PVC when the cluster drops out. No +replica reference counting is involved, because placement no longer drives the +footprint. ```mermaid flowchart TD - A["Replica scheduled onto cluster C"] --> B["ModelCache adds C to the placed set"] - B --> C["Stamp ModelCacheHydration for (cache, C)"] - C --> D["Child composes PVC + hydration Job\nphase: Hydrating"] - D --> E{"PVC Bound and\nJob complete?"} - E -- no --> F["Replica gates on the hydration\nreports Hydrating, holds"] - F --> E - E -- yes --> G["Child phase: Ready"] - G --> H["Replica mounts /mnt/models, becomes Ready"] - H --> I{"Last referencing\nreplica leaves C?"} - I -- no --> H - I -- yes --> J["C drops from the placed set\nchild + PVC reclaimed"] + A["clusterSelector matches cluster C"] --> B["Stamp ModelCacheHydration for (cache, C)"] + B --> C["Child composes PVC + hydration Job\nphase: Hydrating"] + C --> D{"PVC Bound and\nJob complete?"} + D -- no --> C + D -- yes --> E["Child phase: Ready"] + E --> F["Replicas on C mount /mnt/models;\nreplicas elsewhere load from source"] + F --> G{"C still matches\nthe selector?"} + G -- yes --> F + G -- no --> H["C drops out\nchild + PVC reclaimed"] ``` ## What the parent composes `compose-model-cache` becomes fan-out plus roll-up: -- **Resolve** the referencing `ModelReplica`s (the placed set) and, for pre-warm, - the `clusterSelector`. -- **Stamp** one child per cluster in `placed ∪ preWarm`, named +- **Resolve** the clusters the `clusterSelector` matches. +- **Stamp** one child per matched cluster, named `child_name("modelcache", ns, cache-name, cluster)`, copying down `huggingFace`, `clusterName`, the `authSecret` reference, and `cacheName`. - **Roll up** each child's `status.phase`/`Ready` into `status.clusters[]`, @@ -231,13 +234,35 @@ No `mrap.yaml` change is needed, because it composes ## Alternatives considered -### Keep `clusterSelector` as the whole footprint +### Hydrate on-demand from placement + +Derive the footprint from where replicas are placed. When a replica is scheduled +onto a cluster new to the model, the cache follows it there. This was the earlier +shape of this proposal and reads as the most automatic option. Three costs turned +it down: + +1. It needs [crossplane#7572](https://github.com/crossplane/crossplane/pull/7572) + to function. Without a watch on referencing `ModelReplica`s, nothing re-triggers + `compose-model-cache` when a new replica is placed, so the parent never learns to + stamp the new child. That change is approved and expected in Crossplane v2.4, so + the on-demand shape can't be built until then. +2. The first deployment onto a new cluster pays the hydrate and the load in series, + the download twice, which is the hour-long wait pre-warm removes. +3. A model implicitly carries its private weights and download token onto any + cluster it happens to be scheduled to, rather than onto the clusters the platform + team chose. -The status quo, and the divergence #186 is about: the selector and placement -drift, and the failure is a runtime mount error with nothing at apply time. -Deriving the footprint from placement removes the class of bug by construction. -The selector survives only for the case placement can't serve, pre-warming ahead -of a deployment. +Pre-warm avoids all three. A model never carries the cache to a cluster the +platform team didn't pre-hydrate. + +### Constrain placement to the footprint (#189) + +[#189](https://github.com/modelplaneai/modelplane/pull/189) prevents the stuck +mount by making the scheduler refuse to place a replica outside the cache +footprint. That closes the divergence but removes capacity. A cluster with room to +run the model stays unusable for it until the cache is staged there. Pre-warm takes +the opposite tack. A replica may schedule anywhere its own selector allows. Off the +footprint it loads from the source rather than being turned away. ### Drop `modelCacheRef`; derive the model from the deployment @@ -252,20 +277,21 @@ structured only on `ModelCache`. Fully automatic caching would need: serving mount contract. That is a larger, user-facing change with its own migration. This proposal keeps -`modelCacheRef` and solves the divergence, waste, sprawl, and lifecycle problems -without it. Identity-keyed sharing can be added on top later, since the child -already carries a source independent of how it was requested. - -### A separate fleet reconciler kind for reference counting - -A single hydration per cluster serves replicas across deployments and outlives any -one of them, and Crossplane composed resources are single-owner, so a -`ModelReplica` can't compose a shared hydration directly. A new fleet-scoped -reconciler kind could own all hydrations. Reusing `ModelCache` is simpler: it is -already the per-model, per-namespace resource, it already fans out per cluster, -and computing the placed set keeps ownership and the namespace security boundary -where they are. A dedicated reconciler is worth revisiting only if caching becomes -fully deployment-derived (the alternative above). +`modelCacheRef`, which is also what lets `compose-model-deployment` resolve the +footprint for the cache-or-source decision above. Identity-keyed sharing can be +added on top later, since the child already carries a source independent of how it +was requested. + +### A separate fleet reconciler kind + +A single hydration per cluster serves replicas across deployments, and Crossplane +composed resources are single-owner, so a `ModelReplica` can't compose a shared +hydration directly. A new fleet-scoped reconciler kind could own all hydrations. +Reusing `ModelCache` is simpler: it is already the per-model, per-namespace +resource, it already fans out per cluster, and pre-warm keeps ownership and the +namespace security boundary where they are. A dedicated reconciler is worth +revisiting only if caching becomes fully deployment-derived (the alternative +above). ### Conditions-only child status, like `ModelReplica` @@ -276,22 +302,27 @@ so the parent preserves it. ### Deliver the decomposition and the footprint change separately -The decomposition (#210) is a pure refactor and could merge first. They are kept -together because the placed-set footprint only works once the per-cluster -hydration is its own reconciled unit. Splitting them would mean designing the -child's ownership twice. +The decomposition (#210) is a pure refactor and could merge first. The footprint +change (#186) is lighter under pre-warm, since the selector stays authoritative as +it is today. The two are kept together because designing the child's ownership once +against the final footprint model is easier than doing it twice. ## Open questions - **Child kind name:** `ModelCacheHydration` (chosen) versus `ModelCacheReplica` for symmetry with `ModelReplica`. `Hydration` names the lifecycle the child owns; `Replica` implies a copy. -- **Readiness gating:** whether a replica gates its readiness on its cluster's - hydration (the `Hydrating` state above), or lets the mount block until the PVC - appears (less coupling, worse failure surface). Leaning toward the gate. +- **Empty selector:** whether a `ModelCache` with no `clusterSelector` stages + nowhere (nothing to pre-warm, so every replica loads from the source) or is + rejected at apply time as a likely mistake. Leaning toward staging nowhere. +- **Surfacing the source fallback:** a replica loading from the source on an + un-warmed cluster is slower to start, and that should be visible on the + `ModelDeployment` so an operator can see they missed a pre-warm. ## Interaction with related issues +- **#189 (constrain placement):** superseded here. Pre-warm relaxes the placement + constraint and loads from the source off the footprint, as above. - **#115 (Modelplane-owned hydration image):** the Job builder moves to the child; #115 becomes a localized image swap there. - **#281 (multiple models per deployment):** the child stays single-source (one @@ -303,4 +334,3 @@ child's ownership twice. each node's local image store, which a PVC-based cache cannot do (the Kubelet starts containers from node-local image layers, not a mounted volume). It shares only the per-cluster fan-out shape, not the mechanism. - From c12281a5bf157e73ff2fe0df5dab6aa7e3796326 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 28 Jul 2026 06:28:10 -0700 Subject: [PATCH 04/11] State plainly what approving the ModelCache design covers Add an explicit approval line so a review means something clear: it covers the ModelCacheHydration decomposition and the pre-warm-authoritative footprint, including load-from-source off the footprint. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index d4861a1e7..2845d7ffb 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -54,6 +54,10 @@ spec: The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. `ModelCacheHydration` is composed, never authored, the same as `ModelReplica`. +Approving this means agreeing to both changes: the `ModelCacheHydration` +decomposition (#210), and the pre-warm-authoritative footprint (#186) where a +deployment loads from the source on clusters the cache isn't staged to. + ## Architecture Both fan-outs are per-cluster. `ModelCache` stages onto the clusters its From b01b991a3b943495fb7de56b94b35902911a02ec Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 4 Aug 2026 09:08:35 -0700 Subject: [PATCH 05/11] Constrain placement to the cache footprint; inject env, not engine flags A cache-referencing deployment now runs only where the cache is pre-warmed, so the engine command is uniform and Modelplane injects no engine flags. Drop load-from-source off the footprint: it needed per-replica model-arg injection and didn't generalize to loader-plugin caches like ModelExpress. State the two modes with both actors named, the env-values-not-flags rule, and the two engine contracts (path vs loader plugin), and add reference-aware reclaim so a cache is never pulled out from under a live replica. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 175 ++++++++++++++++++++------------- 1 file changed, 105 insertions(+), 70 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 2845d7ffb..49130be67 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -27,11 +27,12 @@ Two changes: cluster's hydration. Pinned by `spec.clusterName`, it resolves its `InferenceCluster` and auth `Secret`, composes the PVC, token `Secret`, and `Job`, runs the phase machine, and drops the `Job`/`Secret` once Ready. -2. **Pre-warm from the selector.** A cache's `clusterSelector` is the authoritative - footprint. The platform team declares which clusters hold the weights, and - Modelplane hydrates them ahead of any deployment. A deployment schedules on its - own selector and loads from the cache where the cache is staged, or from the - source where it isn't. No deployment pays the download twice. +2. **Pre-warm from the selector, and place where it's warmed.** A cache's + `clusterSelector` is the authoritative footprint. The platform team declares which + clusters hold the weights, and Modelplane hydrates them ahead of any deployment. A + deployment that references the cache runs only on those clusters. A deployment that + references no cache runs anywhere its own selector allows and loads from the source + itself. No deployment pays the download twice. ```yaml apiVersion: modelplane.ai/v1alpha1 @@ -55,14 +56,15 @@ The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. `ModelCacheHydration` is composed, never authored, the same as `ModelReplica`. Approving this means agreeing to both changes: the `ModelCacheHydration` -decomposition (#210), and the pre-warm-authoritative footprint (#186) where a -deployment loads from the source on clusters the cache isn't staged to. +decomposition (#210), and a pre-warm-authoritative footprint (#186) where a +deployment that references a cache is placed only where that cache is pre-warmed. +Modelplane injects no engine flags for any of this. The section below explains why. ## Architecture Both fan-outs are per-cluster. `ModelCache` stages onto the clusters its -`clusterSelector` matches. `ModelReplica` mounts the cache PVC where it exists and -loads from the source where it doesn't. The two are placed independently. +`clusterSelector` matches, and a deployment that references the cache is placed only +onto those clusters, so the cache is present wherever its replicas run. ```mermaid flowchart TD @@ -80,7 +82,7 @@ flowchart TD SEC["token Secret"] PVC["PVC (weights)"] end - MD -->|schedules| MR + MD -->|"schedules (within the cache footprint)"| MR MC -->|"fan out per selected cluster"| MCH MD -. modelCacheRef .-> MC MR --> WL @@ -143,45 +145,64 @@ set independently. If a replica is scheduled onto a cluster the cache didn't sta to, the PVC is missing and the pod fails to mount at runtime, with nothing visible at apply time. -There are two ways to close that gap. Make the cache chase placement (hydrate -on-demand wherever a replica lands), or make placement tolerate a missing cache -(load from the source there). This proposal takes the second. The cache's -`clusterSelector` is the footprint the platform team declares, Modelplane pre-warms -those clusters ahead of any deployment, and a deployment that schedules elsewhere -loads from the source. - -Pre-warm is what makes a large model usable. Hydrating a 1.5TB cache PVC and -loading the model into a replica are two sequential copies. Paid on-demand, the -first deployment onto a new cluster must wait for both, which was roughly an hour -for a model the size of Kimi in testing. Pre-warm moves that hydration ahead of -time. The platform team pays it once. An author onto a warmed cluster then waits -only for the load of roughly fifteen minutes. No deployment ever pays the two -copies in series. +The fix is to make the cache's `clusterSelector` the authoritative footprint and +place against it. The platform team declares which clusters hold the weights, +Modelplane pre-warms them ahead of any deployment, and a deployment that references +the cache is scheduled only onto them. -Because the footprint is the static selector, `ModelCache` does not watch replica -placement and does not recompose when replicas move. It needs no cross-resource -watch to stay correct. - -### Cache or source, per replica +Pre-warm is what makes a large model usable. Hydrating a 1.5TB cache PVC and loading +the model into a replica are two sequential copies. Paid on-demand, the first +deployment onto a new cluster waits for both, roughly an hour for a model the size of +Kimi in testing. Pre-warm moves that hydration ahead of time. The platform team +absorbs it once. An author onto a warmed cluster then waits only for the load, +roughly fifteen minutes. -A deployment keeps `modelCacheRef`, and the resolution already exists. -`compose-model-deployment` resolves the referenced cache's footprint today -(`resolve_cache_footprint`), which is how #189 constrains placement. Here it serves -a different purpose. For each replica, the function knows whether that replica's -cluster is in the footprint and passes the answer down. On a footprint cluster the -replica mounts the PVC and `base.cache_mount` injects `--model=`. Off the -footprint it injects nothing and the engine loads from the source, the existing -no-cache path. The cache-versus-source decision is known at compose time, so -forming engine args needs no runtime lookup. +Because the footprint is the static selector, `ModelCache` does not watch replica +placement and does not recompose when replicas move. + +### Two modes + +A deployment is in one of two modes, and the two actors split cleanly. + +- **With a cache.** The ML team references a `ModelCache` and writes the engine + command. Modelplane places the deployment only on the clusters where the platform + team pre-warmed that cache. The cache is present wherever the replica runs, so the + start command is the same on every cluster it runs on. +- **Without a cache.** The ML team references no cache and writes the engine to load + from the source, with its own token. Modelplane places it on any cluster the ML + team's selector allows. This suits small or experimental models, and is slow for + large ones, since every replica downloads from the source. + +If a deployment references a cache but no cluster both matches its own selector and +holds that cache, it isn't placed, and the scheduler reports why rather than leaving +it Pending without a reason. + +### Modelplane injects env values, never engine flags + +The engine command stays the ML team's. Where a value depends on the cluster, +Modelplane injects an env var the command references, the way it injects +`MODELPLANE_LEADER_ADDRESS` today. It never writes an engine flag. + +A cache presents its weights to the engine one of two ways. Most backends put them at +a path (a PVC mount, an object-store CSI mount, a node-local cache), so the ML team +writes `--model=` with nothing engine-specific. A few, such as NVIDIA +ModelExpress or the Run:ai streamer, are engine loader plugins that need an +engine-specific `--load-format` and a loader-capable image; there the ML team writes +that flag and uses that image, because they chose that cache, and Modelplane wires the +env and the cluster-side pieces. Either way, placing only where the cache is +pre-warmed makes the command valid wherever the replica runs. How the bytes reach the +cluster (a shared filesystem, peer-to-peer distribution, GPU-to-GPU streaming) is the +platform team's concern and orthogonal to the command. The catalog of backends is a +separate design. Each has to present one of these two contracts. ### Hydrating before ready -A replica can be scheduled onto a footprint cluster while its pre-warm is still -hydrating. Rather than fail the mount, the replica gates readiness on that cluster's -`ModelCacheHydration`, holding at `Hydrating` until the PVC is Bound. A replica -loading from the source doesn't gate at all. The replica watches the -`ModelCacheHydration` object, not the PVC directly, so a future cache that doesn't -use a PVC keeps the same readiness contract. +Placement puts a replica on a footprint cluster as soon as the cluster matches the +selector, which can be before its pre-warm has finished hydrating. Rather than fail +the mount, the replica gates readiness on that cluster's `ModelCacheHydration`, +holding at `Hydrating` until the PVC is Bound. It watches the `ModelCacheHydration` +object, not the PVC directly, so a future cache that doesn't use a PVC keeps the same +readiness contract. Gating can't be open-ended. If hydration fails, a bad token, a bad revision, or exhausted storage, the child reports `Failed`, the parent surfaces @@ -192,11 +213,12 @@ than retried forever. ### Lifecycle -The selector is the footprint. A cluster enters when it starts matching and leaves -when it stops (or when the `ModelCache` is deleted). The parent stamps a child per -matched cluster and reclaims the child and its PVC when the cluster drops out. No -replica reference counting is involved, because placement no longer drives the -footprint. +The selector is the footprint. A cluster enters when it starts matching and drops out +when it stops. The parent stamps a child per matched cluster and reclaims the child +and its PVC once the cluster drops out. One guard: reclaim holds while a live replica +still uses the cache on that cluster, so it is never pulled out from under a running +pod. For the same reason a `ModelCache` with live referencing deployments can't be +deleted. A finalizer holds it until they leave. ```mermaid flowchart TD @@ -205,10 +227,12 @@ flowchart TD C --> D{"PVC Bound and\nJob complete?"} D -- no --> C D -- yes --> E["Child phase: Ready"] - E --> F["Replicas on C mount /mnt/models;\nreplicas elsewhere load from source"] + E --> F["Cache-referencing deployments\nplace only on C and mount there"] F --> G{"C still matches\nthe selector?"} G -- yes --> F - G -- no --> H["C drops out\nchild + PVC reclaimed"] + G -- no --> H{"Any live replica\nstill using the cache on C?"} + H -- yes --> F + H -- no --> I["C drops out\nchild + PVC reclaimed"] ``` ## What the parent composes @@ -259,14 +283,27 @@ it down: Pre-warm avoids all three. A model never carries the cache to a cluster the platform team didn't pre-hydrate. -### Constrain placement to the footprint (#189) +### Load from the source off the footprint + +An earlier shape let a cache-referencing deployment run anywhere and load from the +source on clusters without the cache, so a cache never limited placement. It needs +Modelplane to vary the model reference per replica (the mount path where the cache +is, the source repo where it isn't). That works for a plain mount, through an +injected env var, but it doesn't generalize. A loader-plugin cache like ModelExpress +needs an engine-specific flag and image that are invalid off the cache, so those +deployments have to be placement-constrained anyway. Constraining placement for every +cache is simpler and consistent, and it keeps Modelplane out of the engine's flags. A +deployment that wants to run without a cache references none and loads from the source +directly. + +### Always populate a fixed path -[#189](https://github.com/modelplaneai/modelplane/pull/189) prevents the stuck -mount by making the scheduler refuse to place a replica outside the cache -footprint. That closes the divergence but removes capacity. A cluster with room to -run the model stays unusable for it until the cache is staged there. Pre-warm takes -the opposite tack. A replica may schedule anywhere its own selector allows. Off the -footprint it loads from the source rather than being turned away. +Two ways to make the start command uniform: require every deployment to use a cache, +or reintroduce `spec.model` and pre-fetch it into a path before the engine starts. +Requiring a cache drops the lightweight no-cache path for quick or experimental +models. A per-pod pre-fetch has no reuse, since every replica still downloads the +whole model, so it costs a load from the source plus an extra copy with none of a +cache's benefit. Neither earns the uniformity. ### Drop `modelCacheRef`; derive the model from the deployment @@ -281,10 +318,10 @@ structured only on `ModelCache`. Fully automatic caching would need: serving mount contract. That is a larger, user-facing change with its own migration. This proposal keeps -`modelCacheRef`, which is also what lets `compose-model-deployment` resolve the -footprint for the cache-or-source decision above. Identity-keyed sharing can be -added on top later, since the child already carries a source independent of how it -was requested. +`modelCacheRef`, which is what a deployment references to say it wants a cache, and +what the scheduler resolves to a footprint to place against. Identity-keyed sharing +can be added on top later, since the child already carries a source independent of +how it was requested. ### A separate fleet reconciler kind @@ -316,17 +353,15 @@ against the final footprint model is easier than doing it twice. - **Child kind name:** `ModelCacheHydration` (chosen) versus `ModelCacheReplica` for symmetry with `ModelReplica`. `Hydration` names the lifecycle the child owns; `Replica` implies a copy. -- **Empty selector:** whether a `ModelCache` with no `clusterSelector` stages - nowhere (nothing to pre-warm, so every replica loads from the source) or is - rejected at apply time as a likely mistake. Leaning toward staging nowhere. -- **Surfacing the source fallback:** a replica loading from the source on an - un-warmed cluster is slower to start, and that should be visible on the - `ModelDeployment` so an operator can see they missed a pre-warm. +- **Empty selector:** a `ModelCache` with no `clusterSelector` stages nowhere, so a + deployment that references it has no cluster to run on. Reject it at apply time as a + likely mistake, or treat an empty selector as every cluster. Leaning toward reject. ## Interaction with related issues -- **#189 (constrain placement):** superseded here. Pre-warm relaxes the placement - constraint and loads from the source off the footprint, as above. +- **#189 (constrain placement):** kept, not superseded. #189's rule (a + cache-referencing deployment places only within the cache footprint) is the design + here; pre-warm makes the cache's `clusterSelector` the footprint that rule uses. - **#115 (Modelplane-owned hydration image):** the Job builder moves to the child; #115 becomes a localized image swap there. - **#281 (multiple models per deployment):** the child stays single-source (one From bbfd46e06c2dc46430546f9fae4ceeb734afb328 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 4 Aug 2026 09:23:35 -0700 Subject: [PATCH 06/11] Resolve open questions and tighten the alternatives Fold the two open questions into decisions in the body: the child is named ModelCacheHydration for the lifecycle it owns, and a ModelCache with no clusterSelector is rejected at apply time. Remove the Open questions section. Cut the Alternatives roughly in half so each rejected path states its reason once, so a reader can see what the doc proposes without wading through the paths not taken. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 121 ++++++++++++--------------------- 1 file changed, 45 insertions(+), 76 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 49130be67..6f23a4678 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -98,8 +98,9 @@ flowchart TD ## The per-cluster child `ModelCacheHydration` is a namespaced composite pinned to one cluster, the cache -analogue of `ModelReplica`. Its spec carries what one cluster's hydration needs -and nothing about fan-out: +analogue of `ModelReplica`. The name states the lifecycle the child owns, where +`ModelCacheReplica` would imply a copy of the parent. Its spec carries what one +cluster's hydration needs and nothing about fan-out: - **`clusterName`**, the cluster it stages onto. Pinned at creation; the parent re-places only if the cluster disappears. @@ -158,7 +159,9 @@ absorbs it once. An author onto a warmed cluster then waits only for the load, roughly fifteen minutes. Because the footprint is the static selector, `ModelCache` does not watch replica -placement and does not recompose when replicas move. +placement and does not recompose when replicas move. A `ModelCache` with no +`clusterSelector` stages nowhere, so Modelplane rejects it at apply time rather +than admit a cache no deployment can place against. ### Two modes @@ -264,98 +267,64 @@ No `mrap.yaml` change is needed, because it composes ### Hydrate on-demand from placement -Derive the footprint from where replicas are placed. When a replica is scheduled -onto a cluster new to the model, the cache follows it there. This was the earlier -shape of this proposal and reads as the most automatic option. Three costs turned -it down: - -1. It needs [crossplane#7572](https://github.com/crossplane/crossplane/pull/7572) - to function. Without a watch on referencing `ModelReplica`s, nothing re-triggers - `compose-model-cache` when a new replica is placed, so the parent never learns to - stamp the new child. That change is approved and expected in Crossplane v2.4, so - the on-demand shape can't be built until then. -2. The first deployment onto a new cluster pays the hydrate and the load in series, - the download twice, which is the hour-long wait pre-warm removes. -3. A model implicitly carries its private weights and download token onto any - cluster it happens to be scheduled to, rather than onto the clusters the platform - team chose. - -Pre-warm avoids all three. A model never carries the cache to a cluster the -platform team didn't pre-hydrate. +Derive the footprint from where replicas are placed, so a cache follows a replica +onto a cluster new to the model. This was the earlier shape of this proposal and +reads as the most automatic. Three costs turned it down. It needs +[crossplane#7572](https://github.com/crossplane/crossplane/pull/7572) to re-trigger +`compose-model-cache` when a new replica appears, which isn't out until Crossplane +v2.4. The first deployment onto a new cluster still pays the hydrate and the load in +series, the hour-long wait pre-warm removes. And a model carries its private weights +and token onto whatever cluster it happens to run on, rather than the clusters the +platform team chose. ### Load from the source off the footprint -An earlier shape let a cache-referencing deployment run anywhere and load from the -source on clusters without the cache, so a cache never limited placement. It needs -Modelplane to vary the model reference per replica (the mount path where the cache -is, the source repo where it isn't). That works for a plain mount, through an -injected env var, but it doesn't generalize. A loader-plugin cache like ModelExpress -needs an engine-specific flag and image that are invalid off the cache, so those -deployments have to be placement-constrained anyway. Constraining placement for every -cache is simpler and consistent, and it keeps Modelplane out of the engine's flags. A -deployment that wants to run without a cache references none and loads from the source -directly. +Let a cache-referencing deployment run anywhere and load from the source where the +cache is absent, so a cache never limits placement. This needs Modelplane to vary the +model reference per replica, which works for a plain mount but not for a loader-plugin +cache like ModelExpress, whose flag and image are invalid off the cache. Constraining +placement for every cache is simpler and keeps Modelplane out of the engine's flags. +A deployment that wants no cache references none. ### Always populate a fixed path -Two ways to make the start command uniform: require every deployment to use a cache, -or reintroduce `spec.model` and pre-fetch it into a path before the engine starts. -Requiring a cache drops the lightweight no-cache path for quick or experimental -models. A per-pod pre-fetch has no reuse, since every replica still downloads the -whole model, so it costs a load from the source plus an extra copy with none of a -cache's benefit. Neither earns the uniformity. +Make the start command uniform another way: require every deployment to use a cache, +or reintroduce `spec.model` and pre-fetch it before the engine starts. Requiring a +cache drops the lightweight no-cache path. A per-pod pre-fetch has no reuse, so it +costs a load plus an extra copy with none of a cache's benefit. Neither earns the +uniformity. ### Drop `modelCacheRef`; derive the model from the deployment -#186 floats going further: drop the explicit `ModelCache` and `modelCacheRef`, and -derive caching from the model the deployment declares. Nothing structured declares -it today. The model lives in opaque engine args (`--model=…`), with the source -structured only on `ModelCache`. Fully automatic caching would need: - -- a structured model source on the deployment (`spec.template.spec.model`), and -- hydration keyed by model identity (a content hash) rather than cache name, so - two deployments of one model share a copy per cluster, which also changes the - serving mount contract. - -That is a larger, user-facing change with its own migration. This proposal keeps -`modelCacheRef`, which is what a deployment references to say it wants a cache, and -what the scheduler resolves to a footprint to place against. Identity-keyed sharing -can be added on top later, since the child already carries a source independent of -how it was requested. +#186 floats dropping the explicit `ModelCache` and `modelCacheRef` and deriving +caching from the model the deployment declares. Nothing structured declares the model +today. It lives in opaque engine args. Automatic caching would need a structured model +source on the deployment and hydration keyed by model identity rather than cache name, +which also changes the serving mount contract. That is a larger user-facing change with +its own migration. This proposal keeps `modelCacheRef`, and identity-keyed sharing can +be added on top later, since the child already carries a source independent of how it +was requested. ### A separate fleet reconciler kind A single hydration per cluster serves replicas across deployments, and Crossplane composed resources are single-owner, so a `ModelReplica` can't compose a shared -hydration directly. A new fleet-scoped reconciler kind could own all hydrations. -Reusing `ModelCache` is simpler: it is already the per-model, per-namespace -resource, it already fans out per cluster, and pre-warm keeps ownership and the -namespace security boundary where they are. A dedicated reconciler is worth -revisiting only if caching becomes fully deployment-derived (the alternative -above). +hydration. A new fleet-scoped reconciler could own all hydrations. Reusing `ModelCache` +is simpler, since it is already the per-model, per-namespace resource that fans out per +cluster and keeps the namespace boundary where it is. A dedicated reconciler is worth +revisiting only if caching becomes fully deployment-derived. -### Conditions-only child status, like `ModelReplica` +### Conditions-only child status -Mirroring `ModelReplica` exactly would collapse `ModelCache`'s per-cluster `phase` -(Pending/Hydrating/Ready/Failed) to a boolean, losing detail that -`kubectl get modelcache` shows today. The child carries a structured `status.phase` -so the parent preserves it. +Mirroring `ModelReplica`'s boolean conditions would collapse `ModelCache`'s per-cluster +`phase` (Pending/Hydrating/Ready/Failed) that `kubectl get modelcache` shows today. The +child carries a structured `status.phase` so the parent preserves it. ### Deliver the decomposition and the footprint change separately -The decomposition (#210) is a pure refactor and could merge first. The footprint -change (#186) is lighter under pre-warm, since the selector stays authoritative as -it is today. The two are kept together because designing the child's ownership once -against the final footprint model is easier than doing it twice. - -## Open questions - -- **Child kind name:** `ModelCacheHydration` (chosen) versus `ModelCacheReplica` - for symmetry with `ModelReplica`. `Hydration` names the lifecycle the child - owns; `Replica` implies a copy. -- **Empty selector:** a `ModelCache` with no `clusterSelector` stages nowhere, so a - deployment that references it has no cluster to run on. Reject it at apply time as a - likely mistake, or treat an empty selector as every cluster. Leaning toward reject. +The decomposition (#210) is a pure refactor and could merge first. The two are kept +together because designing the child's ownership once against the final footprint model +is easier than doing it twice. ## Interaction with related issues From 74470db64f0f3f0105832d44979f9fe80922e965 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 4 Aug 2026 09:26:26 -0700 Subject: [PATCH 07/11] Format the two-contract and rejected-alternative lists for scanning The two engine contracts (path vs loader plugin) and the three costs of the on-demand alternative were dense paragraphs describing parallel items. Set each as a short list so a reader scans the choices instead of parsing them out of prose. No wording change beyond the split. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 42 +++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 6f23a4678..16884a1a2 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -186,17 +186,21 @@ The engine command stays the ML team's. Where a value depends on the cluster, Modelplane injects an env var the command references, the way it injects `MODELPLANE_LEADER_ADDRESS` today. It never writes an engine flag. -A cache presents its weights to the engine one of two ways. Most backends put them at -a path (a PVC mount, an object-store CSI mount, a node-local cache), so the ML team -writes `--model=` with nothing engine-specific. A few, such as NVIDIA -ModelExpress or the Run:ai streamer, are engine loader plugins that need an -engine-specific `--load-format` and a loader-capable image; there the ML team writes -that flag and uses that image, because they chose that cache, and Modelplane wires the -env and the cluster-side pieces. Either way, placing only where the cache is -pre-warmed makes the command valid wherever the replica runs. How the bytes reach the -cluster (a shared filesystem, peer-to-peer distribution, GPU-to-GPU streaming) is the -platform team's concern and orthogonal to the command. The catalog of backends is a -separate design. Each has to present one of these two contracts. +A cache presents its weights to the engine one of two ways. + +- **A path.** Most backends put the weights at a path (a PVC mount, an object-store + CSI mount, a node-local cache), so the ML team writes `--model=` with nothing + engine-specific. +- **A loader plugin.** A few, such as NVIDIA ModelExpress or the Run:ai streamer, are + engine loader plugins that need an engine-specific `--load-format` and a + loader-capable image. The ML team writes that flag and uses that image, because they + chose that cache, and Modelplane wires the env and the cluster-side pieces. + +Either way, placing only where the cache is pre-warmed makes the command valid wherever +the replica runs. How the bytes reach the cluster (a shared filesystem, peer-to-peer +distribution, GPU-to-GPU streaming) is the platform team's concern and orthogonal to +the command. The catalog of backends is a separate design, and each backend has to +present one of these two contracts. ### Hydrating before ready @@ -269,13 +273,15 @@ No `mrap.yaml` change is needed, because it composes Derive the footprint from where replicas are placed, so a cache follows a replica onto a cluster new to the model. This was the earlier shape of this proposal and -reads as the most automatic. Three costs turned it down. It needs -[crossplane#7572](https://github.com/crossplane/crossplane/pull/7572) to re-trigger -`compose-model-cache` when a new replica appears, which isn't out until Crossplane -v2.4. The first deployment onto a new cluster still pays the hydrate and the load in -series, the hour-long wait pre-warm removes. And a model carries its private weights -and token onto whatever cluster it happens to run on, rather than the clusters the -platform team chose. +reads as the most automatic. Three costs turned it down. + +- It needs [crossplane#7572](https://github.com/crossplane/crossplane/pull/7572) to + re-trigger `compose-model-cache` when a new replica appears, which isn't out until + Crossplane v2.4. +- The first deployment onto a new cluster still pays the hydrate and the load in + series, the hour-long wait pre-warm removes. +- A model carries its private weights and token onto whatever cluster it happens to + run on, rather than the clusters the platform team chose. ### Load from the source off the footprint From f65af879853d7c67b941fabcb1036c7ad37cbe16 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 4 Aug 2026 09:33:59 -0700 Subject: [PATCH 08/11] Ground the two engine contracts in SGLang and TensorRT-LLM, not just vLLM The contract text cited only vLLM flags and read as if --load-format were universal. Stress-testing against SGLang and TensorRT-LLM shows the model holds but the flag names are engine-specific: SGLang reads --model-path, not --model, and ModelExpress provides loaders across vLLM, SGLang, and TensorRT-LLM. Name the concrete flags and state that the model-naming flag belongs to the engine in both contracts, which is the reason Modelplane injects env values and not flags. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 16884a1a2..c9e25db24 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -189,18 +189,22 @@ Modelplane injects an env var the command references, the way it injects A cache presents its weights to the engine one of two ways. - **A path.** Most backends put the weights at a path (a PVC mount, an object-store - CSI mount, a node-local cache), so the ML team writes `--model=` with nothing - engine-specific. -- **A loader plugin.** A few, such as NVIDIA ModelExpress or the Run:ai streamer, are - engine loader plugins that need an engine-specific `--load-format` and a - loader-capable image. The ML team writes that flag and uses that image, because they - chose that cache, and Modelplane wires the env and the cluster-side pieces. - -Either way, placing only where the cache is pre-warmed makes the command valid wherever -the replica runs. How the bytes reach the cluster (a shared filesystem, peer-to-peer -distribution, GPU-to-GPU streaming) is the platform team's concern and orthogonal to -the command. The catalog of backends is a separate design, and each backend has to -present one of these two contracts. + CSI mount, a node-local cache). The ML team points the engine at it with the engine's + own flag, `--model` on vLLM or `--model-path` on SGLang, and the path reads the same + whatever transport staged it there. +- **A loader plugin.** A few backends are engine loader plugins. NVIDIA ModelExpress + provides first-class loaders for vLLM, SGLang, and TensorRT-LLM, and the Run:ai streamer + plugs into vLLM and SGLang. Each needs the engine's own `--load-format` and a + loader-capable image. The ML team writes that, because they chose that cache, and + Modelplane wires the env and the cluster-side pieces. + +The flag that names the model belongs to the engine either way, which is exactly why +Modelplane stays out of it and injects only env values. Placing only where the cache is +pre-warmed makes whatever the ML team wrote valid wherever the replica runs. How the +bytes reach the cluster (a shared filesystem, peer-to-peer distribution, GPU-to-GPU +streaming) is the platform team's concern and orthogonal to the command. The catalog of +backends is a separate design, and each backend has to present one of these two +contracts. ### Hydrating before ready From 390a67429a497171229c39f33d42f721719ffde1 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 4 Aug 2026 09:40:26 -0700 Subject: [PATCH 09/11] Match the voice and sentence rhythm of the existing design docs The doc read like a generated draft next to design.md: decorative bold-bullet lists where prose belonged, third-person framing, and trailing comma-appositives where a colon or parenthetical reads cleaner. Move the summary to first person ("I propose two changes"), set the two modes and the two contracts as bold lead-in paragraphs rather than bullets, echo the established persona split (platform owns what's cached and where, ML owns what to run), and use colons and parentheses for the inline expansions. No change to the proposal itself. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 98 ++++++++++++++++++---------------- 1 file changed, 51 insertions(+), 47 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index c9e25db24..49c5a9ab0 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -21,18 +21,19 @@ drop-Job-after-Ready cleanup), threading a `cluster_name` through every method. It's also the only fan-out that doesn't follow the `ModelDeployment` → `ModelReplica` pattern. -Two changes: - -1. **Split out `ModelCacheHydration`**, a per-cluster child that owns one - cluster's hydration. Pinned by `spec.clusterName`, it resolves its - `InferenceCluster` and auth `Secret`, composes the PVC, token `Secret`, and - `Job`, runs the phase machine, and drops the `Job`/`Secret` once Ready. -2. **Pre-warm from the selector, and place where it's warmed.** A cache's - `clusterSelector` is the authoritative footprint. The platform team declares which - clusters hold the weights, and Modelplane hydrates them ahead of any deployment. A - deployment that references the cache runs only on those clusters. A deployment that - references no cache runs anywhere its own selector allows and loads from the source - itself. No deployment pays the download twice. +I propose two changes. + +**Split out `ModelCacheHydration`.** A per-cluster child that owns one cluster's +hydration. Pinned by `spec.clusterName`, it resolves its `InferenceCluster` and auth +`Secret`, composes the PVC, token `Secret`, and `Job`, runs the phase machine, and +drops the `Job`/`Secret` once Ready. + +**Pre-warm from the selector, and place where it's warmed.** A cache's `clusterSelector` +is the authoritative footprint. The platform team declares which clusters hold the +weights, and Modelplane hydrates them ahead of any deployment. A deployment that +references the cache runs only on those clusters. A deployment that references no cache +runs anywhere its own selector allows and loads from the source itself. No deployment +pays the download twice. ```yaml apiVersion: modelplane.ai/v1alpha1 @@ -58,7 +59,7 @@ The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. Approving this means agreeing to both changes: the `ModelCacheHydration` decomposition (#210), and a pre-warm-authoritative footprint (#186) where a deployment that references a cache is placed only where that cache is pre-warmed. -Modelplane injects no engine flags for any of this. The section below explains why. +Modelplane injects no engine flags for any of this. The sections below explain why. ## Architecture @@ -153,10 +154,10 @@ the cache is scheduled only onto them. Pre-warm is what makes a large model usable. Hydrating a 1.5TB cache PVC and loading the model into a replica are two sequential copies. Paid on-demand, the first -deployment onto a new cluster waits for both, roughly an hour for a model the size of -Kimi in testing. Pre-warm moves that hydration ahead of time. The platform team -absorbs it once. An author onto a warmed cluster then waits only for the load, -roughly fifteen minutes. +deployment onto a new cluster waits for both. For a model the size of Kimi that's +roughly an hour in our testing. Pre-warm moves that hydration ahead of time, and the +platform team absorbs it once. An author onto a warmed cluster then waits only for the +load, roughly fifteen minutes. Because the footprint is the static selector, `ModelCache` does not watch replica placement and does not recompose when replicas move. A `ModelCache` with no @@ -165,20 +166,22 @@ than admit a cache no deployment can place against. ### Two modes -A deployment is in one of two modes, and the two actors split cleanly. +A deployment runs in one of two modes, split along the personas: the platform team +owns *what's cached and where*, the ML team owns *what to run*. + +**With a cache.** The ML team references a `ModelCache` and writes the engine command. +Modelplane places the deployment only on the clusters where the platform team pre-warmed +that cache, so the cache is present wherever the replica runs and the start command is +the same on every cluster. -- **With a cache.** The ML team references a `ModelCache` and writes the engine - command. Modelplane places the deployment only on the clusters where the platform - team pre-warmed that cache. The cache is present wherever the replica runs, so the - start command is the same on every cluster it runs on. -- **Without a cache.** The ML team references no cache and writes the engine to load - from the source, with its own token. Modelplane places it on any cluster the ML - team's selector allows. This suits small or experimental models, and is slow for - large ones, since every replica downloads from the source. +**Without a cache.** The ML team references no cache and writes the engine to load from +the source with its own token. Modelplane places it on any cluster its own selector +allows. This suits small or experimental models, and is slow for large ones, because +every replica downloads from the source. -If a deployment references a cache but no cluster both matches its own selector and -holds that cache, it isn't placed, and the scheduler reports why rather than leaving -it Pending without a reason. +When a deployment references a cache but no cluster both matches its own selector and +holds that cache, it isn't placed, and the scheduler reports why rather than leaving it +Pending without a reason. ### Modelplane injects env values, never engine flags @@ -188,21 +191,22 @@ Modelplane injects an env var the command references, the way it injects A cache presents its weights to the engine one of two ways. -- **A path.** Most backends put the weights at a path (a PVC mount, an object-store - CSI mount, a node-local cache). The ML team points the engine at it with the engine's - own flag, `--model` on vLLM or `--model-path` on SGLang, and the path reads the same - whatever transport staged it there. -- **A loader plugin.** A few backends are engine loader plugins. NVIDIA ModelExpress - provides first-class loaders for vLLM, SGLang, and TensorRT-LLM, and the Run:ai streamer - plugs into vLLM and SGLang. Each needs the engine's own `--load-format` and a - loader-capable image. The ML team writes that, because they chose that cache, and - Modelplane wires the env and the cluster-side pieces. - -The flag that names the model belongs to the engine either way, which is exactly why -Modelplane stays out of it and injects only env values. Placing only where the cache is -pre-warmed makes whatever the ML team wrote valid wherever the replica runs. How the -bytes reach the cluster (a shared filesystem, peer-to-peer distribution, GPU-to-GPU -streaming) is the platform team's concern and orthogonal to the command. The catalog of +**A path.** Most backends put the weights at a path (a PVC mount, an object-store CSI +mount, or a node-local cache). The ML team points the engine at it with the engine's own +flag, `--model` on vLLM or `--model-path` on SGLang, and the path reads the same +whatever transport staged it there. + +**A loader plugin.** A few backends are engine loader plugins. NVIDIA ModelExpress +provides first-class loaders for vLLM, SGLang, and TensorRT-LLM, and the Run:ai streamer +plugs into vLLM and SGLang. Each needs the engine's own `--load-format` and a +loader-capable image. The ML team writes that, because they chose the cache, and +Modelplane wires the env and the cluster-side pieces. + +Either way the flag that names the model belongs to the engine, so Modelplane stays out +of it and injects only env values. Placing only where the cache is pre-warmed +makes whatever the ML team wrote valid wherever the replica runs. How the bytes reach +the cluster, whether a shared filesystem, peer-to-peer distribution, or GPU-to-GPU +streaming, is the platform team's concern and orthogonal to the command. The catalog of backends is a separate design, and each backend has to present one of these two contracts. @@ -215,8 +219,8 @@ holding at `Hydrating` until the PVC is Bound. It watches the `ModelCacheHydrati object, not the PVC directly, so a future cache that doesn't use a PVC keeps the same readiness contract. -Gating can't be open-ended. If hydration fails, a bad token, a bad revision, or -exhausted storage, the child reports `Failed`, the parent surfaces +Gating can't be open-ended. If hydration fails (a bad token, a bad revision, +exhausted storage), the child reports `Failed`, the parent surfaces `ArtifactReady=False` with reason `HydrationFailed`, and the gated replica fails with that reason instead of sitting in `Hydrating`. The hydration `Job`'s `backoffLimit` bounds retries, so a permanent failure stops and is reported rather @@ -283,7 +287,7 @@ reads as the most automatic. Three costs turned it down. re-trigger `compose-model-cache` when a new replica appears, which isn't out until Crossplane v2.4. - The first deployment onto a new cluster still pays the hydrate and the load in - series, the hour-long wait pre-warm removes. + series: the hour-long wait pre-warm removes. - A model carries its private weights and token onto whatever cluster it happens to run on, rather than the clusters the platform team chose. From e25ff617d5b88da52466f6c044372169e47cfa9a Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 07:27:06 -0700 Subject: [PATCH 10/11] Show the env-injection mechanism with a ModelDeployment example The env section asserted that Modelplane injects env values without showing what value or how a deployment uses it. State the stronger truth: in the common path-cache case it injects nothing, because placement constrains the mount to a uniform path the ML team writes directly. Add a ModelDeployment snippet, and keep env injection as the mechanism for a value that varies per cluster (the leader address today, a loader-plugin's server address), referenced with Kubernetes $(VAR) expansion. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 49c5a9ab0..384fb3d2b 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -185,9 +185,29 @@ Pending without a reason. ### Modelplane injects env values, never engine flags -The engine command stays the ML team's. Where a value depends on the cluster, -Modelplane injects an env var the command references, the way it injects -`MODELPLANE_LEADER_ADDRESS` today. It never writes an engine flag. +The engine command stays the ML team's, and Modelplane never writes an engine flag. In +the common case it injects nothing. The cache mounts at the same path on every footprint +cluster, so the ML team writes that path directly. + +```yaml +engines: +- name: kimi-k2 + members: + - role: Standalone + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.11.0 + args: + - --model=/mnt/models # the cache mount, uniform across the footprint +``` + +Injection covers the case where a value varies from one cluster to the next. Modelplane +sets it as an env var and the ML team references it with Kubernetes `$(VAR)` expansion, +the way `MODELPLANE_LEADER_ADDRESS` supplies a multi-node leader address today. A +loader-plugin cache uses the same mechanism when its client needs a cluster-specific +server address. A cache presents its weights to the engine one of two ways. @@ -202,8 +222,8 @@ plugs into vLLM and SGLang. Each needs the engine's own `--load-format` and a loader-capable image. The ML team writes that, because they chose the cache, and Modelplane wires the env and the cluster-side pieces. -Either way the flag that names the model belongs to the engine, so Modelplane stays out -of it and injects only env values. Placing only where the cache is pre-warmed +Either way the flag that names the model belongs to the engine, and Modelplane stays out +of it. Placing only where the cache is pre-warmed makes whatever the ML team wrote valid wherever the replica runs. How the bytes reach the cluster, whether a shared filesystem, peer-to-peer distribution, or GPU-to-GPU streaming, is the platform team's concern and orthogonal to the command. The catalog of From d21ce3dbddcab5b59199c64ef9fe8b30b4bc86eb Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Thu, 6 Aug 2026 10:11:11 -0700 Subject: [PATCH 11/11] Lock the injected var name and show the $(MODELPLANE_MODEL) UX Nic asked for the concrete env var name and the caller-facing shape. With a cache Modelplane owns the mount path, so it injects it as MODELPLANE_MODEL and the ML team writes --model=$(MODELPLANE_MODEL), never hardcoding the mount. Same $(VAR) mechanism as MODELPLANE_LEADER_ADDRESS. Without a cache they write --model= directly. Replaces the earlier injects-nothing framing. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/modelcache-hydration.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md index 384fb3d2b..3c8cc1439 100644 --- a/design/modelcache-hydration.md +++ b/design/modelcache-hydration.md @@ -185,9 +185,10 @@ Pending without a reason. ### Modelplane injects env values, never engine flags -The engine command stays the ML team's, and Modelplane never writes an engine flag. In -the common case it injects nothing. The cache mounts at the same path on every footprint -cluster, so the ML team writes that path directly. +The engine command stays the ML team's, and Modelplane never writes an engine flag. When +Modelplane owns a value the engine needs, it injects that value as an env var and the ML +team references it with Kubernetes `$(VAR)` expansion. With a cache Modelplane owns where +the weights are mounted, so it injects the mount path as `MODELPLANE_MODEL`. ```yaml engines: @@ -200,21 +201,21 @@ engines: - name: engine image: vllm/vllm-openai:v0.11.0 args: - - --model=/mnt/models # the cache mount, uniform across the footprint + - --model=$(MODELPLANE_MODEL) # Modelplane sets this to the cache mount ``` -Injection covers the case where a value varies from one cluster to the next. Modelplane -sets it as an env var and the ML team references it with Kubernetes `$(VAR)` expansion, -the way `MODELPLANE_LEADER_ADDRESS` supplies a multi-node leader address today. A -loader-plugin cache uses the same mechanism when its client needs a cluster-specific -server address. +The ML team never hardcodes Modelplane's mount path, so Modelplane is free to change it. +It's the same mechanism `MODELPLANE_LEADER_ADDRESS` uses for a multi-node leader address +today. Without a cache the ML team writes `--model=` directly, since the source is +theirs and Modelplane doesn't own it. A cache presents its weights to the engine one of two ways. **A path.** Most backends put the weights at a path (a PVC mount, an object-store CSI -mount, or a node-local cache). The ML team points the engine at it with the engine's own -flag, `--model` on vLLM or `--model-path` on SGLang, and the path reads the same -whatever transport staged it there. +mount, or a node-local cache). Modelplane sets `MODELPLANE_MODEL` to that path, and the ML +team passes it with the engine's own flag, `--model=$(MODELPLANE_MODEL)` on vLLM or +`--model-path=$(MODELPLANE_MODEL)` on SGLang. The path reads the same whatever transport +staged it there. **A loader plugin.** A few backends are engine loader plugins. NVIDIA ModelExpress provides first-class loaders for vLLM, SGLang, and TensorRT-LLM, and the Run:ai streamer @@ -222,8 +223,8 @@ plugs into vLLM and SGLang. Each needs the engine's own `--load-format` and a loader-capable image. The ML team writes that, because they chose the cache, and Modelplane wires the env and the cluster-side pieces. -Either way the flag that names the model belongs to the engine, and Modelplane stays out -of it. Placing only where the cache is pre-warmed +Either way the ML team writes the flag and Modelplane supplies the value it references, +never the flag itself. Placing only where the cache is pre-warmed makes whatever the ML team wrote valid wherever the replica runs. How the bytes reach the cluster, whether a shared filesystem, peer-to-peer distribution, or GPU-to-GPU streaming, is the platform team's concern and orthogonal to the command. The catalog of