diff --git a/design/modelcache-hydration.md b/design/modelcache-hydration.md new file mode 100644 index 000000000..3c8cc1439 --- /dev/null +++ b/design/modelcache-hydration.md @@ -0,0 +1,379 @@ +# ModelCacheHydration + +**Status:** Draft +**Date:** July 2026 +**Author:** Dennis Ramdass + +This document proposes `ModelCacheHydration`, a per-cluster child of `ModelCache`, +and makes a cache's `clusterSelector` the authoritative footprint that Modelplane +pre-warms ahead of deployment. It builds on [modelcache.md](./modelcache.md) and +[design.md](./design.md), and addresses +[#210](https://github.com/modelplaneai/modelplane/issues/210) (decompose +`ModelCache`) and [#186](https://github.com/modelplaneai/modelplane/issues/186) +(footprint diverges from placement) together. + +## Summary + +`ModelCache` today does two jobs in one function: it fans a cache out to matched +clusters, and it runs each cluster's hydration lifecycle inline (the PVC, the +hydration `Job`, the token `Secret`, a four-phase machine, and the +drop-Job-after-Ready cleanup), threading a `cluster_name` through every method. +It's also the only fan-out that doesn't follow the `ModelDeployment` → +`ModelReplica` pattern. + +I propose two changes. + +**Split out `ModelCacheHydration`.** A per-cluster child that owns one cluster's +hydration. Pinned by `spec.clusterName`, it resolves its `InferenceCluster` and auth +`Secret`, composes the PVC, token `Secret`, and `Job`, runs the phase machine, and +drops the `Job`/`Secret` once Ready. + +**Pre-warm from the selector, and place where it's warmed.** A cache's `clusterSelector` +is the authoritative footprint. The platform team declares which clusters hold the +weights, and Modelplane hydrates them ahead of any deployment. A deployment that +references the cache runs only on those clusters. A deployment that references no cache +runs anywhere its own selector allows and loads from the source itself. No deployment +pays the download twice. + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: ModelCache +metadata: + name: kimi-k2 + namespace: ml-team +spec: + source: HuggingFace + huggingFace: + repo: moonshotai/Kimi-K2-Instruct + authSecret: + name: hf-token + sizeGiB: 1500 + clusterSelector: # the footprint: pre-warm these clusters + matchLabels: + modelplane.ai/tier: a100 +``` + +The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. +`ModelCacheHydration` is composed, never authored, the same as `ModelReplica`. + +Approving this means agreeing to both changes: the `ModelCacheHydration` +decomposition (#210), and a pre-warm-authoritative footprint (#186) where a +deployment that references a cache is placed only where that cache is pre-warmed. +Modelplane injects no engine flags for any of this. The sections below explain why. + +## Architecture + +Both fan-outs are per-cluster. `ModelCache` stages onto the clusters its +`clusterSelector` matches, and a deployment that references the cache is placed only +onto those clusters, so the cache is present wherever its replicas run. + +```mermaid +flowchart TD + subgraph ml["ML team creates"] + MD["ModelDeployment"] + MC["ModelCache"] + end + subgraph cp["Composed on the control cluster"] + MR["ModelReplica\n(one per placed cluster)"] + MCH["ModelCacheHydration\n(one per selected cluster)"] + end + subgraph wc["Workload cluster"] + WL["serving workload\n(Deployment / LeaderWorkerSet)"] + JOB["hydration Job"] + SEC["token Secret"] + PVC["PVC (weights)"] + end + MD -->|"schedules (within the cache footprint)"| MR + MC -->|"fan out per selected cluster"| MCH + MD -. modelCacheRef .-> MC + MR --> WL + MCH --> JOB + MCH --> SEC + MCH --> PVC + JOB -->|writes| PVC + WL -->|"mounts /mnt/models where staged"| PVC + classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; + class MCH new +``` + +## The per-cluster child + +`ModelCacheHydration` is a namespaced composite pinned to one cluster, the cache +analogue of `ModelReplica`. The name states the lifecycle the child owns, where +`ModelCacheReplica` would imply a copy of the parent. Its spec carries what one +cluster's hydration needs and nothing about fan-out: + +- **`clusterName`**, the cluster it stages onto. Pinned at creation; the parent + re-places only if the cluster disappears. +- **`huggingFace`**, the source, copied down verbatim (`repo`, `revision`, + `sizeGiB`). +- **`authSecret`**, the reference only (`name`, `key`, the cache's `namespace`), + never the token value. The child resolves it and propagates it to the workload + cluster itself, so a credential never lives in a CR spec. Same trust boundary as + today. +- **`cacheName`**, the parent `ModelCache` name, so the child reproduces the + stable PVC/Job/Secret names the serving side mounts by. + +Its status carries a `phase` (Pending/Hydrating/Ready/Failed), so the parent reads +`status.clusters[].phase` straight from the child. This is richer than +`ModelReplica`'s conditions-only status, to preserve the per-cluster phase +`ModelCache` shows today. + +The child is the current per-cluster body of `compose-model-cache` lifted out +whole: the PVC/Job/Secret manifests, the completion-marker skip, the +`_JOB_MANAGEMENT` cleanup, and the Ready-latch (now read from its own prior +`status.phase`). The hydration mechanism is unchanged. It just lives in one place. + +### Naming continuity + +The child keeps the names the monolith produced: PVC +`child_name("modelcache", , )`, Job `…,"hydrate"`, Secret `…,"auth"`. +Two things depend on this: + +1. The serving side mounts the PVC by that name (`base.cache_pvc_name` in + `compose-model-replica` is the same function), so the mount contract is + untouched. +2. On upgrade the child adopts the existing, already-Bound PVC instead of + provisioning a new one and re-downloading the weights. + +`child_name` truncates the prefix and appends a hash, so every composed name stays +a valid DNS label at or under 63 characters, including the child's own name with +the cluster folded in. + +## Pre-warm-authoritative footprint + +The divergence (#186): a cache's `clusterSelector` and a deployment's placement are +set independently. If a replica is scheduled onto a cluster the cache didn't stage +to, the PVC is missing and the pod fails to mount at runtime, with nothing visible +at apply time. + +The fix is to make the cache's `clusterSelector` the authoritative footprint and +place against it. The platform team declares which clusters hold the weights, +Modelplane pre-warms them ahead of any deployment, and a deployment that references +the cache is scheduled only onto them. + +Pre-warm is what makes a large model usable. Hydrating a 1.5TB cache PVC and loading +the model into a replica are two sequential copies. Paid on-demand, the first +deployment onto a new cluster waits for both. For a model the size of Kimi that's +roughly an hour in our testing. Pre-warm moves that hydration ahead of time, and the +platform team absorbs it once. An author onto a warmed cluster then waits only for the +load, roughly fifteen minutes. + +Because the footprint is the static selector, `ModelCache` does not watch replica +placement and does not recompose when replicas move. A `ModelCache` with no +`clusterSelector` stages nowhere, so Modelplane rejects it at apply time rather +than admit a cache no deployment can place against. + +### Two modes + +A deployment runs in one of two modes, split along the personas: the platform team +owns *what's cached and where*, the ML team owns *what to run*. + +**With a cache.** The ML team references a `ModelCache` and writes the engine command. +Modelplane places the deployment only on the clusters where the platform team pre-warmed +that cache, so the cache is present wherever the replica runs and the start command is +the same on every cluster. + +**Without a cache.** The ML team references no cache and writes the engine to load from +the source with its own token. Modelplane places it on any cluster its own selector +allows. This suits small or experimental models, and is slow for large ones, because +every replica downloads from the source. + +When a deployment references a cache but no cluster both matches its own selector and +holds that cache, it isn't placed, and the scheduler reports why rather than leaving it +Pending without a reason. + +### Modelplane injects env values, never engine flags + +The engine command stays the ML team's, and Modelplane never writes an engine flag. When +Modelplane owns a value the engine needs, it injects that value as an env var and the ML +team references it with Kubernetes `$(VAR)` expansion. With a cache Modelplane owns where +the weights are mounted, so it injects the mount path as `MODELPLANE_MODEL`. + +```yaml +engines: +- name: kimi-k2 + members: + - role: Standalone + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.11.0 + args: + - --model=$(MODELPLANE_MODEL) # Modelplane sets this to the cache mount +``` + +The ML team never hardcodes Modelplane's mount path, so Modelplane is free to change it. +It's the same mechanism `MODELPLANE_LEADER_ADDRESS` uses for a multi-node leader address +today. Without a cache the ML team writes `--model=` directly, since the source is +theirs and Modelplane doesn't own it. + +A cache presents its weights to the engine one of two ways. + +**A path.** Most backends put the weights at a path (a PVC mount, an object-store CSI +mount, or a node-local cache). Modelplane sets `MODELPLANE_MODEL` to that path, and the ML +team passes it with the engine's own flag, `--model=$(MODELPLANE_MODEL)` on vLLM or +`--model-path=$(MODELPLANE_MODEL)` on SGLang. The path reads the same whatever transport +staged it there. + +**A loader plugin.** A few backends are engine loader plugins. NVIDIA ModelExpress +provides first-class loaders for vLLM, SGLang, and TensorRT-LLM, and the Run:ai streamer +plugs into vLLM and SGLang. Each needs the engine's own `--load-format` and a +loader-capable image. The ML team writes that, because they chose the cache, and +Modelplane wires the env and the cluster-side pieces. + +Either way the ML team writes the flag and Modelplane supplies the value it references, +never the flag itself. Placing only where the cache is pre-warmed +makes whatever the ML team wrote valid wherever the replica runs. How the bytes reach +the cluster, whether a shared filesystem, peer-to-peer distribution, or GPU-to-GPU +streaming, is the platform team's concern and orthogonal to the command. The catalog of +backends is a separate design, and each backend has to present one of these two +contracts. + +### Hydrating before ready + +Placement puts a replica on a footprint cluster as soon as the cluster matches the +selector, which can be before its pre-warm has finished hydrating. Rather than fail +the mount, the replica gates readiness on that cluster's `ModelCacheHydration`, +holding at `Hydrating` until the PVC is Bound. It watches the `ModelCacheHydration` +object, not the PVC directly, so a future cache that doesn't use a PVC keeps the same +readiness contract. + +Gating can't be open-ended. If hydration fails (a bad token, a bad revision, +exhausted storage), the child reports `Failed`, the parent surfaces +`ArtifactReady=False` with reason `HydrationFailed`, and the gated replica fails +with that reason instead of sitting in `Hydrating`. The hydration `Job`'s +`backoffLimit` bounds retries, so a permanent failure stops and is reported rather +than retried forever. + +### Lifecycle + +The selector is the footprint. A cluster enters when it starts matching and drops out +when it stops. The parent stamps a child per matched cluster and reclaims the child +and its PVC once the cluster drops out. One guard: reclaim holds while a live replica +still uses the cache on that cluster, so it is never pulled out from under a running +pod. For the same reason a `ModelCache` with live referencing deployments can't be +deleted. A finalizer holds it until they leave. + +```mermaid +flowchart TD + A["clusterSelector matches cluster C"] --> B["Stamp ModelCacheHydration for (cache, C)"] + B --> C["Child composes PVC + hydration Job\nphase: Hydrating"] + C --> D{"PVC Bound and\nJob complete?"} + D -- no --> C + D -- yes --> E["Child phase: Ready"] + E --> F["Cache-referencing deployments\nplace only on C and mount there"] + F --> G{"C still matches\nthe selector?"} + G -- yes --> F + G -- no --> H{"Any live replica\nstill using the cache on C?"} + H -- yes --> F + H -- no --> I["C drops out\nchild + PVC reclaimed"] +``` + +## What the parent composes + +`compose-model-cache` becomes fan-out plus roll-up: + +- **Resolve** the clusters the `clusterSelector` matches. +- **Stamp** one child per matched cluster, named + `child_name("modelcache", ns, cache-name, cluster)`, copying down `huggingFace`, + `clusterName`, the `authSecret` reference, and `cacheName`. +- **Roll up** each child's `status.phase`/`Ready` into `status.clusters[]`, + `status.summary`, and the `ClustersMatched`/`ArtifactReady` conditions, marking a + child `READY_TRUE` only when observed Ready. +- **Shed** the per-cluster machinery to the child: `_wrap_remote`, the + PVC/Job/Secret builders, `derive_cluster_phase`, `_observed_status`, + `_resolve_auth_data`, the phase constants, and the hydration image. + +Registering the new kind takes the usual four touches, plus a schema regen: + +- `apis/modelcachehydrations/` with its XRD and Composition; +- a `functionNames` entry in `flake.nix`; +- a Tarball entry in `crossplane-project.yaml`; +- `nix run .#build` to regenerate the model. + +No `mrap.yaml` change is needed, because it composes +`objects.kubernetes.m.crossplane.io`, already activated. + +## Alternatives considered + +### Hydrate on-demand from placement + +Derive the footprint from where replicas are placed, so a cache follows a replica +onto a cluster new to the model. This was the earlier shape of this proposal and +reads as the most automatic. Three costs turned it down. + +- It needs [crossplane#7572](https://github.com/crossplane/crossplane/pull/7572) to + re-trigger `compose-model-cache` when a new replica appears, which isn't out until + Crossplane v2.4. +- The first deployment onto a new cluster still pays the hydrate and the load in + series: the hour-long wait pre-warm removes. +- A model carries its private weights and token onto whatever cluster it happens to + run on, rather than the clusters the platform team chose. + +### Load from the source off the footprint + +Let a cache-referencing deployment run anywhere and load from the source where the +cache is absent, so a cache never limits placement. This needs Modelplane to vary the +model reference per replica, which works for a plain mount but not for a loader-plugin +cache like ModelExpress, whose flag and image are invalid off the cache. Constraining +placement for every cache is simpler and keeps Modelplane out of the engine's flags. +A deployment that wants no cache references none. + +### Always populate a fixed path + +Make the start command uniform another way: require every deployment to use a cache, +or reintroduce `spec.model` and pre-fetch it before the engine starts. Requiring a +cache drops the lightweight no-cache path. A per-pod pre-fetch has no reuse, so it +costs a load plus an extra copy with none of a cache's benefit. Neither earns the +uniformity. + +### Drop `modelCacheRef`; derive the model from the deployment + +#186 floats dropping the explicit `ModelCache` and `modelCacheRef` and deriving +caching from the model the deployment declares. Nothing structured declares the model +today. It lives in opaque engine args. Automatic caching would need a structured model +source on the deployment and hydration keyed by model identity rather than cache name, +which also changes the serving mount contract. That is a larger user-facing change with +its own migration. This proposal keeps `modelCacheRef`, and identity-keyed sharing can +be added on top later, since the child already carries a source independent of how it +was requested. + +### A separate fleet reconciler kind + +A single hydration per cluster serves replicas across deployments, and Crossplane +composed resources are single-owner, so a `ModelReplica` can't compose a shared +hydration. A new fleet-scoped reconciler could own all hydrations. Reusing `ModelCache` +is simpler, since it is already the per-model, per-namespace resource that fans out per +cluster and keeps the namespace boundary where it is. A dedicated reconciler is worth +revisiting only if caching becomes fully deployment-derived. + +### Conditions-only child status + +Mirroring `ModelReplica`'s boolean conditions would collapse `ModelCache`'s per-cluster +`phase` (Pending/Hydrating/Ready/Failed) that `kubectl get modelcache` shows today. The +child carries a structured `status.phase` so the parent preserves it. + +### Deliver the decomposition and the footprint change separately + +The decomposition (#210) is a pure refactor and could merge first. The two are kept +together because designing the child's ownership once against the final footprint model +is easier than doing it twice. + +## Interaction with related issues + +- **#189 (constrain placement):** kept, not superseded. #189's rule (a + cache-referencing deployment places only within the cache footprint) is the design + here; pre-warm makes the cache's `clusterSelector` the footprint that rule uses. +- **#115 (Modelplane-owned hydration image):** the Job builder moves to the child; + #115 becomes a localized image swap there. +- **#281 (multiple models per deployment):** the child stays single-source (one + repo, one PVC). #281 fans out per source later. +- **#204 (EFS load slow), #72 (KVOffloadTier), #71 (routing affinity):** + orthogonal. These are storage read performance and KV or prefix cache, not + model-weight staging. +- **#341 (ImageCache):** a different problem, pre-pulling the container image into + each node's local image store, which a PVC-based cache cannot do (the Kubelet + starts containers from node-local image layers, not a mounted volume). It shares + only the per-cluster fan-out shape, not the mechanism.