Add a per-cluster serving stack: Standard or Dynamo by negz · Pull Request #406 · modelplaneai/modelplane

negz · 2026-08-21T00:35:08Z

Description of your changes

Towards #111.
Fixes #407.

Modelplane composed one serving stack on every cluster: a LeaderWorkerSet for multi-node engines, with no alternative. InferenceCluster.spec.stack now selects Standard (the default, unchanged) or Dynamo, which swaps in NVIDIA's components: Grove and the KAI Scheduler for gang scheduling, and a ModelExpress server for peer-to-peer weight transfer. The choice is per-cluster and immutable.

One ModelDeployment can run on both: a Standalone engine is a Deployment, a Leader/Worker gang a LeaderWorkerSet or a Grove PodCliqueScalingGroup. $(MODELPLANE_LEADER_ADDRESS) resolves on both. That needs Grove v0.1.0-alpha.12-rc2, the first release carrying ai-dynamo/grove#753. $(MODELPLANE_RANK) has no Dynamo equivalent until ai-dynamo/grove#755 is merged, so a gang command must compute its rank from GROVE_PCLQ_POD_INDEX.

ModelExpress is inferred from the stack. A Dynamo cluster runs one metadata server brokering which replica holds a model in GPU memory. Weights move peer-to-peer. An engine opts in with `--load-format modelexpress. The first replica loads from the cache PVC and publishes itself, later replicas pull from a peer's GPU over the fabric. The injected client env is inert otherwise, so the same deployment runs on a Standard cluster.

A ModelCache used to hydrate a flat model directory, which vLLM can only load by path, so the engine command depended on whether the model was cached: --model=/mnt/models with a cache, --model=<repo> without. Modelplane papered over the simplest case by injecting --model=<mount>, which goes against our design philosophy. A HuggingFace source now stages weights in HuggingFace's disk cache layout with the engine's HF_HUB_CACHE pointed at the mount, so --model=<repo> resolves against the staged snapshot and the command is the same either way.

# InferenceCluster
spec:
  stack: Dynamo   # new; omit for the default Standard

Validated end to end on EKS with a Standard and a Dynamo cluster.
I have:

Read and followed Modelplane's contribution process.
Run nix flake check (or ./nix.sh flake check) and made sure it passes.
Added or updated tests covering any composition function changes.
Signed off every commit with git commit -s.

Copilot

Pull request overview

Adds per-cluster selection between the existing Standard serving stack and a Dynamo stack using Grove, KAI Scheduler, and ModelExpress.

Changes:

Adds stack and cache-distribution API fields.
Composes Grove/KAI workloads and shared ModelExpress infrastructure.
Updates scheduling, tests, schemas, examples, and documentation.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`schemas/python/models/ai/modelplane/modelreplica/v1alpha1.py`	Adds cache distribution metadata.
`schemas/python/models/ai/modelplane/modeldeployment/v1alpha1.py`	Updates workload documentation.
`schemas/python/models/ai/modelplane/modelcache/v1alpha1.py`	Adds distribution selection.
`schemas/python/models/ai/modelplane/infrastructure/servingstack/v1alpha1.py`	Adds Dynamo versions and storage configuration.
`schemas/python/models/ai/modelplane/inferencecluster/v1alpha1.py`	Adds per-cluster stack selection.
`schemas/.lock.json`	Updates generated schema lock.
`functions/compose-serving-stack/tests/test_fn.py`	Tests Dynamo stack composition.
`functions/compose-serving-stack/function/modelexpress_crds.yaml`	Vendors ModelExpress CRDs.
`functions/compose-serving-stack/function/fn.py`	Installs Grove, KAI, and ModelExpress.
`functions/compose-model-replica/tests/test_backends.py`	Tests Grove and ModelExpress backends.
`functions/compose-model-replica/function/routing.py`	Normalizes Grove routing templates.
`functions/compose-model-replica/function/fn.py`	Selects backend from cluster stack.
`functions/compose-model-replica/function/backends/native.py`	Injects ModelExpress settings.
`functions/compose-model-replica/function/backends/llmd.py`	Adds ModelExpress support to LWS.
`functions/compose-model-replica/function/backends/grove.py`	Implements the Grove backend.
`functions/compose-model-replica/function/backends/dynamo.py`	Removes the former Dynamo stub.
`functions/compose-model-replica/function/backends/base.py`	Adds shared Grove and ModelExpress helpers.
`functions/compose-model-deployment/tests/test_scheduling.py`	Tests Dynamo-only placement.
`functions/compose-model-deployment/tests/test_fn.py`	Tests distribution propagation.
`functions/compose-model-deployment/function/scheduling.py`	Restricts new ModelExpress placements.
`functions/compose-model-deployment/function/fn.py`	Propagates cache distribution.
`functions/compose-model-cache/tests/test_fn.py`	Tests ModelExpress hydration.
`functions/compose-model-cache/function/fn.py`	Implements ModelExpress cache staging.
`functions/compose-inference-cluster/tests/test_fn.py`	Updates ServingStack expectations.
`functions/compose-inference-cluster/function/fn.py`	Relays stack and storage settings.
`docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt`	Adds Dynamo terminology.
`docs/manifests/getting-started/prerequisites.yaml`	Updates stack examples.
`docs/manifests/examples/qwen3-coder/model-deployment.yaml`	Clarifies Standard gang behavior.
`docs/manifests/concepts/model-deployment-multinode.yaml`	Documents both gang backends.
`docs/manifests/concepts/model-cache.yaml`	Shows ModelExpress selection.
`docs/content/overview/how-it-works.md`	Describes per-cluster stacks.
`docs/content/overview/faq.md`	Updates multi-node guidance.
`docs/content/models/model-cache.md`	Documents ModelExpress loading.
`docs/content/getting-started/build-the-platform.md`	Mentions Dynamo stack selection.
`docs/content/architecture/_index.md`	Updates workload architecture.
`apis/servingstacks/definition.yaml`	Defines Dynamo stack configuration.
`apis/modelreplicas/definition.yaml`	Defines propagated distribution.
`apis/modeldeployments/definition.yaml`	Updates workload API descriptions.
`apis/modelcaches/definition.yaml`	Defines ModelExpress distribution.
`apis/inferenceclusters/definition.yaml`	Defines cluster stack selection.

Suppressed comments (1)

apis/modeldeployments/definition.yaml:243

For Grove, copies does not map to PodCliqueSet.spec.replicas: the backend fixes that field at 1 and maps copies to the PodCliqueScalingGroup's replicas. The description should also retain LeaderWorkerSet for Standard clusters so users understand the actual scaling target.

                                (spec.replicas), never by varying copies. Maps to the
                                composed Deployment's or PodCliqueSet's replica
                                count. Defaults to 1.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 7 comments.

Suppressed comments (2)

functions/compose-model-replica/function/backends/base.py:194

MX_MODEL_REVISION is not unique per cache: ModelCacheRef is namespace-local, so team-a/qwen and team-b/qwen may contain different models while both publish the same /mnt/models + qwen source identity to the cluster-wide ModelExpress server. The PVC path already handles this exact collision by qualifying the cache name with the control-plane namespace. Use the same namespace-qualified identity here to prevent a replica from selecting a peer that serves another tenant's weights.

        {"name": "MX_MODEL_REVISION", "value": ref.name},

functions/compose-model-replica/function/routing.py:17

The Standard multi-node path still uses the llm-d backend and this module explicitly handles LeaderWorkerSet below. Keep it in the backend list; otherwise the module header incorrectly implies routing only supports native and Grove workloads.

The workload backends (native, Grove) compose engines only; this layer

dennis-upbound

Reviewed the Dynamo stack and the ModelExpress bundle. Roughly by severity. I've marked what I verified against the diff or upstream and what needs your confirmation, since a couple of the Grove points rest on reading alpha.6 rather than running it.

`engine.copies > 1` on a gang never becomes ready

grove.py maps engine.copies onto the PodCliqueScalingGroup replica count, but a gang pod addresses its leader as $(GROVE_PCS_NAME)-$(GROVE_PCS_INDEX)-gang-0-leader-0, with the scaling-group index hardcoded to 0. Grove alpha.6 injects PCS_NAME, PCS_INDEX, PCLQ_NAME, PCLQ_POD_INDEX and HEADLESS_SERVICE, and no PCSG_INDEX. So with copies: 2, copy 1's workers dial copy 0's leader, copy 1 never forms, availableReplicas never reaches spec.replicas, and the ModelReplica stays unready. Nothing rejects copies > 1 for a gang on Dynamo and the XRD allows up to 64.

The hardcoded index is in the diff. The absence of PCSG_INDEX in alpha.6 is worth double-checking before acting. If it holds, a CEL rule rejecting copies > 1 for a Leader/Worker engine on a Dynamo cluster fails closed instead of hanging. Relatedly, the comment above the mapping says each copy gets its own group-wide pod index, which the module docstring elsewhere says the pinned release cannot provide.

`HF_HUB_CACHE` is not inert, and the layout it points at cannot satisfy it

modelexpress_env() injects HF_HUB_CACHE=/mnt/models whenever the cluster is Dynamo and the replica references a cache, from both the native and Grove backends, with no opt-in check. The docstring says the bundle is inert unless the engine opts in. That holds for the MX_* variables, but HF_HUB_CACHE is stock huggingface_hub and takes effect regardless.

Two consequences. Mount-path hydration still runs hf download --local-dir, so /mnt/models is a flat snapshot rather than a hub cache tree, which is the LocalEntryNotFoundError in #407. And any engine that fetches something by repo id at startup, such as the kimi-k2 tokenizer the docstring itself cites, now reads and writes the shared RWX PVC, putting huggingface_hub .lock files on EFS or NFS contended by every pod. filelock semantics over NFS are unreliable.

This is #407 reached from the other direction, and the docstring's "deliberately no HF_HUB_OFFLINE" is the opposite of what #407 proposes. Landing #407 first, or gating the variable on the engine opting in, resolves it either way. Verified against the diff.

`minAvailable` is immutable but derived from mutable fields

Grove's PodCliqueSet webhook rejects changes to minAvailable on both cliques and scaling groups, and the backend derives both from worker.nodes and engine.copies, which the ModelDeployment XRD lets a user edit freely. Bumping nodes from 1 to 3 leaves provider-kubernetes failing its apply indefinitely with an admission error while the old gang keeps serving, so the replica silently stops converging rather than reporting anything. An immutability rule on the XRD, or a condition when apply is rejected, would surface it. I have not verified the webhook myself.

The shared ModelExpress PVC has no capacity accounting

modelExpressCacheSize defaults to 100Gi and sizes one PVC for every ModelExpress cache on the cluster, while each ModelCache still requires huggingFace.sizeGiB, which does nothing in this mode. Three 60Gi models on a default Dynamo cluster overflow, and the failure arrives as a hydration Job hitting ENOSPC with the cache reporting Failed and nothing pointing at the cluster-level size as the cause. The XRD description does tell the operator to size for the combined footprint, so this is about diagnosis rather than docs: a condition naming the shared PVC when hydration fails on space would make it findable.

`modelExpressCacheSize` is unvalidated where the analogous field is bounded

type: string, minLength: 1, maxLength: 32, no pattern, with the quantity contract carried only in the description. "500 Gi" or "500GB" passes admission and then fails when the PVC reaches the workload cluster, surfacing as an Object sync error on the ServingStack rather than a rejection on the InferenceCluster the user actually edited. huggingFace.sizeGiB is an integer with bounds for the same concept, and three XRDs in the repo already use pattern, so a quantity pattern would be consistent with what is there.

The pinned server image is two releases behind, in the area of the workaround

0.4.1 is dated 2026-06-30. v0.5.0 (2026-08-01) lands consistent cache accounting, where the server verifies disk on a cache hit and touches downloaded hits so recency is tracked, plus provider-scoped registry keys to prevent false cache hits. v0.5.1 (2026-08-20) is small and self-described as touching nothing in the loader, registry or P2P plane beyond a TensorRT-LLM backport and a protobuf 6 client-compatibility fix.

That matters here because --strategy direct exists to work around server-delegating strategies no-oping on the pinned image, and the 0.5.0 fixes sit in exactly that area, so the workaround may be version-specific rather than permanent. The protobuf 6 fix also lands on modelexpress-cli, which the hydration Job runs out of the server image. A bump needs the vendored modelexpress_crds.yaml moved with it, and there is nothing at the image constant noting that coupling. Worth linking ai-dynamo/modelexpress#72 as well, a closed path-resolution bug where the server looked for hub/<org>/<name> while HuggingFace writes hub/models--<org>--<name>, in case the accept-then-idle behaviour is the same one.

Questions rather than findings

Why does the hydration Job pass --endpoint when --strategy direct has the CLI fetch the weights itself? If it is not needed, dropping it removes hydration's dependency on server health entirely.

With --strategy direct, does anything register the staged model with the server? The Role grants create and update on modelmetadatas and modelcacheentries and the metadata backend is Kubernetes, so the server writes CRs, but the CLI does not register. If the server only discovers on first request, the CRDs and RBAC may be unused for pre-staged models.

The server Deployment gates on ProviderConfigs and the StorageClass but not on its CRDs being established, and Crossplane applies desired resources concurrently, so it can be listening before the CRDs exist. A TCP readiness probe passes at that point and the Service gets endpoints. A gRPC probe would notice the difference.

The server runs one replica with no resource requests, so it is BestEffort and first out under node memory pressure, in the path of every load. NVIDIA documents the loader falling back to storage when no compatible peer is available, but not when the server itself is unreachable, so it is unclear whether a restart degrades or breaks an in-flight load.

Minor

_GROVE_PCS_NAME_MAX = 24 looks conservative. Grove sums template names only (pcs + pcsg + pclq), so with gang and worker the budget is 35, and names truncate and hash-suffix earlier than needed.
engine_name's docstring now says it is for the native Deployment, but llmd.py still calls it, and the removed paragraph was the only record of why the name must differ from the replica name.
llmd.py names its parameter _stack where the Backend protocol declares stack, so a keyword call breaks that backend alone. fn.py only happens to call positionally.
spec.versions.leaderWorkerSet moves to spec.standard.leaderWorkerSet with no migration, so an existing ServingStack that pinned a non-default LWS chart has the field pruned and reverts to the default.
The InferenceCluster XRD has a stray run of spaces near "picker. Dynamo", and the committed generated schema still carries the older wording, so a regen will produce a diff.
KAI v0.16.8 does create default-parent-queue and default-queue on install, so "neither Grove nor KAI creates one" is not right, even though owning a modelplane queue still is.
model-cache.md links "Dynamo" to the inference-cluster page, which does not mention Dynamo, spec.stack or Grove.

Copilot

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

functions/compose-inference-cluster/function/fn.py:496

The new non-default propagation path is untested: every updated expectation in compose-inference-cluster/tests/test_fn.py exercises the default Standard value, so a regression that drops or rewrites Dynamo would still pass. Add a composition test with spec.stack: Dynamo and assert that the desired ServingStack.spec.stack remains Dynamo, as required for composition-function changes by CONTRIBUTING.md:118-131.

        spec = ssv1alpha1.Spec(secrets=backend_secrets, stack=self.xr.spec.stack)

Copilot

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated no new comments.

dennis-upbound · 2026-08-27T18:04:29Z

The Grove/KAI/ModelExpress composition looks good — dispatch, the _serving_pod_templates normalization, KAI queue teardown, and readiness gating all hold up, and the affected suites are green. My comments are almost all on the ModelCache layout change that rode along.

Bump the hydration marker. compose-model-cache/function/fn.py:123 is still .modelplane-hydrated, so every already-hydrated PVC skips re-staging and keeps the flat tree. The engine then gets HF_HUB_CACHE=/mnt/models + --model=<repo>, misses, and goes to the network — and for the gated qwen3-coder cache the engine carries no HF_TOKEN, so it fails to start rather than falling back. .modelplane-hydrated-v2 closes it.
A pinned revision now double-downloads. Hydration stages under refs/<revision>, but cache_env (backends/base.py:129) sets only HF_HUB_CACHE, so --model=<repo> resolves at main and pulls the model again onto the same PVC. --model=/mnt/models used to be revision-agnostic; with sizeGiB sized for one copy plus headroom, the second copy fills the volume for everything sharing it.
Dropping the --model injection fails silently. The old apply_cache_args docstring said it: "without it vLLM silently serves facebook/opt-125m." Cache + no command + no explicit --model now serves the wrong model. Right call to remove the papering-over, but it wants a release note or a validation that rejects the now-broken shape.
The MODELPLANE_RANK portability claim isn't enforced. compose-model-deployment never reads stack, so a mixed fleet can place an existing gang deployment on a Dynamo cluster; the unresolved $(MODELPLANE_RANK) reaches vLLM as a literal and argparse dies. That contradicts design/serving-stacks.md:163 — worth a scheduler constraint or a status warning until grove#755.
PCSG minAvailable == copies cascades (grove.py:207, as your own comment notes): one stuck gang takes out every healthy copy once terminationDelay elapses. Per-gang atomicity already comes from the clique-level minAvailable, so minAvailable: 1 on the group avoids the cascade — at the cost of reporting available with only some copies up.
IPC_LOCK is unconditional for cache-referencing engines on Dynamo, including the TCP fallback that doesn't need it; per its own docstring that breaks any restricted-PSS namespace. Gating on the member actually claiming a fabric device would narrow it.

Nits: HF_HUB_CACHE is a cache root, so kimi-k2's --tokenizer=moonshotai/Kimi-K2-Instruct will download onto the shared PVC from both pods, racing on .locks over RWX. And the rank gap is grove#755 in the code but #754 in your comment above.

Multi-node serving installed a LeaderWorkerSet on every cluster, with no alternative. This adds a per-cluster choice: InferenceCluster.spec.stack selects Standard (the default) or Dynamo, and compose-inference-cluster propagates it to the cluster's ServingStack. Standard is today's Modelplane-composed stack: compose-serving-stack fronts the workload with Gateway API and an endpoint picker. Dynamo installs NVIDIA's components instead: Grove and the KAI Scheduler for multi-node gang scheduling (KAI needs its Queue hierarchy and teardown-ordering Usages), and a ModelExpress server for peer-to-peer weight distribution. The ModelExpress server is metadata-only: it brokers which replica holds a model in GPU memory so peers can transfer weights directly, and never handles the weight bytes itself. It needs no shared cache volume, so it's one server per cluster in the serving namespace, backed by an emptyDir. The Grove workload and the engine-side ModelExpress wiring land in the next commit. spec.stack is immutable: a cluster's serving components are foundational, so switching stacks means recreating the cluster rather than swapping them under running workloads. Because the choice is per-cluster, a fleet can adopt Dynamo one cluster at a time. Signed-off-by: Nic Cope <nicc@rk0n.org>

On a Dynamo cluster a Leader/Worker engine composes a Grove PodCliqueSet holding one PodCliqueScalingGroup (a "gang") of a leader clique and a worker clique, rather than a LeaderWorkerSet; engine.copies is the scaling group's replica count. select_backend dispatches on the target cluster's stack: native for a Standalone engine, LeaderWorkerSet for Standard, Grove for Dynamo. The cliques carry no startsAfter. A multi-node leader only becomes Ready once its workers have joined to form the parallel group, so gating the workers on the leader's readiness - which Grove's startsAfter does - deadlocks. The scaling group co-schedules the gang regardless, and a worker reaches the leader through its stable DNS name. The released Grove this targets can't back the backend-neutral MODELPLANE_LEADER_ADDRESS / MODELPLANE_RANK the LeaderWorkerSet backend injects (it leaves GROVE_PCSG_* empty), so a Grove gang engine's command references Grove's GROVE_PCS_* vars directly, with a TODO to switch once grove#753 (merged, unreleased) and grove#755 (open) ship. Any engine on a Dynamo cluster that references a ModelCache also gets the ModelExpress client wiring: the shared server's address, P2P metadata, the HuggingFace cache env, and IPC_LOCK. A command that opts into --load-format modelexpress then loads the first replica from the cache PVC and every later one from a peer's GPU over the cluster fabric (NIXL over RDMA/EFA), falling back to the PVC when there's no peer or no fabric. This is inferred from the cluster's stack, not a ModelCache field: a cache stays a plain per-cache PVC that works on either stack, so the same deployment is portable between them. It applies to a Standalone engine - several replicas of one deployment are as valid a peer set as a gang - as much as to a Grove gang. Modelplane injects no --load-format flag; the engine command decides whether to use it. Signed-off-by: Nic Cope <nicc@rk0n.org>

The docs described only LeaderWorkerSet for multi-node serving and didn't cover the Dynamo stack. This documents the per-cluster spec.stack choice across the architecture, how-it-works, and cluster concept pages; keeps the multi-node ModelDeployment example on the default Standard form (MODELPLANE_LEADER_ADDRESS / MODELPLANE_RANK) with a note on the interim Grove form; and documents ModelExpress P2P on the model-cache page, where a Dynamo cluster runs the server and injects the client env and an engine opts in with --load-format modelexpress. Vale's vocabulary gains the new terms. Signed-off-by: Nic Cope <nicc@rk0n.org>

An engine command was coupled to whether its model was cached. A ModelCache hydrated its PVC with `hf download --local-dir`, writing a flat model directory that vLLM can only load by path, so a cached deployment had to say `--model=/mnt/models` while an uncached deployment of the same model said `--model=Qwen/Qwen2.5-7B-Instruct`. Adding or removing a cache meant rewriting the engine command, and one model spec couldn't move between the two. Modelplane papered over this for the simplest case by injecting `--model=<mount>` into a vLLM engine that brought no command of its own. That injection was a wart: it is engine-specific in a package that is otherwise unopinionated about the engine, it silently did nothing for an engine that brought its own command, and it made Modelplane responsible for naming the model - which belongs to the engine command, like every other flag. This hydrates a HuggingFace source into HuggingFace's own cache layout (models--<org>--<name>/snapshots/<sha>/) by pointing the hydration Job's HF_HUB_CACHE at the mount, and sets the same HF_HUB_CACHE on every engine that references a cache. `--model=<repo>` then resolves against the pre-staged snapshot rather than pulling from HuggingFace, so the engine command is identical with or without a cache, and the `--model` injection is gone. HF_HUB_OFFLINE is deliberately not set. Resolution doesn't need it - HuggingFace finds the cached snapshot when it can reach the Hub and falls back to it when it can't - and forcing it would break an engine that fetches a separate repo at startup, like kimi-k2's independently gated tokenizer. The layout is a property of the source, not of the mount: a cache is always mounted at the same path, and Modelplane injects whatever env makes that source's native addressing resolve against it. A future non-HuggingFace source would stage in whatever layout its own tooling expects, and its engines would name the model the way that source names it. Fixes modelplaneai#407. Signed-off-by: Nic Cope <nicc@rk0n.org>

The Standard/Dynamo serving stack landed with no design record here. The proposal it was built from was written with NVIDIA's Dynamo team as its audience, so it needs adjusting for Modelplane's. This copies it in verbatim, before any edit, so the next commit's diff shows exactly what changed. Towards modelplaneai#111. Signed-off-by: Nic Cope <nicc@rk0n.org>

The design was written with NVIDIA's Dynamo team as its reader, and described a ModelExpress integration we since decided against. ModelExpress does two things: it runs a cache service that downloads and deduplicates weights, and it moves weights GPU-to-GPU over NIXL. The design took both, with the server owning a shared cache PVC and hydrating it through its own registry. We take only the transfer, so a ModelCache stays a plain per-cache PVC on either stack and the deployment referencing it stays portable. The obvious question for a Modelplane reader is what changed since modelplaneai#15 added a Dynamo backend and modelplaneai#44 removed it over the lowest common denominator problem. A new section answers it: unopinionated deployments turned the API into a contract a stack either meets or doesn't, so the burden sits on the stack rather than the API. Grove and kai-scheduler meet it, a DGD could, KServe couldn't. Gang scheduling is the one capability with credible alternatives, so it gets an entry under alternatives considered. Volcano and Kueue can both gang-schedule a LeaderWorkerSet, and we could add one to the Standard stack. That closes one gap and leaves the ones nothing else closes. The asks to NVIDIA are now what we need from Dynamo, both already in flight upstream: ai-dynamo/dynamo#10835 for the sidecar and ai-dynamo/dynamo#12696 for per-role pod specs and optional flag injection. The document no longer proposes an API for the second, since Dynamo is designing one. Structure follows the other designs here: summary, background, goals, proposal, future improvements, alternatives considered. Towards modelplaneai#111. Signed-off-by: Nic Cope <nicc@rk0n.org>

negz · 2026-08-28T05:40:55Z

@dennis-upbound thanks!

minAvailable. Fixed, and you were right about the mechanism though not the reason: per-gang atomicity comes from PodGang membership, not the clique-level minAvailable. Grove puts every replica below the threshold into one shared PodGang, so matching it to copies is exactly what fuses them. It's 1 now. The cost is that a replica reports available once one copy serves, since availableReplicas is the only signal Grove populates — status.podGangStatuses is on the type but nothing writes it. Noted in the code.
IPC_LOCK. Agreed it isn't inert. Gating on a claimed fabric device isn't available though: nothing marks a device as fabric, and ModelReplica carries no fabric at all. The signal would be the member's pool declaring one, which means threading the cluster into the helper. Left as a conditional TODO rather than guessing at DeviceClass names, since an unrecognised one would silently drop the capability and quietly fall back to disk.
MODELPLANE_RANK. Real, and it's A multi-worker gang isn't portable across serving stacks until Grove exposes a gang-wide pod index #418. Treating it as a known shortcoming until feat(operator): expose PCSG-wide pod index ai-dynamo/grove#755 lands rather than adding a shim — the design is for it to be portable, we just can't land that half yet.
Marker bump. Not doing it. Nothing is running Modelplane at v0.3, so there's no staged cache to migrate, and the bump wouldn't achieve it anyway: a cluster whose cache reports Ready doesn't compose the Job at all, so the marker is never checked. I had -hub in for a while and removed it for that reason.
Pinned revision. Documented rather than fixed: the engine has to name the revision its cache staged. Modelplane can't fill it in without going back to injecting --model, which is what this change removes.
--model removal. Intentional, and in the PR's breaking-changes note.

On the nits: the tokenizer landing on the shared PVC is real and called out on cache_env — HF_HUB_CACHE is a cache root, so any repo-id fetch goes there. And you're right, I cited #754 above where the code says #755.

dennis-upbound

lgtm!

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 00:36 View deployment

negz requested a balanced review from Copilot August 21, 2026 00:55

Copilot started reviewing on behalf of negz August 21, 2026 00:55 View session

Copilot AI reviewed Aug 21, 2026

View reviewed changes

negz force-pushed the clique-bait branch from 81d3eab to 706e83f Compare August 21, 2026 03:59

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 04:00 View deployment

negz force-pushed the clique-bait branch from 706e83f to bc1de84 Compare August 21, 2026 04:17

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 04:18 View deployment

negz force-pushed the clique-bait branch from bc1de84 to 35ff7b1 Compare August 21, 2026 19:54

negz changed the title ~~Add a per-cluster Dynamo serving stack (Grove, KAI, ModelExpress)~~ Add a per-cluster serving stack: Standard or Dynamo Aug 21, 2026

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 19:55 View deployment

negz force-pushed the clique-bait branch from 35ff7b1 to 9160cb7 Compare August 21, 2026 20:10

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 20:11 View deployment

negz commented Aug 21, 2026

View reviewed changes

Comment thread apis/servingstacks/definition.yaml Outdated

Comment thread apis/servingstacks/definition.yaml Outdated

Comment thread docs/manifests/getting-started/prerequisites.yaml Outdated

negz force-pushed the clique-bait branch from 9160cb7 to 94378bf Compare August 21, 2026 20:34

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 20:36 View deployment

negz requested a balanced review from Copilot August 21, 2026 20:41

Copilot started reviewing on behalf of negz August 21, 2026 20:42 View session

Copilot AI reviewed Aug 21, 2026

View reviewed changes

negz commented Aug 21, 2026

View reviewed changes

negz force-pushed the clique-bait branch from 94378bf to a3a11bc Compare August 21, 2026 22:36

vercel Bot deployed to Preview – modelplane-docs August 21, 2026 22:37 View deployment

negz marked this pull request as ready for review August 22, 2026 04:16

Copilot AI review requested due to automatic review settings August 22, 2026 04:16

Copilot started reviewing on behalf of negz August 22, 2026 04:17 View session

Copilot AI reviewed Aug 22, 2026

View reviewed changes

dennis-upbound reviewed Aug 24, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 26, 2026 20:02

negz force-pushed the clique-bait branch from a3a11bc to 58273ff Compare August 26, 2026 20:02

Copilot started reviewing on behalf of negz August 26, 2026 20:03 View session

Copilot started reviewing on behalf of negz August 26, 2026 21:43 View session

vercel Bot deployed to Preview – modelplane-docs August 26, 2026 21:44 View deployment

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread functions/compose-model-replica/function/backends/base.py Outdated

negz force-pushed the clique-bait branch from cdce3f5 to cc42dfb Compare August 26, 2026 22:11

Copilot AI review requested due to automatic review settings August 26, 2026 22:11

Copilot started reviewing on behalf of negz August 26, 2026 22:12 View session

vercel Bot deployed to Preview – modelplane-docs August 26, 2026 22:12 View deployment

Copilot AI reviewed Aug 26, 2026

View reviewed changes

negz force-pushed the clique-bait branch from cc42dfb to 68a859c Compare August 26, 2026 22:36

vercel Bot deployed to Preview – modelplane-docs August 26, 2026 22:37 View deployment

vercel Bot deployed to Preview – modelplane-docs August 27, 2026 02:28 View deployment

negz force-pushed the clique-bait branch from d716155 to ef5c77e Compare August 27, 2026 02:53

vercel Bot deployed to Preview – modelplane-docs August 27, 2026 02:54 View deployment

negz force-pushed the clique-bait branch from ef5c77e to e21da96 Compare August 27, 2026 02:59

vercel Bot deployed to Preview – modelplane-docs August 27, 2026 03:01 View deployment

negz mentioned this pull request Aug 27, 2026

Omitting a member's template.spec crashes compose-model-replica instead of failing admission #416

Open

negz added 6 commits August 27, 2026 22:12

negz force-pushed the clique-bait branch from e21da96 to 2c0460d Compare August 28, 2026 05:34

vercel Bot deployed to Preview – modelplane-docs August 28, 2026 05:35 View deployment

negz mentioned this pull request Aug 28, 2026

A multi-worker gang isn't portable across serving stacks until Grove exposes a gang-wide pod index #418

Open

negz requested a review from dennis-upbound August 28, 2026 05:46

dennis-upbound approved these changes Aug 31, 2026

View reviewed changes

negz merged commit 960c645 into modelplaneai:main Aug 31, 2026
6 checks passed

Uh oh!

Conversation

negz commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of your changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dennis-upbound left a comment

Choose a reason for hiding this comment

engine.copies > 1 on a gang never becomes ready

HF_HUB_CACHE is not inert, and the layout it points at cannot satisfy it

minAvailable is immutable but derived from mutable fields

The shared ModelExpress PVC has no capacity accounting

modelExpressCacheSize is unvalidated where the analogous field is bounded

The pinned server image is two releases behind, in the area of the workaround

Questions rather than findings

Minor

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

dennis-upbound commented Aug 27, 2026

Uh oh!

negz commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennis-upbound left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

negz commented Aug 21, 2026 •

edited

Loading

`engine.copies > 1` on a gang never becomes ready

`HF_HUB_CACHE` is not inert, and the layout it points at cannot satisfy it

`minAvailable` is immutable but derived from mutable fields

`modelExpressCacheSize` is unvalidated where the analogous field is bounded

negz commented Aug 28, 2026 •

edited

Loading