What problem are you facing?
Modelplane passes the engine's command and args through untouched. This is deliberate: a new engine, a new flag, or a renamed flag needs no change to Modelplane, and a user who disagrees with our choices can override them. Most of the design depends on us not knowing or caring what the flags mean.
We already break this in one place. apply_cache_args appends --model=/mnt/models when a ModelDeployment references a cache and the engine has no explicit command:
|
def apply_cache_args(args: list[str], replica: v1alpha1.ModelReplica, engine: v1alpha1.Container) -> list[str]: |
|
"""Inject --model=<mount> for the turnkey vLLM path only. |
|
|
|
KServe used to inject this; nothing does now, and without it vLLM silently |
|
serves facebook/opt-125m. It is vLLM-specific (the `--model` flag), so it is |
|
skipped when: |
|
- no cache is referenced; |
|
- the engine brings its own `command` — a non-vLLM engine like SGLang owns |
|
its args and points at the mount with its own flag (`--model-path`), so |
|
injecting `--model` would hand it an unknown flag; or |
|
- the user already set `--model`. |
|
|
|
The cache *volume/mount* (cache_mounts) is added regardless of engine shape; |
|
only this arg injection is vLLM-specific. |
|
""" |
|
if not replica.spec.modelCacheRef or engine.command: |
|
return args |
|
if any(a == "--model" or a.startswith("--model=") for a in args): |
|
return args |
|
return [*args, f"--model={CACHE_MOUNT_PATH}"] |
That's a vLLM flag; SGLang spells it --model-path. So the turnkey cache path only works for vLLM, and it does nothing at all once a user writes their own command.
I expect more of these. Different ModelCache implementations want different flags to load weights efficiently: a PVC mount and something like ModelExpress need different loader flags. If a deployment always landed on one predictable, homogeneous environment the user could just write the right flags themselves. The trouble is if a fleet is heterogeneous, and a deployment's replicas can land on clusters that differ in ways that change the right flags: the cache implementation is one example. There's no single correct set to write, because it depends on where each replica ends up, which we only decide at scheduling.
How could Modelplane help solve your problem?
Put the flags we'd suggest in an environment variable on the engine container, and let the user decide whether to use them. We fill it in per replica, after scheduling; the user interpolates it into their command:
containers:
- name: engine
image: vllm/vllm-openai:v0.11.0
command: [/bin/sh, -c, "exec vllm serve $(MODELPLANE_CACHE_FLAGS) --served-model-name=qwen"]
The point is you don't have to know the right flags for your setup; we work them out. A new engine still works, because you can ignore the variable and write your own flags. A renamed vLLM flag doesn't block you. Neither does disagreeing with us. We never touch your command, we just offer a value you can use or ignore. MODELPLANE_LEADER_ADDRESS already works this way.
A few open questions:
How much to put in one variable. A single MODELPLANE_ENGINE_FLAGS with everything we'd suggest is the least to type, but you can't take the cache flags without taking the rest. Splitting by concern (MODELPLANE_CACHE_FLAGS first, more later) lets you pick, but you have to know which variable does what.
What the variable holds. Kubernetes expands $(VAR) in command and args but doesn't split on spaces, so the whole value arrives as a single argument. That means it only works inside a shell command (/bin/sh -c "... $(VAR) ..."), not a plain args: list, which is what the simplest deployments use. If we accept it's shell-only anyway, maybe a $(MODELPLANE_COMMAND) that expands to a whole suggested vllm serve ... (for example) line is a better fit than a bag of flags.
Predictability. You can't know what the computed flags will be ahead of time, so mixing them with your own flags is guesswork.
What problem are you facing?
Modelplane passes the engine's command and args through untouched. This is deliberate: a new engine, a new flag, or a renamed flag needs no change to Modelplane, and a user who disagrees with our choices can override them. Most of the design depends on us not knowing or caring what the flags mean.
We already break this in one place.
apply_cache_argsappends--model=/mnt/modelswhen a ModelDeployment references a cache and the engine has no explicit command:modelplane/functions/compose-model-replica/function/backends/base.py
Lines 104 to 123 in d3609d0
That's a vLLM flag; SGLang spells it
--model-path. So the turnkey cache path only works for vLLM, and it does nothing at all once a user writes their own command.I expect more of these. Different ModelCache implementations want different flags to load weights efficiently: a PVC mount and something like ModelExpress need different loader flags. If a deployment always landed on one predictable, homogeneous environment the user could just write the right flags themselves. The trouble is if a fleet is heterogeneous, and a deployment's replicas can land on clusters that differ in ways that change the right flags: the cache implementation is one example. There's no single correct set to write, because it depends on where each replica ends up, which we only decide at scheduling.
How could Modelplane help solve your problem?
Put the flags we'd suggest in an environment variable on the engine container, and let the user decide whether to use them. We fill it in per replica, after scheduling; the user interpolates it into their command:
The point is you don't have to know the right flags for your setup; we work them out. A new engine still works, because you can ignore the variable and write your own flags. A renamed vLLM flag doesn't block you. Neither does disagreeing with us. We never touch your command, we just offer a value you can use or ignore.
MODELPLANE_LEADER_ADDRESSalready works this way.A few open questions:
How much to put in one variable. A single
MODELPLANE_ENGINE_FLAGSwith everything we'd suggest is the least to type, but you can't take the cache flags without taking the rest. Splitting by concern (MODELPLANE_CACHE_FLAGSfirst, more later) lets you pick, but you have to know which variable does what.What the variable holds. Kubernetes expands
$(VAR)in command and args but doesn't split on spaces, so the whole value arrives as a single argument. That means it only works inside a shell command (/bin/sh -c "... $(VAR) ..."), not a plainargs:list, which is what the simplest deployments use. If we accept it's shell-only anyway, maybe a$(MODELPLANE_COMMAND)that expands to a whole suggestedvllm serve ...(for example) line is a better fit than a bag of flags.Predictability. You can't know what the computed flags will be ahead of time, so mixing them with your own flags is guesswork.