Skip to content

A multi-worker gang isn't portable across serving stacks until Grove exposes a gang-wide pod index #418

Description

@negz

What problem are you facing?

A Leader/Worker engine with more than one worker needs each worker pod to get a distinct rank, and the two serving stacks added in #406 expose that differently. On Standard Modelplane injects $(MODELPLANE_RANK), 0 on the leader and 1..N on the workers, so one worker template covers every pod. Grove numbers pods within a clique rather than across the gang, so on Dynamo the command has to do the arithmetic itself:

# Standard
- --node-rank=$(MODELPLANE_RANK)
# Dynamo
- --node-rank=$$((GROVE_PCLQ_POD_INDEX + 1))

So the same ModelDeployment can't move between stacks unmodified, which is the portability that PR's design aims for. The leader address already behaves that way on both.

Nothing stops the mismatch, either. compose-model-deployment doesn't read spec.stack, so a deployment authored for Standard can be placed on a Dynamo cluster, where $(MODELPLANE_RANK) is never injected and reaches the engine as a literal string. A single-worker gang is fine, since you can hardcode 0 on the leader and 1 on the worker.

How could Modelplane help solve your problem?

ai-dynamo/grove#755 exposes a PodCliqueScalingGroup-wide pod index. Once that's in a release the Grove backend can alias MODELPLANE_RANK to it the same way it already aliases MODELPLANE_LEADER_ADDRESS to the PCSG name and index, and the gang's coordination env reads identically on both stacks.

I'd treat it as a known shortcoming until then. A shim would need either parsing the user's engine args or arithmetic that env expansion can't do, and a scheduler constraint rejecting multi-worker gangs on Dynamo would trade a working configuration for a clearer error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions