Design: shared model storage, define the bar before committing to an RWX backend

[haarchri]

What problem are you facing?

While adding ModelCache support for Vultr we reached for Longhorn as an RWX backend. The discussion that followed (see #370) surfaced that we don't have a shared understanding of what shared model storage is supposed to achieve, so we can't tell whether any given backend is a good choice.

Longhorn was intended as an implementation detail behind the ModelCache backend, not a commitment. Rather than debating it in a PR, this issue is to step back and design the wider thing.

Every RWX option we've looked at funnels traffic through a single point:

• Longhorn attaches the volume RWO to one node and re-exports it over NFS via the share-manager. All engines read through that one pod, capped at its NIC. (Per volume, so one PVC per model spreads it somewhat.)
• EFS caps at ~500 MiB/s–1.5 GiB/s per client. In Bursting mode a 200 GB filesystem drops to 10 MiB/s once credits are exhausted.
• Vultr VFS is native RWX, but on bare metal it goes through a single Storage Gateway, same shape as share-manager.

So we aren't choosing between "funnel" and "no funnel". We're choosing between funnels, and we should know which one clears the bar.

What we know today

• EFS: ~1.4 GB/s per engine observed with two engines loading simultaneously. With --load-format=runai_streamer, a cold read of ~509 GiB per engine takes ~6 minutes (vs ~45 with the default loader), see the kimi-k2 example manifest. Assuming a ~20 GB/s sender-side limit, that suggests ~14 engines loading concurrently before throughput degrades.
• vLLM pulling directly from HuggingFace: ~500 MB/s per engine (Model load from the EFS-backed ModelCache is slow #204).
• no published NIC speeds. Assuming 10G NICs, ~1 GB/s ceiling shared across all engines, which would mean cache is roughly break-even at 2 engines and slower than HF beyond that. This is an estimate; nobody has measured it

The last point is the risk: a ModelCache that makes start time worse in the common case is worse than no ModelCache.

How could Modelplane help solve your problem?

1. Define the bar, then measure against it. N engines, model size S, ready within T. Any backend that can't hit it doesn't ship, Longhorn, EFS or VFS alike. Real numbers from real clusters, not estimates. If nothing clears the bar, that's a signal to change shape rather than pick a winner.
2. Decide whether a shared filesystem is the right shape at all. Given every RWX option funnels, consider alternatives: per-node local cache with pre-pull/warming, P2P distribution between engines, object storage + streaming loader, DaemonSet-based hydration, etc.
3. Decide HuggingFace's role in the scale-up path. If every scale-out pulls from HF, then HF rate limits or an HF outage means no scale-out. That may be an acceptable trade, but it should be a deliberate decision, not a default.
4. Account for traffic cost, not just wall clock. Without a cache we pull S per engine on every cold start and every scale-out, billed per GB read on Object Storage or as egress elsewhere. A cache can be slower on wall clock and still be the right call if it turns N pulls into 1.

A short design doc covering the bar, the measurements, the shape decision, and the HF/cost trade-offs, with enough numbers behind it that the backend choice falls out of the design rather than driving it.

Related

• Keep cluster provisioning implementations consistent as we add providers #331 keeping cluster provisioning implementations consistent across providers. A provider contract should treat "cache StorageClass" as optional and declared, not mandatory, so clouds without a good RWX story don't block the whole provider. The gap needs to be visible in the docs and in the API.
• Model load from the EFS-backed ModelCache is slow #204 vLLM direct-from-HF throughput.