ModelCache sizeGiB has no effect below 1TiB on GKE: managed StorageClass is Filestore Enterprise

[dennis-upbound]

What happened?

A ModelCache with sizeGiB: 5 on a GKE cluster provisioned a 1024 GB Filestore instance. The PVC bound and hydration succeeded, so nothing failed — but the cache holding a 1 GB model is backed by a terabyte of Filestore Enterprise, which is roughly $300/month at the current us-central1 rate.

The ModelCache asked for 5 GiB:

spec:
  source: OCI
  oci:
    ref: us-docker.pkg.dev/.../qwen2.5-0.5b-instruct:modelpack-v1
    sizeGiB: 5

What was provisioned:

$ kubectl -n default get pvc -l modelplane.ai/modelcache=qwen25-05b-oci
NAME                                      STATUS   CAPACITY   ACCESS MODES   STORAGECLASS
modelcache-ml-team-qwen25-05b-oci-65dde   Bound    1Ti        RWX            modelplane-rwx

$ gcloud filestore instances list
INSTANCE_NAME                             TIER        CAPACITY_GB  STATE
pvc-3cd6164d-a3e2-49ee-b6cd-ff923bf53f0f  ENTERPRISE  1024         READY

compose-gke-cluster composes the modelplane-rwx StorageClass with tier: enterprise: functions/compose-gke-cluster/function/fn.py#L591. Filestore Enterprise has a 1 TiB minimum share, so GKE rounds any smaller request up to it. There is no error or warning on the PVC — the only signal is that binding takes about ten minutes instead of seconds.

Two consequences beyond the surprise. sizeGiB is documented as the cache's size and reads like a cost control, but below 1 TiB on GKE it has no effect on either capacity or spend. And deriving the size from the model itself — summing OCI descriptor bytes, which the model-delivery work proposes and #411 discusses — computes a number the platform then rounds up by two orders of magnitude, so sizing logic is not the lever. Tier choice is.

Related but not the same as #186, which is about a cache's footprint diverging from where replicas are scheduled. This makes that issue's cost argument concrete: "stage the cache to every cluster the deployment could ever land on" means a 1 TiB Filestore Enterprise instance per cluster, whatever the model's size.

Worth asking separately whether the PVC needs ReadWriteMany unconditionally. It is always RWX, which is what forces Filestore, but a Standalone engine on one node does not share the volume — dynamic-rwo (pd.csi.storage.gke.io) would serve it at a fraction of the cost. A Leader/Worker gang does need RWX.

How can we reproduce it?

1. Create a GKE InferenceCluster and wait for it to report READY.

2. Apply a ModelCache with a small sizeGiB, on the cluster from step 1:

   apiVersion: modelplane.ai/v1alpha1
   kind: ModelCache
   metadata:
     name: small-cache
     namespace: ml-team
   spec:
     source: HuggingFace
     huggingFace:
       repo: Qwen/Qwen2.5-0.5B-Instruct
     sizeGiB: 5

3. On the workload cluster, kubectl -n default get pvc -l modelplane.ai/modelcache=small-cache reports 1Ti, not 5Gi.

4. gcloud filestore instances list shows an ENTERPRISE instance at 1024 GB.

Workaround: none that keeps the managed StorageClass. A user who wants a smaller or cheaper volume has to bring their own StorageClass and accept that Modelplane is not managing it.

What environment did it happen in?

Modelplane version: xpkg.upbound.io/modelplane/modelplane:v0.0.0-1787773615 (built from main at 7ceccb8, plus #411)
Crossplane: v2.4.0
Cluster: GKE, g2-standard-8 + 1x NVIDIA L4, Kubernetes 1.35 (the kubernetesVersion default), us-central1
Providers: provider-family-gcp v2.6.0, provider-gcp-container v2.6.0, provider-gcp-compute v2.6.0, provider-helm v1.2.1, provider-kubernetes v1.2.1
Inference backend: vLLM 0.23.0 (vllm/vllm-openai:v0.23.0)