[Web]BucketCacheManager per-bucket limits are far below LLM KV-tensor counts, causing permanent ~3x decode degradation for dynamic-KV workloads

[kylo5aby]

Describe the issue

BUCKET_DEFAULT_LIMIT_TABLE (onnxruntime/core/providers/webgpu/buffer_manager.cc) caps retained buffers per size bucket:

{4194304, 20}, {8388608, 10}, {12582912, 10}, {16777216, 10}, {26214400, 15}, ...

LLM decode with a growing KV cache (the standard generate() pattern: feed past_key_values, read present.* as gpu-buffer) requests one same-sized buffer per KV output per step and releases the previous set — 64 buffers/step for a 32-layer model. 64 ≫ cap, so 64 − cap buffers are truly created (alloc + zero-fill) and destroyed on every step.

Result: a permanent per-token latency staircase, steps exactly on bucket boundaries. Phi-4-mini q4f16 (64 KV tensors, KV tensor = 2048·S bytes), onnxruntime-web 1.29, Intel iGPU:

S	bucket	cap	miss	ms/token
< 2048	4 MiB	20	44	~46
2048–4096	8 MiB	10	54	~138
4096–6144	12 MiB	10	54	~233
> 6144	16 MiB	10	54	~298

Decode drops ~22 → ~4 TPS over 8k context.

Evidence pinning this to the bucket table:

• Step at S=6144 = the 12→16 MiB boundary; 12 MiB is not a power of two, so generic size-class effects predict nothing there.
• Qwen3.5-4B (16 growing KV tensors, fits under caps): no comparable staircase, and the predicted negative step at 16→25 MiB (cap 10→15, miss 6→1) measured as −6.0 ms.
• • Zero-churn control (pre-allocated KV bound in-place via fetches, same graph): no staircase, ~3x faster at long context.

Why existing options don't cover this

• storageBufferCacheMode: 'disabled': destroys released buffers immediately → use-after-destroy validation error with GPU-resident KV I/O binding.
• 'simple': caches by exact size, never evicts → dynamic shapes miss every step and the pool grows unboundedly → device lost. (Fine for the static-shape case in [Web] Expose WebGPU EP buffer cache mode options in JS #29016 — different workload.)
• 'bucket' (default): works, but fixed caps produce the above.
• [js/web] Forward WebGPU EP buffer cache mode options from JS #29017 exposes mode selection to JS; no mode handles "N same-sized buffers per step, N > cap, size drifting across buckets".

Proposed direction

Simply raising the caps doesn't work: the needed cap is model-dependent (40 layers → 80 tensors), and decode walks through every bucket over time, so cap=64 puts worst-case retained memory near ~4 GiB (vs ~815 MiB today). Alternatives:

1. Adapt per-bucket caps to the observed acquire rate (with decay), or
2. Cross-bucket eviction under a total-bytes budget — the bucket a growing KV has moved out of is pure dead weight today, or
3. Expose the limit table via EP options (following [js/web] Forward WebGPU EP buffer cache mode options from JS #29017), letting callers size it to their KV tensor count.

To reproduce

In Transformers.js/ort-web:

1. Load Phi-4-mini-instruct ONNX q4f16, device: "webgpu".
2. Prompt ~1900 tokens, generate 256 greedily, timestamp each TextStreamer.put.
3. Per-token latency jumps ~46 → ~138 ms permanently at position ~2050. Prefill ~3900 / ~6000 shows the 4096 / 6144 steps.

Urgency

No response

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.29.0(commit id: 7ad118c)

Execution Provider

'webgpu' (WebGPU)

[qjia7]

For best performance, we recommend using a static/shared KV cache when the model supports it. Preallocate to max_sequence_length and reuse the same buffers for past and present KV states, similar to GenAI’s past_present_share_buffer=true.

Dynamic KV cache is still valid, but it exposes a limitation in the current WebGPU bucket allocator. LLMs may need dozens of similarly sized KV buffers per run, exceeding the fixed per-bucket limits and causing repeated allocations. Simply increasing the limits or using simple mode is not a scalable solution.

Maybe the bucket policy could be made adaptive:

• Track peak demand for each bucket during a run.
• Retain enough buffers for the recently observed working set.
• Evict buffers from cold buckets and gradually decay inactive bucket limits.

Separating model I/O buffers from intermediate tensors could be considered later, but it requires plumbing allocation roles through ORT. An adaptive bucket policy is a smaller, more general improvement for dynamic KV cache.

[kylo5aby]

For best performance, we recommend using a static/shared KV cache when the model supports it. Preallocate to max_sequence_length and reuse the same buffers for past and present KV states, similar to GenAI’s past_present_share_buffer=true.

Dynamic KV cache is still valid, but it exposes a limitation in the current WebGPU bucket allocator. LLMs may need dozens of similarly sized KV buffers per run, exceeding the fixed per-bucket limits and causing repeated allocations. Simply increasing the limits or using simple mode is not a scalable solution.

Maybe the bucket policy could be made adaptive:

• Track peak demand for each bucket during a run.
• Retain enough buffers for the recently observed working set.
• Evict buffers from cold buckets and gradually decay inactive bucket limits.

Separating model I/O buffers from intermediate tensors could be considered later, but it requires plumbing allocation roles through ORT. An adaptive bucket policy is a smaller, more general improvement for dynamic KV cache.

Thanks for the detailed response.

On static/shared KV: that's exactly the route we took at the framework level. We've implemented an opt-in StaticCache for Transformers.js huggingface/transformers.js#1745. Results (q4f16, 256 new tokens, Intel iGPU(PTL, B390H)):

Model	Prefill	Dynamic (TPS)	Static (TPS)
Phi-4-mini-instruct	2048	10.1	25.4
Phi-4-mini-instruct	4096	5.6	20.8
Qwen3.5-4B	2048	17.7	19.2
Qwen3.5-4B	4096	15.1	18.2

**On the adaptive bucket policy: ** it composes with static KV (which shifts pressure from cache buffers to the remaining per-step intermediates). Two behaviors from our data that the policy would need to handle, which could serve as acceptance tests:

1. Burst demand within one step: a 32-layer model acquires ~64 same-sized buffers in a single session.run. Peak-demand tracking needs to observe the intra-run high-water mark, not just acquire/release rates across runs.
2. Working set migrating across buckets: as the KV grows, demand moves 4->8>12->16 mb, and never returns. The bucket just vacated holds cap x size of dead weight, so the decay/eviction path is what keeps adaptive caps memory-safe.