Event-driven benchmark of prefix-aware request reordering policies for LLM serving, measuring the ROI trade-off between prefix hit rate, prefill savings, and added queueing delay.
python performance-engineering benchmark simulation latency scheduler inference batching systems throughput serving prefill event-driven-simulation kv-cache llm prefix-caching radix-attention request-reordering
-
Updated
Jul 24, 2026 - Python