NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 21·6 min read
SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips
TL;DR
LLM inference servers struggle to meet latency targets (SLOs) when request volume exhausts GPU memory for storing attention key-value caches, causing head-of-line blocking where early requests stall later ones. SuperInfer introduces rotary scheduling and memory management that intelligently prioritizes requests and moves cache data between GPU and CPU RAM to maintain responsiveness.
✦ Why It Matters
Engineers deploying LLMs at scale can now handle higher concurrent request loads without sacrificing response time guarantees.
Key Takeaways
- Rotary scheduling reorders requests across GPUs to prioritize SLO compliance over strict FIFO ordering.
- Dynamic KV cache allocation prevents memory exhaustion from triggering cascading request delays.
- Superchip-aware design exploits tight GPU coupling for faster memory coordination than PCIe offloading.
- Measure latency percentiles (p99, p95) and throughput together; SLO compliance alone is insufficient.
Full Summary
LLM serving requires balancing two competing demands: meeting Service Level Objectives (SLOs)—strict latency guarantees—while handling high request rates on limited GPU memory. The KV cache (key-value cache stores intermediate computations needed for each request) becomes a bottleneck; when memory fills, incoming requests experience head-of-line blocking (queuing delays where early requests delay all subsequent ones).
Prior work used PCIe-based offloading (moving data to CPU memory via slower interconnects) but failed under sustained high load. SuperInfer addresses this by introducing rotary scheduling (intelligently reordering request execution across multiple GPUs) and coordinated memory management for superchips (tightly-coupled multi-GPU systems).
The technique prioritizes requests likely to meet SLOs and manages KV cache allocation dynamically. Results demonstrate improved throughput while maintaining latency compliance compared to existing systems.