Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·27d ago·5 min read
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
TL;DR
MAPS is a Memory-Aware Predictive Scheduling framework designed for serving large language models (LLMs) in cloud environments. It significantly reduces average end-to-end latency by 42.6% and tail latency by up to 84.8% compared to existing systems.
✦ Why It Matters
Implement MAPS to optimize your LLM serving infrastructure and significantly reduce latency.
Key Takeaways
- Adopt MAPS for improved LLM serving efficiency.
- Utilize uncertainty-aware calibration for better scheduling decisions.
- Implement hierarchical scheduling to reduce queue buildup.
- Evaluate performance improvements with real-world workloads.
How It Works
MAPS predicts output lengths using device-assisted methods, allowing for proactive scheduling of resources. It combines this prediction with cloud-side prefill processes to minimize latency.
The framework also incorporates uncertainty-aware calibration to set safe upper bounds for output lengths, which informs scheduling decisions. By employing a hierarchical global-local scheduling strategy, MAPS effectively mitigates issues like queue buildup and head-of-line blocking, leading to more efficient resource utilization.