Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 17·5 min read
A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
TL;DR
A comprehensive year-long study of Large Language Model (LLM) serving reveals how workloads evolve and user interactions shape production traffic. The findings will be shared through a publicly available production trace, enhancing future research in this area.
✦ Why It Matters
Access the one-year production trace to enhance your understanding of LLM serving workloads today.
Key Takeaways
- Analyze diverse user interactions to improve LLM performance.
- Leverage the one-year trace for benchmarking your models.
- Explore both popular and long-tail models for comprehensive insights.
- Understand workload evolution to optimize resource allocation.
Full Summary
Large Language Model (LLM) serving has emerged as a vital cloud workload, yet previous studies have been limited in their scope and duration. This research presents a detailed analysis of a one-year production trace from Chutes, capturing diverse user interactions and model behaviors.
By examining the workload from various perspectives—aggregate, temporal, model-level, and user-level—the study uncovers patterns of workload evolution and user-model dynamics that were previously obscured. The analysis includes both popular and long-tail models, providing a richer understanding of real-world usage.
To facilitate further research, the complete one-year trace will be made publicly available, allowing others to explore production behaviors without relying on synthetic data. This work sets a new benchmark for future studies in LLM serving.