Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 15·5 min read
vToken: Token-Level Virtualization for Reclaimable KV Caches
TL;DR
vToken introduces a token-level virtualization layer for key-value (KV) caches, significantly reducing memory fragmentation and improving throughput in large language model serving. It achieves up to 72.3% reduction in retained KV blocks and 1.37x throughput improvement compared to traditional methods.
✦ Why It Matters
Implement vToken in your KV cache management to optimize memory usage and improve model serving performance today.
Key Takeaways
- Integrate vToken to reduce memory fragmentation in KV caches.
- Leverage asynchronous token repacking for better memory management.
- Use vToken to enhance throughput in large language model serving.
- Simplify your codebase by adopting vToken's lightweight design.
How It Works
vToken operates by decoupling the logical view of token liveness from their physical storage. It maintains a token-table that maps logical tokens to their physical locations, allowing for efficient reclamation of memory.
This design enables asynchronous repacking of live tokens, which minimizes wasted memory and fragmentation, ultimately improving the performance of KV cache management.