Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·May 26·5 min read
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
TL;DR
Long-horizon large language model (LLM) inference leads to high GPU memory usage due to key-value (KV) cache demands. CONF-KV is a new cache management tool that uses a confidence score derived from the model's next-token distribution to optimize cache retention and eviction.
✦ Why It Matters
Engineers can leverage CONF-KV to optimize memory usage in LLM applications without sacrificing performance.
Key Takeaways
- Use CONF-KV for efficient memory management in LLMs.
- Leverage confidence scores to optimize cache retention.
- Achieve high retrieval accuracy with reduced memory footprint.
- Consider mixed-precision storage for better performance.
How It Works
CONF-KV utilizes a confidence score derived from the next-token distribution to dynamically adjust the cache budget for each decoding step. When the model indicates uncertainty, it retains more tokens in the cache, ensuring that relevant context is available.
Conversely, when the model is confident, it aggressively prunes the cache, optimizing memory usage. Tokens are ranked based on a combination of their accumulated attention mass and recency, while a protected recent window helps maintain local coherence in the generated text.