NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
TL;DR
Long-horizon large language model (LLM) inference leads to high GPU memory usage due to key-value (KV) cache demands. CONF-KV is a new cache management tool that uses a confidence score derived from the model's next-token distribution to optimize cache retention and eviction.
✦ Why It Matters
Engineers can leverage CONF-KV to optimize memory usage in LLM applications without sacrificing performance.
Key Takeaways
- Utilize CONF-KV for efficient memory management in LLMs.
- Leverage confidence scores to optimize cache retention strategies.
- Implement mixed-precision storage to reduce memory footprint.
- Achieve high retrieval accuracy with lower peak memory usage.
Full Summary
As large language models (LLMs) are used for longer text generation, their key-value (KV) caches become the primary consumers of GPU memory, making per-token attention costly. Traditional cache eviction methods often rely on static recency or historical data, neglecting the model's current uncertainty about predictions.
CONF-KV addresses this by converting the next-token distribution into a scalar confidence score, which informs how much context to retain or prune. Within a defined cache budget, tokens are prioritized based on a combination of accumulated attention and recency, while also ensuring local coherence through a protected recent window.
The implementation includes mixed-precision storage (using FP16 and INT8 formats) and a pyramidal budget variant for efficiency. Testing across four model families and up to 4,000 tokens shows that CONF-KV achieves performance within 1.5 to 2.1 perplexity points of full KV, while on specific tasks, it retains 95.3% of full-KV success at 2.8 times lower peak memory usage.
These results suggest significant improvements in memory efficiency without sacrificing model performance.