NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference
TL;DR
Long-horizon agents using large language models (LLMs) face bottlenecks due to excessive key-value (KV) memory usage. IntentKV is a pruning technique that optimizes KV cache while keeping the LLM unchanged, focusing on cross-turn intent.
✦ Why It Matters
Engineers can implement IntentKV to optimize resource usage in LLM applications without compromising performance.
Key Takeaways
- Adopt IntentKV for efficient memory management in LLMs.
- Reduce KV reads significantly without accuracy loss.
- Utilize session-level QueryMemory for better context retention.
- Implement slot-map redirection for effective cache eviction.
How It Works
IntentKV employs a learned pruning strategy that maintains a session-level QueryMemory to track user intent across multiple turns. It scores historical tokens using a memory-attention rule, allowing the model to prioritize relevant information.
A zero-initialized residual head is added to facilitate cross-attention over current-query K-vectors, optimizing the cache's efficiency. The eviction process uses a slot-map redirection technique, ensuring that the cache structure remains intact while optimizing memory usage.