Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·12d ago·5 min read
ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
TL;DR
ARC-KV introduces a novel method for compacting key-value (KV) caches in long-context language models, significantly improving efficiency and accuracy. It reduces compaction time by over 25 times while enhancing accuracy on specific benchmarks.
✦ Why It Matters
Implement ARC-KV to enhance the efficiency of your long-context language model applications today.
Key Takeaways
- Adopt ARC-KV for efficient KV cache management in language models.
- Utilize a value-aware indexer for selecting key anchors.
- Leverage compact cache reuse to save computation time.
- Test performance improvements on benchmarks like QuALITY and LongBench.
How It Works
ARC-KV employs a selective amortization principle, where a reusable anchor-selection policy is learned across contexts. It first trains a value-aware indexer to efficiently select real-key anchors in a single scoring pass.
Then, it merges keys using convex-hull constraints and fits an attention-mass bias against the full cache. This allows for a compact cache to be built once per context, which can be reused for all subsequent queries, significantly reducing compaction time.