Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·1d ago
AI Research
arXiv CS.AI·May 25·8 min read
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
TL;DR
Transformer models cache key-value pairs to speed up inference, but memory grows linearly with context length; sliding-window caching bounds memory but loses evicted tokens entirely. Tensor Cache introduces a two-level memory system pairing fast local attention (L1) with a compressed outer-product matrix (L2) that retains evicted tokens.
✦ Why It Matters
Engineers can reduce Transformer inference memory costs while retaining access to long-context information, enabling longer sequences on memory-constrained hardware.
Key Takeaways
- Adopt Tensor Cache for improved memory management in Transformers.
- Utilize the two-level caching system for better context retention.
- Train decay and write-rate parameters end-to-end for optimal performance.
- Explore controlled associative recall capabilities in your applications.
How It Works
Tensor Cache employs a two-level caching strategy where the first level (L1) uses sliding-window softmax attention to manage recent tokens, while the second level (L2) stores evicted key-value pairs in a compressed matrix format. This allows for efficient retrieval of relevant information through matrix multiplication, leveraging the linear-attention identity to maintain performance without excessive memory usage.