NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 25·8 min read
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
TL;DR
Transformer models cache key-value pairs to speed up inference, but memory grows linearly with context length; sliding-window caching bounds memory but loses evicted tokens entirely. Tensor Cache introduces a two-level memory system pairing fast local attention (L1) with a compressed outer-product matrix (L2) that retains evicted tokens.
✦ Why It Matters
Engineers can reduce Transformer inference memory costs while retaining access to long-context information, enabling longer sequences on memory-constrained hardware.
Key Takeaways
- Implement two-level KV caching: L1 for recent tokens, L2 for compressed evicted pairs.
- Use outer-product matrices to compress evicted key-value pairs instead of discarding them.
- Apply parallel weighted-sum scan instead of chunked-mean shortcuts to avoid numerical errors.
- Train per-head decay and write-rate parameters end-to-end with learned scalar gates.
Full Summary
Autoregressive Transformers—models that generate text token-by-token—maintain a key-value (KV) cache to avoid recomputing attention over past tokens. This cache grows linearly with sequence length, creating memory bottlenecks.
Sliding-window caching bounds memory by keeping only recent tokens, but discards older context entirely. Tensor Cache solves this by maintaining two cache levels: a first-level (L1) sliding-window for recent tokens with exact attention, and a second-level (L2) fixed-size outer-product fast-weight memory that compresses evicted KV pairs into a per-layer matrix A.
When new queries arrive, they read from L2 via a single matrix multiplication using the linear-attention identity. The authors identified and corrected a numerical error in prior chunked-mean training shortcuts that introduced spurious cross-token products, replacing it with a parallel weighted-sum scan.
Experiments across language modeling, associative recall, and memory diagnostics show Tensor Cache improves the memory-quality frontier compared to bounded-state baselines.