Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 9·5 min read
FastE: Readout-Triggered Token Compression for LLM Embedding Inference
TL;DR
FastE is a novel method for compressing token embeddings in large language models (LLMs) that significantly reduces computational costs while maintaining high performance. It achieves a 40.11% reduction in decoder-backbone FLOPs on NarrativeQA, preserving 99.53% of the original model's effectiveness.
✦ Why It Matters
Implement FastE today to optimize your LLM embedding inference without retraining.
Key Takeaways
- Use FastE to reduce computational costs in LLMs.
- Adjust the maximum removal ratio for customizable efficiency.
- Apply FastE in retrieval and indexing tasks.
- Leverage FastE's training-free approach for quick implementation.
How It Works
FastE identifies and compresses prefix states in LLMs based on their depth in the network. It uses a shared threshold on batch-mean readout-prefix alignment to decide when to compress and ranks prefix states by their attention scores from the readout position.
This allows for effective state retention in deeper layers while removing less critical states in shallower layers.