Accelerating the search for antimicrobial peptides with Google Antigravity
deepmind.google·2h ago
AI Research
arXiv CS.AI·1d ago·5 min read
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
TL;DR
SAGE introduces a novel approach to 2-bit vector quantization for large language models, achieving state-of-the-art performance while significantly reducing deployment costs. It retains 95.3% accuracy of the original model and offers a 2.55-fold speedup over FP16.
✦ Why It Matters
Consider implementing SAGE for your LLMs to enhance performance and reduce costs immediately.
Key Takeaways
- Explore SAGE for efficient LLM deployment.
- Utilize Energy-Stratified Grouping to prioritize sensitive channels.
- Implement Adaptive Bit Split for optimal bit allocation.
- Leverage the custom CUDA kernel for performance improvements.
How It Works
SAGE combines two innovative techniques to enhance vector quantization. Energy-Stratified Grouping (ESG) organizes weight channels based on their energy levels, ensuring that more critical channels are preserved during quantization.
Adaptive Bit Split (ABS) intelligently divides the bit budget between direction and radius components, minimizing reconstruction errors while adhering to a fixed bit limit.