Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·May 26·5 min read
Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization
TL;DR
Low-bit quantization of large language models (LLMs) often leads to performance degradation. A new method called BBT-spectral applies Walsh-Hadamard transformations to weight matrices, improving quantization.
✦ Why It Matters
Engineers can leverage BBT-spectral for efficient low-bit quantization in LLMs, enhancing performance without increased resource demands.
Key Takeaways
- Apply influence-adaptive Walsh geometry for LLM quantization.
- Expect 15-58% perplexity reduction on benchmark datasets.
- Ensure compatibility with OpenVINO IR for deployment.
- Explore adaptations for models that struggle with quantization.
How It Works
The method involves a single mathematical transformation where each linear layer's weight matrix undergoes a Walsh-Hadamard transform (WHT) rotation. This is followed by rescaling the columns based on the activation energy derived from the Walsh basis, which biases the rounding process towards channels with higher spectral energy.
This innovative approach allows for more effective quantization, leading to improved model performance.