Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 26·5 min read
Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization
TL;DR
Low-bit quantization of large language models (LLMs) often leads to performance degradation. A new method called BBT-spectral applies Walsh-Hadamard transformations to weight matrices, improving quantization.
✦ Why It Matters
Engineers can leverage BBT-spectral for efficient low-bit quantization in LLMs, enhancing performance without increased resource demands.
Key Takeaways
- Implement BBT-spectral for efficient LLM quantization.
- Utilize Walsh-Hadamard Transform for weight matrix adjustments.
- Expect significant perplexity improvements in quantized models.
- Test across different hardware to ensure performance consistency.
Full Summary
Large language models (LLMs) typically require significant computational resources, and quantization is a technique used to reduce model size and improve efficiency. The BBT-spectral method introduced in this paper utilizes influence-adaptive Walsh geometry to perform weight-only quantization by applying Walsh-Hadamard Transform (WHT) rotations to weight matrices and rescaling them based on activation energy.
This technique biases rounding towards channels with higher spectral energy, leading to improved performance. Testing on four pretrained decoder-only models revealed a perplexity reduction of 15-58% compared to traditional auto-rounding methods.
Additionally, the method was adapted for models that initially struggled with quantization, achieving notable improvements in perplexity metrics. The results indicate that the quantized weights maintain performance across various hardware platforms, suggesting broad applicability.
Overall, this work provides a practical engineering solution for enhancing LLM quantization.