NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 27·8 min read
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
TL;DR
Large language models (LLMs) are computationally expensive; quantization (reducing numerical precision) helps, but low-bit quantization degrades accuracy because activation values (intermediate outputs) have irregular distributions. InfoQuant reshapes activation distributions to be more quantization-friendly before applying low-bit quantization.
✦ Why It Matters
Engineers can deploy LLMs on resource-constrained hardware with minimal accuracy loss using distribution-aware quantization techniques.
Key Takeaways
- Utilize InfoQuant for efficient low-bit quantization of LLMs.
- Apply Peak Suppression Orthogonal Transformation to reshape activations.
- Select outlier tokens adaptively to improve optimization robustness.
- Achieve high accuracy retention while reducing quantization error.
How It Works
InfoQuant reshapes activation distributions using Peak Suppression Orthogonal Transformation (PSOT), which minimizes the numerical range and enhances dispersion. This transformation allows activations to better fit the constraints of low-bit quantization, reducing quantization error significantly.