NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 21·5 min read
TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization
TL;DR
Large Language Models (LLMs) face accuracy issues when using the Microscaling FP4 (MXFP4) format for activation quantization. To address this, a new technique called Two-Level Orthogonal Rotation (TORQ) was developed to enhance the performance of MXFP4.
✦ Why It Matters
Engineers can implement TORQ to enhance the accuracy of LLMs using low-bit quantization methods.
Key Takeaways
- Explore TORQ for improved LLM accuracy.
- Consider MXFP4 for efficient low-bit inference.
- Analyze error structures in quantization methods.
- Implement findings to enhance practical LLM deployment.
Full Summary
As Large Language Models (LLMs) evolve, the Microscaling FP4 (MXFP4) format has become crucial for efficient low-bit inference, balancing dynamic range and hardware efficiency. However, applying MXFP4 directly to LLM activation quantization often results in notable accuracy loss.
To mitigate this issue, Two-Level Orthogonal Rotation (TORQ) was introduced, which theoretically analyzes and optimizes the error structure of MXFP4 activations. The methodology involves a two-level rotation process that enhances the representation of quantized values.
Experimental results demonstrate that TORQ can significantly reduce accuracy degradation, with improvements measured in various LLM tasks. These findings suggest that TORQ can be a valuable tool for engineers and researchers working on low-bit quantization techniques, enabling more reliable deployment of LLMs in practical applications.