Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 4·5 min read
FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
TL;DR
FOCUS is a novel post-training quantization framework that optimizes FP4 quantization for large language models, achieving state-of-the-art accuracy without additional inference costs. It introduces Coupled-Relaxation Scaling and Dual-Granularity Scaling to enhance scale optimization and adapt to local weight distributions.
✦ Why It Matters
Implement FOCUS to enhance the accuracy of your FP4 quantized models without increasing inference time.
Key Takeaways
- Explore FOCUS for optimizing FP4 quantization in LLMs.
- Utilize Coupled-Relaxation Scaling for better scale optimization.
- Apply Dual-Granularity Scaling for improved local weight adaptation.
- Test FOCUS on your existing models to enhance accuracy.
How It Works
FOCUS employs Coupled-Relaxation Scaling (CRS) to separate the quantization and dequantization scales, allowing for a learnable full-precision coefficient that enhances optimization. This decoupling enables more effective adjustments to the quantization process without violating hardware constraints.
Additionally, Dual-Granularity Scaling (DGS) allows for finer adjustments at a sub-block level, which helps tailor the quantization scale to the specific distribution of weights in the model, leading to improved accuracy.