NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Optimal Post-Training Quantization Scales and Where to Find Them
TL;DR
Post-training quantization (PTQ) often relies on simple heuristics to compress large language models, which can lead to suboptimal performance. The authors developed PiSO (Piecewise Scale Optimization), an algorithm that uses calibration data to determine optimal weight scales for quantization.
✦ Why It Matters
Engineers can leverage PiSO to optimize quantization in their models, improving performance while reducing resource usage.
Key Takeaways
- Utilize PiSO for optimal quantization scales in your models.
- Expect improved perplexity with lower bit-widths using PiSO.
- Incorporate calibration data for better quantization outcomes.
- Test PiSO on various model sizes for consistent performance gains.
How It Works
PiSO optimizes quantization scales by partitioning the scale search space into intervals, allowing for closed-form minimization of quantization error. This method leverages calibration data to compute channel-wise weight scales more accurately than traditional heuristics.