NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
Data Science
Towards AI·Jul 2·5 min read
Why 4-Bit Weights Are Easy and 8-Bit Activations Break Models: Inside LLM Inference, Part 3
TL;DR
Quantizing the weights of large language models (LLMs) to 4 bits is effective, but quantizing activations to 8 bits can severely degrade model performance. This discrepancy arises from the different roles weights and activations play in model inference.
✦ Why It Matters
Engineers should prioritize careful quantization of activations to avoid degrading model performance.
Key Takeaways
- Quantize weights to 4 bits for efficiency without loss.
- Avoid quantizing activations to 8 bits to maintain performance.
- Understand the asymmetry between weights and activations.
- Optimize LLMs by carefully applying quantization techniques.
Full Summary
Quantization is the process of reducing the precision of numbers in machine learning models to save memory and improve speed. In large language models (LLMs), weights can be quantized from 16 bits to 4 bits without significant loss of performance.
However, quantizing activations, which are the intermediate values between layers, to just 8 bits can lead to catastrophic failures in model behavior. This study highlights the asymmetry between weights and activations, revealing that while both are numerical arrays, their impact on model performance differs greatly.
The findings suggest that engineers must be cautious when applying quantization techniques to activations. By understanding these nuances, researchers can better optimize LLMs for efficiency without sacrificing accuracy.