Intel squeezed a 1.58-bit LLM down to 1.485 bits without changing a single weight

TL;DR

Intel researchers have successfully compressed a 1.58-bit ternary language model to 1.485 bits per weight without altering its weights. This innovation enhances decoding throughput by up to 27% on GPUs.

Key Takeaways

• Explore BITCOS for compressing ternary model weights.
• Evaluate the proportion of zero weights in your models.
• Test BITCOS on your specific hardware for performance gains.
• Consider bandwidth limitations when optimizing model inference.