EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·26d ago·5 min read
Breaking the 1.58-bit Barrier for Ternary LLMs
TL;DR
BITCOS, a new layout for Ternary Large Language Models (LLMs), reduces weight storage to as low as 1.485 bits per weight, outperforming traditional methods. This innovation enhances model efficiency and inference speed across various platforms.
✦ Why It Matters
Consider implementing BITCOS in your LLMs to optimize storage and inference performance today.
Key Takeaways
- Analyze weight distributions in your LLMs to identify zero density.
- Implement BITCOS for more efficient weight storage.
- Optimize unpacking sequences for modern processors.
- Evaluate performance gains in inference across different hardware.
How It Works
BITCOS utilizes a bitmap to track the presence of non-zero weights and a compact sign vector for the remaining weights. This adaptive layout reduces the effective storage cost based on the actual distribution of weights, particularly benefiting models with high zero density.