EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·Aug 19·5 min read
LFM2.5 Q4\_0 Checkpoints from Quantization-Aware Distillation
TL;DR
Quantization-Aware Distillation (QAD) has been applied to create Q4_0 checkpoints, achieving high performance while maintaining low memory usage. The QAD models recover up to 97% of accuracy lost during quantization, significantly enhancing their capabilities.
✦ Why It Matters
Implement QAD in your model training to enhance performance while minimizing resource usage.
Key Takeaways
- Utilize QAD for efficient model distillation in your projects.
- Leverage the LFM2.5 Q4_0 checkpoints for high performance on edge devices.
- Test the new models on various hardware for optimal throughput.
- Access the QAD GGUFs on Hugging Face to integrate into your applications.
Full Summary
Quantization-Aware Distillation (QAD) is a technique where a high-precision teacher model is distilled into a quantized student model, optimizing performance for edge devices. The LFM2.5 Q4_0 checkpoints, released by Liquid AI, achieve a memory-efficient and high-throughput design similar to native Q4_0 models.
They recover 97% of the average BF16 accuracy lost during quantization across four model sizes: 230M, 350M, 1.2B-Instruct, and 2.6B. Performance evaluations show that the QAD models match or exceed the throughput of previous models while retaining high accuracy, with improvements of 4-33% in decode throughput.
These checkpoints are compatible with llama.cpp and other runtimes supporting GGUF Q4_0 artifacts, making them accessible for developers. The QAD GGUFs are now available on Hugging Face, encouraging further innovation in AI applications.