We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·29d ago·5 min read
I packed 16 GB of GGUF quants into 1.8 GB, losslessly
TL;DR
A new tool, ggufpacker, compresses 16 GB of GGUF quantizations into just 1.8 GB without losing any data. It achieves this by storing a single F16 source and generating quant variants on demand.
✦ Why It Matters
Engineers can adopt ggufpacker to significantly reduce storage requirements for model quantizations in their projects.
Key Takeaways
- Implement ggufpacker to optimize storage for model quantizations.
- Use a single F16 source to regenerate quant variants on demand.
- Reduce redundancy in model weight storage for efficiency.
- Achieve bit-exact file regeneration to maintain model integrity.
Full Summary
Quantization is a technique used to reduce the size of machine learning models by approximating weights with lower precision formats. The ggufpacker tool was developed to pack multiple GGUF quantizations into a compact storage format, significantly reducing the storage footprint from 16 GB to 1.8 GB.
It does this by retaining only one F16 source file and a small recipe for each quant variant, along with a zstd 'correction delta' for accurate reconstruction. In a test with a real repository, all 17 quant files were regenerated in 283 seconds, with checksums confirming they were bit-exact to the originals.
This approach eliminates redundancy in storing nearly identical model weights, making it more efficient for publishers who often ship multiple quant variants. The implications for engineers include reduced storage costs and faster deployment times for machine learning models.