We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·May 27·5 min read
Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL
TL;DR
Reinforcement learning systems waste GPU time synchronizing full model weights between training and inference clusters. Delta Weight Sync in TRL (a Hugging Face library) transfers only the changed parameters between checkpoints instead of entire models.
✦ Why It Matters
Engineers can reduce RL training infrastructure costs and latency by shipping only weight deltas instead of full checkpoints.
Key Takeaways
- Use delta weight synchronization to reduce data transfer size.
- Leverage Hugging Face Buckets for efficient weight management.
- Train large models without needing shared clusters.
- Achieve over 99% weight sparsity to save bandwidth.
How It Works
The method exploits the fact that most weights in a model remain unchanged between training steps. By only transmitting the small percentage of weights that do change, the system reduces the data transfer size significantly.
This is achieved by encoding the changed weights into a sparse safetensors file, which is then uploaded to a Hugging Face Bucket. The inference engine can fetch these updates independently, allowing for more efficient training and inference processes.