Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·May 27·5 min read
GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
TL;DR
Training large language models typically uses either supervised fine-tuning (SFT, learning from labeled examples) or reinforcement learning (RL, learning from rewards), but combining both introduces noise that degrades performance. GAC (Noise-Aware Adaptive Mixing) dynamically adjusts the balance between SFT and RL training based on data quality signals.
✦ Why It Matters
Engineers can use GAC to improve model quality when combining supervised and reinforcement learning training without manual hyperparameter tuning.
Key Takeaways
- Adopt GAC for dynamic mixing in hybrid training setups.
- Monitor gradient variance to optimize training signal weights.
- Utilize existing training tensors to minimize overhead.
- Expect larger performance gains with larger model scales.
How It Works
GAC employs a noise-aware controller that calculates adaptive mixing weights by analyzing the variance of gradients and the level of disagreement between SFT and RL signals. This allows the training process to adjust in real-time, ensuring that the model learns effectively from both types of signals.
The method also integrates smoothing techniques and prior guidance to stabilize updates, which helps in maintaining consistent performance.