Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 27·5 min read
GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
TL;DR
Training large language models typically uses either supervised fine-tuning (SFT, learning from labeled examples) or reinforcement learning (RL, learning from rewards), but combining both introduces noise that degrades performance. GAC (Noise-Aware Adaptive Mixing) dynamically adjusts the balance between SFT and RL training based on data quality signals.
✦ Why It Matters
Engineers can use GAC to improve model quality when combining supervised and reinforcement learning training without manual hyperparameter tuning.
Key Takeaways
- Implement adaptive mixing ratios instead of fixed SFT-RL weights to handle variable data quality.
- Monitor training data quality signals to detect and downweight noisy batches automatically.
- Use GAC when combining multiple training objectives to prevent low-quality data from degrading performance.
- Test adaptive mixing on your post-training pipeline if you use both SFT and RL components.
Full Summary
Large language models are typically improved through post-training using either supervised fine-tuning (SFT)—learning directly from human-labeled examples—or reinforcement learning (RL)—optimizing for reward signals. Recent work attempts to combine both approaches for better results, but mixing them introduces challenges: noisy or low-quality data from either source can degrade overall performance.
GAC addresses this by implementing noise-aware adaptive mixing, a technique that monitors data quality during training and dynamically adjusts how much the model learns from SFT versus RL at each step. Rather than using a fixed ratio, GAC detects when training data contains errors or inconsistencies and automatically reduces reliance on that noisier source.
Empirical results demonstrate that this adaptive approach outperforms fixed-ratio mixing strategies, particularly when training data quality varies across batches or time periods.