Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 2·5 min read
Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO
TL;DR
Post-training methods like GRPO, SFT, and DPO enhance language model grounding but largely rely on existing model capabilities rather than introducing new mechanisms. The study reveals that grounding gains are modest and depend on pre-existing machinery.
✦ Why It Matters
Evaluate your model's existing capabilities before applying post-training methods for grounding improvements.
Key Takeaways
- Assess existing model capabilities before post-training.
- Consider using DPO for optimal grounding improvements.
- Evaluate grounding gains from GRPO and SFT methods.
- Understand the limitations of post-training approaches.
Full Summary
Language models often disregard prompt evidence when it contradicts their memorized knowledge. To address this, post-training methods such as GRPO (Gradient Reinforcement Policy Optimization), SFT (Supervised Fine-Tuning), and DPO (Direct Preference Optimization) were evaluated to see if they require new mechanisms or strengthen existing ones.
The study compared nine post-training approaches from a single model checkpoint, finding that grounding improvements were generally small, particularly with GRPO variants. While Conflict-SFT showed moderate improvements, DPO achieved near-optimal grounding.
Notably, both methods utilized the same causal attention-head set as the original model. By manipulating the model's starting direction, researchers could suppress or recover grounding gains, indicating that these improvements largely stem from pre-existing model capabilities.
After a supervised warm start, further grounding gains from GRPO were negligible, suggesting limited additional benefits from this approach.