Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 7·5 min read
What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
TL;DR
On-Policy Distillation (OPD) can effectively enhance large language models using only a few hard examples, achieving performance comparable to training on extensive datasets. This study reveals that longer reasoning paths in challenging examples drive significant improvements in model performance.
✦ Why It Matters
Consider implementing a hard example selection strategy in your OPD training to enhance model performance.
Key Takeaways
- Explore 1-shot OPD for efficient model training.
- Focus on hard examples to drive performance improvements.
- Utilize longer chains of thought for better reasoning.
- Implement a data selection strategy prioritizing challenging tasks.
How It Works
The study reveals that training on longer chains of thought (CoT) from hard examples helps models maintain alignment with teacher models over extended reasoning tasks. This approach encourages the development of critical thinking patterns, such as reflection, which are often absent in shorter reasoning paths.