Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·12d ago·5 min read
ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
TL;DR
ROSS (Relearning from Self-Generated Rollouts through Selective Supervision) enhances reinforcement learning by selectively supervising historical rollouts, leading to improved performance in various tasks. It boosts benchmark scores significantly, demonstrating the value of reusing past experiences in model training.
✦ Why It Matters
Implement ROSS to enhance your model's performance by reusing historical training data effectively.
Key Takeaways
- Utilize ROSS to improve reinforcement learning models.
- Focus on selective supervision of historical rollouts.
- Leverage past experiences for better model training.
- Achieve higher benchmark scores with ROSS methodology.
Full Summary
ROSS introduces a method for leveraging self-generated rollouts from reinforcement learning, which are often overlooked as policies evolve. Traditional approaches discard these historical experiences, but ROSS retains the full trajectory while applying loss only to selected continuations, allowing for more effective learning.
The method was tested across multiple domains, including mathematics and software engineering, showing consistent improvements over baseline models. For instance, on the Qwen3.6-35B-A3B model, ROSS increased the MOPD average from 58.40% to 62.20% and the SWE-bench Verified score from 64.20% to 68.40%.
These results indicate that previously generated rollouts can be fine-tuned through offline supervised learning, enhancing model performance without the need for new policy rollouts. ROSS thus demonstrates a novel approach to maximizing the utility of past training data.