Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
TL;DR
Reinforcement Learning (RL) often struggles to improve models that have undergone excessive Supervised Fine-Tuning (SFT), due to a loss of model plasticity. To address this, a method called Rejuvenation was developed, which restores plasticity while retaining useful SFT-acquired knowledge.
✦ Why It Matters
Engineers can apply Rejuvenation to improve RL outcomes for models that have been heavily fine-tuned.
Key Takeaways
- Apply Rejuvenation to restore model plasticity after excessive SFT.
- Monitor token distribution to avoid over-confidence in outputs.
- Utilize targeted neuron resets to enhance model adaptability.
- Test RL performance on diverse tasks to ensure generalization.
How It Works
Rejuvenation employs a technique called base-anchored model fusion, which integrates the strengths of the original model with the fine-tuned model. This process includes targeted neuron resets that selectively adjust parameters, allowing the model to regain flexibility without losing the advantages gained from SFT.