Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
TL;DR
Large language models (LLMs) often suffer from catastrophic forgetting, losing previous skills during fine-tuning. Reinforcement learning (RL) has been shown to better preserve these capabilities compared to supervised fine-tuning (SFT) due to its policy-gradient updates.
✦ Why It Matters
Engineers can leverage reinforcement learning to enhance the retention of capabilities in large language models during training.
Key Takeaways
- Use reinforcement learning to fine-tune models for better retention of prior skills.
- Implement differential circuit vulnerability to assess model stability during training.
- Balance adaptation speed and circuit preservation based on task requirements.
- Explore the trade-offs between SFT and RL in your model training strategies.
How It Works
The study introduces differential circuit vulnerability, a metric that evaluates how much a model's internal circuits degrade during fine-tuning. By comparing RL and SFT, the authors demonstrate that RL's policy-gradient updates maintain closer alignment with the base policy, leading to better circuit preservation.
This mechanism allows RL to retain more of the model's original capabilities while adapting to new tasks.