Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
TL;DR
Large language models (LLMs) often suffer from catastrophic forgetting, losing previous skills during fine-tuning. Reinforcement learning (RL) has been shown to better preserve these capabilities compared to supervised fine-tuning (SFT) due to its policy-gradient updates.
✦ Why It Matters
Engineers can leverage reinforcement learning to enhance the retention of capabilities in large language models during training.
Key Takeaways
- Explore reinforcement learning for model training.
- Consider internal circuit preservation in LLMs.
- Evaluate training methods to reduce forgetting.
- Implement policy-gradient techniques in fine-tuning.
Full Summary
Catastrophic forgetting occurs when large language models (LLMs) lose previously learned information during the fine-tuning process. Recent findings indicate that reinforcement learning (RL) is more effective than supervised fine-tuning (SFT) in retaining prior capabilities, as RL's policy-gradient updates are closer to the original model's policy.
This research extends the behavioral observations to a mechanistic level, examining whether RL also preserves the internal computational circuits of LLMs better than SFT. The study employs comparative analysis of model performance and internal representations during training.
Results suggest that RL not only retains external capabilities but also maintains the integrity of internal structures more effectively than SFT. These findings have significant implications for the design of training protocols for LLMs, emphasizing the potential of RL in mitigating forgetting.
Understanding these mechanisms can guide engineers in developing more robust AI systems.