Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 9·5 min read
The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists
TL;DR
Post-hoc safety methods for aligning large language models are fragile, as they only mask unwanted behaviors rather than eliminate them. In contrast, safety measures integrated during pretraining show persistent effectiveness against attacks.
✦ Why It Matters
Consider implementing safety measures during pretraining to enhance model robustness against attacks.
Key Takeaways
- Explore pretraining-time safety methods for better model alignment.
- Avoid relying solely on post-hoc safety updates for model robustness.
- Implement continuous safety co-training during model development.
- Monitor refusal rates to assess the effectiveness of safety measures.
Full Summary
Current methods for aligning large language models, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often fail due to their fragility, allowing unwanted behaviors to resurface. This study provides a geometric explanation for this issue, revealing that post-hoc safety updates are thin and only suppress capabilities rather than remove them.
By analyzing a range of model checkpoints, the research shows that safety measures applied during pretraining lead to a significant increase in refusal rates, achieving 87-98% effectiveness compared to 84-91% for post-hoc methods. The findings indicate that models trained with continuous safety co-training maintain robust safety signals, outperforming traditional post-hoc installations.
This suggests that integrating safety during the initial training phase is crucial for long-term effectiveness against potential attacks.