NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Two to Tango: Coupled Task-Reference Selection for Safe LLM Fine-tuning
TL;DR
Fine-tuning large language models (LLMs) can compromise their safety features when adapting to new tasks. The authors developed DualSelect, a framework that couples task and reference selection to maintain safety while improving task performance.
✦ Why It Matters
Engineers can implement DualSelect to enhance safety in LLM fine-tuning without compromising task performance.
Key Takeaways
- Adopt DualSelect for improved safety in LLM fine-tuning.
- Utilize entropy-regularized scoring for better task-reference alignment.
- Refresh safety references dynamically to maintain model performance.
- Evaluate safety metrics rigorously to ensure compliance with standards.
How It Works
DualSelect operates by simultaneously selecting task samples and safety references, ensuring that the chosen references are relevant to the current task. It uses a minimax approach to balance the preservation of safety and the utility of task performance.
The framework incorporates entropy-regularized scoring surrogates to evaluate the compatibility of task samples with the selected safety references, allowing for a more nuanced selection process. Additionally, lazy reference refresh techniques are employed to update safety references only when necessary, reducing computational overhead.