NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 26·5 min read
Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models
TL;DR
Fine-tuning large language models (LLMs) can lead to safety issues when exposed to harmful updates. The authors developed a framework called Buffer-and-Reinforce, which uses temporary jailbreaking to mitigate these risks during user fine-tuning.
✦ Why It Matters
Engineers can implement the Buffer-and-Reinforce framework to enhance safety in fine-tuning LLMs without extra data costs.
Key Takeaways
- Utilize BufferLoRA to reduce harmful updates during fine-tuning.
- Integrate ReinforceLoRA to restore safety behaviors post-adaptation.
- Apply QR decomposition for effective merging of model components.
- Achieve improved safety without additional safety data or high computational costs.
How It Works
The Buffer-and-Reinforce framework operates by first employing BufferLoRA, which temporarily activates a jailbroken state to prevent harmful updates during user fine-tuning. This is followed by ReinforceLoRA, which is specifically trained to recover safety behaviors after the fine-tuning process.
The merging of these components is achieved through QR decomposition, allowing for a seamless integration that maintains user-task performance while enhancing safety.