Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Self-Mined Hardness for Safety Fine-Tuning
TL;DR
Safety fine-tuning of language models often relies on curated adversarial datasets, which can be limiting. A novel method scores candidate prompts based on the frequency of harmful outputs from the model itself, then fine-tunes using the most challenging prompts alongside the model's own outputs.
✦ Why It Matters
Engineers can leverage self-mined hardness to enhance safety fine-tuning processes for language models effectively.
Key Takeaways
- Implement self-mined hardness for safety fine-tuning in language models.
- Evaluate prompt difficulty based on model's harmful output frequency.
- Focus fine-tuning on the hardest prompts identified by the model.
- Monitor and measure success rates of adversarial attacks post-tuning.
Full Summary
Safety fine-tuning aims to reduce harmful outputs from language models, traditionally using curated adversarial datasets. In this study, a new technique is introduced that evaluates the difficulty of prompts by analyzing how often the model generates harmful responses during its own rollouts.
The fine-tuning process then focuses on the hardest prompts, using the model's own outputs that are not manipulated or 'jailbroken.' When applied to the Llama-3-8B-Instruct and Llama-3.2-3B-Instruct models, this method drastically lowers the success rate of WildJailbreak attacks, which are attempts to bypass safety measures, to between 1% and 3%.
This indicates a more effective way to enhance model safety without relying on external datasets. The findings suggest that self-assessment can be a powerful tool in improving AI safety mechanisms.