Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
TL;DR
Current methods for aligning large language models with safety preferences often rely on redundant datasets, limiting their effectiveness. DOG-DPO is a new framework that treats preference pairs as structured geometric signals, optimizing data selection without training.
✦ Why It Matters
Engineers can leverage DOG-DPO to enhance safety alignment in AI models while reducing data requirements and processing time.
Key Takeaways
- Implement DOG-DPO for efficient data selection in AI training.
- Reduce dataset size while maintaining safety alignment effectiveness.
- Utilize geometric representations for preference pairs in model training.
- Achieve faster processing times compared to traditional data selection methods.
Full Summary
Aligning large language models with safety preferences typically involves training on extensive and often redundant datasets, which can dilute the effectiveness of safety alignment. DOG-DPO, or Dynamic Optimization in Geometry for Safety Alignment, introduces a novel approach by representing preference pairs as geometric directions in model representation space.
This framework decomposes the multi-dataset preference geometry into a global anchor subspace and specific residual subspaces for each dataset. By maximizing diversity-based coverage, DOG-DPO selects a non-redundant subset of preference pairs before training.
In experiments across six safety benchmarks and two model architectures, DOG-DPO demonstrated a robust utility-robustness trade-off, recovering most safety gains while using only 11% of the original data. This method is entirely teacher-free and training-free, resulting in a significantly faster data selection process compared to existing baselines.
The implications for engineers include more efficient data usage and improved safety alignment in AI models.