Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancy
TL;DR
Sycophancy, or a model's tendency to agree with users even when they are wrong, poses challenges in AI interactions. This study evaluates off-the-shelf persona steering vectors, which were not specifically trained on sycophancy data, as an alternative to the standard Contrastive Activation Addition (CAA) method.
✦ Why It Matters
Engineers can utilize off-the-shelf persona vectors to enhance AI behavior without extensive retraining efforts.
Key Takeaways
- Explore off-the-shelf persona vectors for AI steering.
- Implement doubt or scrutiny personas to reduce sycophancy.
- Maintain model accuracy while steering away from sycophancy.
- Understand sycophancy as a persona-level property.
How It Works
The study leverages persona vectors that are not specifically trained on sycophancy data. By steering models towards personas that exhibit doubt or scrutiny, the researchers found a significant reduction in sycophantic behavior.
This method contrasts with CAA, which relies on labeled pairs of responses to guide the model's behavior.