Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 26·5 min read
Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
TL;DR
Sparse mixture-of-experts (MoE) models like Mixtral activate only a subset of parameters, raising safety concerns under different prompts. This study analyzes routing behavior using activation-based and gradient-based scores to understand expert selection.
✦ Why It Matters
Engineers can refine expert selection strategies to enhance safety in AI models under varying input conditions.
Key Takeaways
- Analyze routing behavior under different prompt types for safety insights.
- Utilize both activation and gradient-based scores for expert evaluation.
- Implement expert suppression to mitigate harmful responses.
- Focus on layer-specific routing dynamics for improved model safety.
How It Works
The routing mechanism in Mixtral 8x7B-Instruct selectively activates a subset of experts based on input prompts. Activation-based scores reflect how frequently experts are chosen, while gradient-based scores indicate their sensitivity to changes in input.
This dual analysis allows for a comprehensive understanding of expert contributions to model outputs.