Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
Data Science
Towards AI·Jun 30·5 min read
Five AI Systems. Same Prompts, Twice. Wildly Different Responses.
TL;DR
AI systems often provide inconsistent responses to the same ethical and safety prompts, raising concerns about reliability. Five leading models—Claude, Gemini, GPT-5, Mistral, and Cohere—were tested with 116 identical prompts, revealing significant disagreement.
✦ Why It Matters
Engineers should be aware of the variability in AI responses to improve model training and evaluation methods.
Key Takeaways
- Test AI models with identical prompts to assess consistency.
- Analyze disagreement rates to identify reliability issues.
- Consider implications of response variability in AI applications.
- Explore methods to enhance AI decision-making frameworks.
Full Summary
In the realm of artificial intelligence, consistency in responses to ethical and safety questions is crucial for trust and reliability. This study evaluated five prominent AI models: Claude, Gemini, GPT-5, Mistral, and Cohere, using 116 identical prompts related to ethics and safety.
Each model was prompted twice, and they were tasked with grading each other's responses. The findings revealed that the models disagreed with one another on the same prompt as frequently as 66% of the time.
This inconsistency raises questions about the robustness of AI decision-making frameworks. The analysis serves as a precursor to a deeper exploration of the underlying issues in AI response variability.
Understanding these discrepancies is vital for engineers and researchers aiming to enhance AI reliability.