Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
TL;DR
Existing methods for predicting large language model (LLM) behavior using self-reports (SR) have shown inconsistencies, particularly when relying on broad personality traits like the Big 5. This study contrasts the Big 5 with the Theory of Planned Behavior (TPB), which focuses on specific intentions and behaviors, revealing that TPB provides better coherence in predicting LLM behavior.
✦ Why It Matters
Engineers should consider using the Theory of Planned Behavior for more accurate predictions of LLM behavior in specific contexts.
Key Takeaways
- Use the Theory of Planned Behavior for better behavior predictions.
- Avoid relying solely on broad personality traits like the Big 5.
- Consider context when evaluating LLM behavior consistency.
- Implement persona prompting to enhance self-report reliability.
Full Summary
Predicting the behavior of large language models (LLMs) using self-reports (SR) has been problematic, especially when relying on broad personality frameworks like the Big 5, which correlate weakly with specific behaviors. This research introduces the Theory of Planned Behavior (TPB), which assesses intentions related to specific actions, and conducts experiments across four behavioral tasks with 11 advanced LLMs.
The findings reveal that within a shared conversation, TPB achieves human-level coherence, while the Big 5 does not. Coherence persists across separate conversations for behaviors influenced by training, but collapses when context strongly primes behavior.
Additionally, persona prompting improves SR consistency but does not align behavior with self-reports. These results suggest that more targeted psychometric tools are necessary for evaluating LLM behavior effectively across various contexts.