NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 27·6 min read
OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling
TL;DR
Current LLM evaluations of Theory of Mind (the ability to understand others' mental states) rely only on final answers, hiding whether models actually build internal representations of beliefs. OmniToM is a benchmark that explicitly measures whether LLMs construct and track mental-state models, especially for false or changing beliefs.
✦ Why It Matters
Engineers can now detect whether LLMs truly understand mental states or merely pattern-match, essential for trustworthy AI in social reasoning tasks.
Key Takeaways
- Use OmniToM to benchmark LLMs' Theory of Mind capabilities.
- Focus on belief extraction and labeling for social reasoning tasks.
- Identify and address belief-tracking bottlenecks in LLMs.
- Enhance narrative understanding by modeling character beliefs explicitly.
How It Works
OmniToM evaluates LLMs through a two-stage process. In Stage 1, Belief Extraction, the model identifies beliefs relevant to the narrative's social dynamics.
In Stage 2, Belief Labeling, these beliefs are categorized using a seven-dimensional schema that includes aspects like truth status and mental source, allowing for a comprehensive analysis of how well the model understands character motivations and knowledge.