Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 24·5 min read
Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models
TL;DR
MOSAIC, a new benchmark for evaluating vision-language models (VLMs), reveals significant gaps in their ability to translate theory of mind (ToM) reasoning into coordinated social actions. Results show that most VLMs struggle with producing coherent nonverbal signals and interpreting others' behaviors, while PCM-LLM, which includes an explicit ToM module, performs successfully across all scenarios.
✦ Why It Matters
Consider integrating explicit ToM modules in your models to enhance social interaction capabilities.
Key Takeaways
- Explore MOSAIC for evaluating social interaction in AI models.
- Incorporate explicit ToM modules to improve model performance.
- Analyze nonverbal signal coherence in your AI systems.
- Test VLMs in both cooperative and competitive scenarios.
How It Works
MOSAIC evaluates VLMs by requiring them to interpret and respond to verbal and nonverbal cues in social interactions. It systematically varies ToM constraints to assess how well models can translate mental state inferences into coordinated actions.