CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models

TL;DR

CARD introduces a method to diagnose failures in how vision-language models (VLMs) utilize internal representations of beliefs for action predictions. The key finding reveals that these models often neglect valuable belief information when deciding on their next actions.

Key Takeaways

• Implement CARD to diagnose belief-action routing failures in your VLMs.
• Test VLMs on the Relay Chain benchmark for cooperative tasks.
• Focus on integrating belief representations into action prediction models.
• Analyze the impact of belief information on model performance.