Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 12·5 min read
The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation
TL;DR
A new metric called the convention gap measures the difference between predicted and actual failure rates in cooperative AI communication. Results from the card game Hanabi show that human pairs exhibit a significant convention gap, suggesting that AI effectiveness with humans depends on shared communication conventions.
✦ Why It Matters
Evaluate your AI systems for convention compatibility to enhance human-AI collaboration today.
Key Takeaways
- Measure AI performance using the convention gap metric.
- Focus on implicit communication for better AI-human collaboration.
- Analyze failure rates in cooperative tasks to improve AI design.
- Consider human conventions when developing AI evaluation benchmarks.
How It Works
The convention gap is calculated by comparing the predicted failure rates based on literal communication with the actual failure rates observed during gameplay. This metric highlights the discrepancies in understanding between human and AI players, particularly in scenarios where implicit communication is essential.