Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventions
TL;DR
Multimodal large language models (LLMs) struggle to develop efficient communication strategies in repeated reference games, failing to create partner-specific conventions. This study investigates whether their alignment on labels is due to shared vocabulary rather than specific grounding with partners.
✦ Why It Matters
Engineers can focus on improving multimodal LLMs to enhance their adaptability in communication tasks.
Key Takeaways
- Multimodal LLMs do not develop partner-specific language.
- Humans adapt communication for efficiency in reference games.
- LLMs maintain verbosity, lacking human-like compactness.
- Use LLMs for consistent terminology in structured tasks.
How It Works
The study employs a constrained pseudo-dyad baseline to isolate the effects of partner history on label alignment. By comparing the communication strategies of multimodal LLMs and humans, it reveals that LLMs rely on fixed verbosity rather than adapting their language based on shared experiences.