Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs
TL;DR
Omni-Multimodal Large Language Models (Omni-MLLMs) struggle with audio-visual intelligence due to a lack of evaluation benchmarks. AVI-Bench was developed to assess these models through three stages: perception, understanding, and reasoning, using cross-modal tasks.
✦ Why It Matters
Engineers can use AVI-Bench to evaluate and improve the audio-visual capabilities of their Omni-MLLMs.
Key Takeaways
- Implement AVI-Bench to assess your models' audio-visual capabilities.
- Use AVI-Bench-PriSe for testing generalization with unfamiliar stimuli.
- Analyze the four-level AVI taxonomy for model evaluation.
- Identify specific failure modes in your Omni-MLLMs.
How It Works
AVI-Bench evaluates models through a series of tasks that require them to interpret and integrate audio and visual information simultaneously. The benchmark is structured into three stages: perception, understanding, and reasoning, allowing for a comprehensive assessment of model capabilities.
The extension, AVI-Bench-PriSe, introduces low-semantic stimuli to test models' primitive audio-visual sensations, pushing their limits beyond familiar training data.