NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding
TL;DR
Multimodal large language models (MLLMs) struggle to detect when no correct answer is available in video understanding tasks. A diagnostic study evaluated their performance using multiple-choice questions, open-ended generation, and standard evaluation methods.
✦ Why It Matters
Engineers should consider integrating explicit detection mechanisms to enhance MLLM reliability in video understanding tasks.
Key Takeaways
- Evaluate MLLM performance in detecting absent answers.
- Implement explicit detection mechanisms in multimodal systems.
- Explore chain-of-thought prompting for improved detection.
- Focus on temporal reasoning tasks for better model training.
Full Summary
Multimodal large language models (MLLMs) have advanced video understanding but their reliability in detecting absent answers is under-researched. This study investigates how well MLLMs can identify when no correct answer exists by testing them in three scenarios: multiple-choice questions with a 'None of the Above' option, open-ended generation with specific detection instructions, and standard evaluations without guidance.
Across various models and benchmarks, MLLMs frequently chose plausible but incorrect distractors instead of recognizing the absence of a valid answer, especially in tasks requiring temporal reasoning. The study also explored chain-of-thought prompting as a potential solution, which improved detection rates but still yielded unsatisfactory results.
These findings reveal a systematic failure in absent answer detection, indicating a need for more explicit detection mechanisms in multimodal systems. This research highlights the limitations of current MLLMs and suggests areas for future improvement.