NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 28·8 min read
Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning
TL;DR
Multimodal AI systems (those combining text, images, and other data) waste computational resources by processing all visual information upfront, even when only some images are relevant to reasoning tasks. Researchers developed Look on Demand, a cognitive scheduling framework that decides which images to examine and when during problem-solving.
✦ Why It Matters
Engineers can reduce inference costs in multimodal systems by selectively processing images only when reasoning requires them.
Key Takeaways
- Implement dynamic image selection instead of processing all images upfront to reduce computational cost.
- Use learned scheduling to predict which visual inputs are most relevant at each reasoning step.
- Apply cognitive science principles to guide when and which multimodal data to acquire during inference.
- Test selective attention approaches on visual QA and reasoning tasks to measure efficiency gains.
Full Summary
Current multimodal reasoning systems process all available images simultaneously, consuming significant computational resources inefficiently. Look on Demand introduces a scheduling framework inspired by cognitive science principles—specifically, how humans selectively attend to visual information based on task demands.
The framework dynamically decides which images to acquire and examine at each reasoning step, rather than loading everything upfront. This approach treats visual evidence acquisition as a learned scheduling problem, where the system predicts which images will be most informative for the current reasoning task.
Testing on visual question-answering and multimodal reasoning benchmarks showed the framework reduced computational overhead while maintaining or improving answer accuracy. The method enables more efficient deployment of vision-language models in resource-constrained environments.