Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·25d ago·5 min read
MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents
TL;DR
MIRAGE evaluates how conversation state affects the use of historical evidence in multimodal large language model agents. It reveals that context continuity is crucial for accurate responses, with distinct failure patterns observed in evidence retrieval.
✦ Why It Matters
Evaluate your multimodal agents' performance by considering conversation state variations in evidence retrieval.
Key Takeaways
- Assess conversation state impact on evidence retrieval in MLLM agents.
- Implement strategies to maintain context continuity during interactions.
- Explore tool-mediated retrieval options when evidence provenance fails.
- Analyze distinct failure patterns in evidence use for better model training.
Full Summary
MIRAGE (Multimodal Interaction Retrieval, Attribution, and Grounding Evaluation) investigates the role of conversation state in the effectiveness of multimodal large language model (MLLM) agents, which serve as personal assistants. The study maintains fixed evidence objects and questions while varying conversation states to assess how well agents can retrieve and utilize historical evidence.
Results show that pre-compaction depth and post-compaction continuation create unique failure patterns rather than a simple degradation curve. Open-weight models struggle with context continuity and often fail to switch to tool-mediated retrieval when needed.
Additionally, while retrieval pressure enhances source attribution in deep pre-compaction states, it deteriorates after evidence compaction. These findings suggest that evaluating historical evidence use should consider conversation state variations rather than relying solely on outcome correctness.