Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 18·5 min read
Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking
TL;DR
A new evaluation framework, MT-InfoSeek, assesses how well large language models (LLMs) identify and request missing information in multi-turn interactions. Results show that models struggle with underspecified queries, often failing to ask enough questions or to ask them in the correct order.
✦ Why It Matters
Evaluate your LLM's performance on multi-turn queries to identify areas for improvement in information seeking.
Key Takeaways
- Implement MT-InfoSeek to assess your LLM's information-seeking capabilities.
- Train models to recognize and request missing information more effectively.
- Optimize query ordering to enhance accuracy in multi-turn interactions.
- Monitor performance degradation as underspecification increases in user queries.
How It Works
MT-InfoSeek formalizes multi-turn information seeking as a k-underspecified constraint satisfaction problem, where k represents the number of missing variables needed to determine a unique answer. This framework allows for systematic evaluation of how well models identify and request additional information.