Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 2·5 min read
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
TL;DR
InteractBench is a new benchmark designed to evaluate large language models (LLMs) on interactive competitive programming problems, revealing a significant performance gap. The study shows that even advanced models struggle with dynamic information acquisition and state tracking.
✦ Why It Matters
Evaluate your LLMs against interactive benchmarks to identify and address their limitations in dynamic problem-solving.
Key Takeaways
- Incorporate interactive benchmarks like InteractBench in model evaluations.
- Focus on improving dynamic information acquisition capabilities in LLMs.
- Analyze failure types to enhance algorithmic reasoning in models.
- Utilize curated problems from competitive programming platforms for testing.
How It Works
InteractBench consists of 322 interactive problems that require LLMs to engage in multi-round interactions with a judge program. Each problem is designed to test the model's ability to query for information and track state dynamically, simulating real competitive programming scenarios.