NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking
TL;DR
Existing benchmarks for mobile GUI agents often lack realism and complexity, limiting their effectiveness. SimuWoB is a synthetic benchmark that simulates 120 diverse and challenging tasks for evaluating these agents.
✦ Why It Matters
Engineers can leverage SimuWoB to better evaluate and enhance the performance of mobile GUI agents in complex scenarios.
Key Takeaways
- Utilize SimuWoB for realistic agent evaluation.
- Focus on improving agent performance in complex tasks.
- Analyze diagnostic insights for capability enhancement.
- Consider long-horizon interactions in agent design.
Full Summary
Mobile GUI agents, which utilize large language models, require realistic evaluation benchmarks to assess their performance effectively. Existing benchmarks tend to focus on simpler tasks and often use open-source applications, which do not reflect real-world complexities.
To address this, SimuWoB was developed as a synthetic benchmark that includes 120 tasks of varying difficulty, designed to simulate real-world mobile app interactions. A robust virtual environment generation framework was created to synthesize these tasks and automatically provide valid rewards.
Evaluation of several state-of-the-art mobile GUI agents showed an average success rate of only 27.92%, with a drop to 17.82% for long-horizon tasks, indicating substantial weaknesses. Comparisons with real-world tasks suggest that assessments based on SimuWoB generalize well, providing valuable diagnostic insights for future development.
These findings emphasize the need for improved capabilities in mobile GUI agents.