Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·10d ago·5 min read
GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution
TL;DR
GUI-HARVEST is an automatic harness optimizer that enhances GUI agents by aligning model actions with visual outcomes, leading to significant performance improvements. It achieved a 12.33-point gain on the OSWorld-Verified suite and improved GPT-5 by 13.87 percentage points on WindowsAgentArena.
✦ Why It Matters
Consider implementing GUI-HARVEST to optimize your GUI agents for better performance on diverse tasks.
Key Takeaways
- Utilize GUI-HARVEST for automatic harness optimization.
- Align model actions with visual outcomes for better diagnostics.
- Leverage repeated task executions to identify behavioral differences.
- Map recurring failure patterns to source-code edits for efficiency.
How It Works
GUI-HARVEST optimizes GUI agents by aligning model outputs with visual changes in the interface, using before-and-after screenshots to diagnose issues. It treats multiple executions of the same task as a single evidence unit, allowing for the identification of behavioral differences that impact performance.
Verified findings across tasks are consolidated into reusable patterns, which are then linked to specific code modifications, enabling the system to predict and validate behavioral effects.