NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
TL;DR
Current benchmarks for AI agents focus on short tasks, limiting their evaluation in long-term software engineering workflows. SWE-Marathon is introduced as a benchmark consisting of 20 long-horizon tasks that assess agents' planning, memory, and context understanding.
✦ Why It Matters
Engineers can leverage SWE-Marathon to better evaluate and improve AI agents for complex software tasks.
Key Takeaways
- Adopt SWE-Marathon for evaluating AI agents in software tasks.
- Focus on long-term planning capabilities in agent design.
- Measure agents' performance over extended periods.
- Explore improvements in memory utilization for complex workflows.
Full Summary
AI agents are increasingly tasked with completing long-horizon workflows in software engineering, which require sustained effort over hours and involve complex environments. Existing benchmarks primarily evaluate short tasks, such as single pull requests or small tickets, which do not adequately measure an agent's ability to plan, understand long contexts, or utilize memory effectively.
SWE-Marathon was developed as a new benchmark featuring 20 long-horizon tasks designed to challenge these capabilities. The methodology includes evaluating agents on their performance across these tasks, focusing on metrics like planning efficiency and memory usage.
Initial findings suggest that agents struggle with sustained performance over extended periods, highlighting the need for improved long-term planning strategies. This benchmark not only provides a more rigorous evaluation framework but also encourages the development of more capable AI agents for real-world software engineering challenges.