Announcing MCP Toolbox Java SDK v1.0: Agentic data access for the enterprise
cloud.google.com·6d ago
Cloud & Infrastructure
The New Stack·28d ago·5 min read
AI’s best coding agent fails 60% of the time — and the data backs it up
TL;DR
Claude Fable 5.1 achieved a 38.8% success rate on the Real-SWE coding benchmark, failing 60% of the time when tested on private codebases. This highlights the challenges AI coding agents face in real-world scenarios.
✦ Why It Matters
Evaluate your AI coding tools against real-world scenarios to ensure they meet practical needs.
Key Takeaways
- Test AI coding agents with real-world code to assess performance.
- Be cautious of AI's reliability in unfamiliar codebases.
- Understand that high scores in benchmarks may not reflect real-world effectiveness.
- Explore integration and assumption errors in AI coding outputs.
Full Summary
Claude Fable 5.1 recently topped the Real-SWE coding benchmark with a score of 38.8%, despite failing 60% of the time. Real-SWE tests AI coding agents using private codebases from actual companies, making it less likely that the models have encountered the code before.
The benchmark revealed that most models struggled significantly, with six out of ten tasks yielding success rates below 15%. For instance, a billing schedule migration had a fix rate of only 14.1%, and no model successfully solved the analytics stream reducer across 64 attempts.
The results indicate that while AI can perform well in familiar environments, it struggles with real-world complexities, suggesting a gap between theoretical coding capabilities and practical application.