Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
Data Science
Towards AI·24d ago·5 min read
Databricks Benchmarked Coding Agents on Its Own Codebase. The Results Should Change How You Buy
TL;DR
Databricks' benchmarking of coding agents revealed that open-source models have caught up, with GLM-5.2 outperforming others outside of benchmarks. A minimal harness developed by the team matched vendor solutions at half the cost.
✦ Why It Matters
Evaluate coding agents based on real-world performance rather than just cost per token to optimize your development budget.
Key Takeaways
- Consider open-source models like GLM-5.2 for coding tasks.
- Use a minimal harness for cost-effective benchmarking.
- Align benchmarks with your specific codebase for accurate assessments.
- Beware of hidden costs associated with cheaper per-token rates.
Full Summary
Databricks CTO Matei Zaharia's team conducted a benchmarking study on coding agents using their own codebase to assess performance and cost-effectiveness. They discovered that open-source models, particularly GLM-5.2, have made significant strides, proving effective beyond traditional benchmark tests.
The team created a minimal harness for testing, which demonstrated comparable performance to vendor solutions at just half the cost. They also noted that public coding benchmarks, like SWE-bench and Terminal-Bench, often focus on Python-centric repositories with fixed tests, which may not reflect the diversity of real-world codebases.
Their findings suggest that while cheaper per-token rates are appealing, they can lead to higher costs per task, necessitating a more nuanced approach to selecting coding agents. This research emphasizes the importance of aligning benchmarks with actual coding environments to make informed purchasing decisions.