We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Feb 23·5 min read
Why we no longer evaluate SWE-bench Verified
TL;DR
SWE-bench Verified, a benchmark for evaluating software engineering AI systems, has become unreliable due to test contamination and data leakage from training sets. OpenAI analyzed the benchmark and found flawed test cases and evidence that models trained on public data had seen similar problems before.
✦ Why It Matters
Engineers should stop using SWE-bench Verified for model evaluation and adopt SWE-bench Pro to get accurate measurements of coding AI progress.
Key Takeaways
- Discontinue using SWE-bench Verified for evaluating AI coding models due to unreliable measurements.
- Migrate evaluation workflows to SWE-bench Pro for more rigorous and contamination-resistant benchmarking.
- Audit existing model performance claims based on SWE-bench Verified scores for potential overestimation.
- Implement stricter data isolation protocols when creating new coding benchmarks to prevent training leakage.
Full Summary
SWE-bench Verified is a benchmark designed to measure how well AI models solve real software engineering tasks by evaluating their ability to fix bugs and implement features in code repositories. Over time, the benchmark has accumulated contamination—meaning test cases have leaked into training datasets used by AI models, allowing models to memorize answers rather than solve problems genuinely.
OpenAI's analysis discovered both flawed test design and significant training data overlap that artificially inflates performance scores. This contamination makes it impossible to accurately measure true progress in frontier AI coding capabilities.
As a result, OpenAI recommends SWE-bench Pro as a replacement benchmark with stricter controls against data leakage and more rigorous test validation. This shift reflects the broader challenge of maintaining reliable evaluation standards as AI models grow more capable and training datasets expand.