We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Feb 18·5 min read
Introducing EVMbench
TL;DR
Smart contract vulnerabilities—bugs in blockchain programs that can be exploited to steal funds—are difficult to detect and patch at scale. OpenAI and Paradigm built EVMbench, a benchmark dataset that measures how well AI agents can identify, fix, and exploit high-severity vulnerabilities in Ethereum Virtual Machine code.
✦ Why It Matters
Engineers can now measure AI agent progress on smart contract security using a standardized benchmark, enabling data-driven tool development.
Key Takeaways
- Use EVMbench to objectively compare AI-based smart contract security tools on standardized vulnerability detection tasks.
- Evaluate your security automation pipeline against high-severity EVM vulnerabilities using quantitative metrics.
- Benchmark AI agents' ability to generate functional patches, not just identify bugs.
- Measure exploitation detection capabilities to understand if AI can identify attack vectors humans might miss.
Full Summary
Smart contracts are self-executing programs on blockchains like Ethereum that manage financial transactions; vulnerabilities in these contracts can lead to significant losses. OpenAI and Paradigm introduced EVMbench, a standardized benchmark designed to evaluate AI agents' performance across three critical security tasks: detecting vulnerabilities, generating patches to fix them, and identifying exploitable weaknesses in contract code.
The benchmark focuses on high-severity issues in EVM (Ethereum Virtual Machine) bytecode, the low-level language executed on Ethereum. EVMbench enables researchers to systematically measure and compare different AI approaches—from language models to specialized security tools—on a consistent set of real-world contract examples.
This work addresses a gap in AI evaluation: while general-purpose benchmarks exist for code analysis, none previously measured AI agents' ability to handle the specific, high-stakes domain of blockchain security. The benchmark provides quantitative metrics for detection accuracy, patch quality, and exploitation success, allowing engineers to track progress in automated smart contract security.