NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 25·8 min read
WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games
TL;DR
Coding agents that build applications are typically evaluated on code quality rather than whether the final product actually works. WebGameBench is a benchmark that tests if AI agents can convert game specifications into playable browser games, then evaluates the delivered application by actually playing it.
✦ Why It Matters
Engineers can now measure whether AI coding agents produce genuinely functional applications, not just syntactically correct code.
Key Takeaways
- Evaluate coding agents on delivered application behavior, not intermediate code artifacts or test results.
- Use browser-native games as compact testbeds requiring coordinated input, state management, and visible feedback.
- Implement runtime evaluation by actually executing and interacting with deployed applications in real browsers.
- Recognize that 76.9% usable does not equal 20.2% excellent—minimum playability differs from complete requirement satisfaction.
Full Summary
Existing benchmarks for evaluating coding agents focus on source code quality, repository tests, or intermediate execution traces rather than the actual delivered application. WebGameBench addresses this gap by introducing a requirement-to-application benchmark using browser-native games as test cases.
Each task provides a frozen Structured WebGame Specification that agents must convert into a playable browser game. The benchmark builds each generated artifact, deploys it via a unified protocol, and uses a runtime evaluator that interacts with the game in a real browser, assigning labels: EXCELLENT, USABLE, or UNUSABLE.
Across 111 tasks and 12 coding agents, the best configuration achieved 76.9% usable rate but only 20.2% excellent rate. Human gameplay review validated the runtime labels, confirming the benchmark's reliability.