We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Sep 25·5 min read
Measuring the performance of our models on real-world tasks
TL;DR
Prior evaluation benchmarks didn't measure AI model performance on economically valuable real-world work across diverse occupations. OpenAI created GDPval, a new evaluation framework assessing model capabilities across 44 occupations on tasks with measurable economic impact.
✦ Why It Matters
Engineers can use GDPval to benchmark AI models against real economic tasks and identify high-impact deployment opportunities.
Key Takeaways
- Use GDPval to evaluate AI models on actual job tasks rather than abstract benchmarks for deployment decisions.
- Identify which occupations show highest AI productivity gains to prioritize development and deployment efforts.
- Measure model performance against human baselines on economically meaningful work to assess practical value.
- Compare model capabilities across 44 occupations to find domain-specific strengths and capability gaps.
Full Summary
Existing AI evaluation benchmarks often test narrow capabilities through standardized tests rather than measuring performance on actual work tasks that generate economic value. OpenAI developed GDPval, an evaluation framework designed to assess large language models and AI systems across 44 different occupations on real-world economically valuable tasks.
The methodology involves selecting tasks from actual job domains that have clear economic impact and measurable outcomes, then evaluating model performance against human baselines. GDPval measures both task completion rates and quality of outputs across diverse fields, providing quantitative data on where AI adds practical value.
This approach bridges the gap between academic benchmarks and practical deployment by grounding evaluation in actual economic productivity. The framework enables researchers and engineers to identify which occupational domains benefit most from AI assistance and where capability gaps remain.