Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·28d ago·5 min read
How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
TL;DR
Frontier language models, like GPT-5.6-Sol, show significantly improved performance on physics benchmarks after expert re-evaluation, revealing that prior low scores were often due to flawed grading criteria. Corrected evaluations indicate these models can solve well-posed physics problems much more effectively than previously reported.
✦ Why It Matters
Consider revising your evaluation benchmarks to better reflect the capabilities of advanced AI models.
Key Takeaways
- Review and correct existing evaluation benchmarks for AI models.
- Engage domain experts to assess model performance accurately.
- Focus on well-posed problems to evaluate AI capabilities.
- Consider developing more challenging benchmarks for future assessments.
How It Works
Experts audited the models' responses by comparing them against corrected reference solutions and clarifying ambiguous questions, leading to more accurate evaluations.