EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 28·5 min read
Terminal-Bench-Science: Evaluating AI agents on scientific research workflows
TL;DR
Terminal-Bench-Science is a new benchmark developed by Stanford researchers to evaluate AI agents on real scientific workflows, aiming to enhance their utility as research assistants. The initial release features 70 tasks across various scientific disciplines, with the top model achieving a 30% resolution rate.
✦ Why It Matters
Explore how AI can assist in your specific research workflows today.
Key Takeaways
- Participate in the open contribution process for new tasks.
- Utilize the benchmark to evaluate AI models in your research.
- Stay updated on AI capabilities through regular benchmark releases.
- Engage with the scientific community on Discord and GitHub.
Full Summary
Terminal-Bench-Science is a benchmark designed to assess AI agents' capabilities in real scientific workflows, emphasizing contributions from practicing scientists. Developed by Stanford University researchers, it includes 70 tasks from life, physical, Earth, mathematical, and engineering sciences.
The benchmark evolves continuously, allowing for regular updates and contributions from the scientific community. The leading AI model, Claude Opus 5, achieved a 30% resolution rate, indicating the benchmark's challenging nature.
This initiative aims to create AI agents that can assist researchers in complex tasks, thereby accelerating scientific discovery. The benchmark's open-source nature encourages collaboration and ensures that it reflects the priorities of the scientific community.