EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 16·5 min read
I checked 30 frontier model cards. Here are the benchmarks labs report
TL;DR
A review of 30 model cards reveals insights into benchmark reporting practices across organizations, highlighting the importance of publication dates and configuration counts. The findings indicate a saturation in reporting, suggesting a need for more diverse benchmarks.
✦ Why It Matters
Review your model card reporting practices to ensure diverse and up-to-date benchmarks.
Key Takeaways
- Track benchmark publication dates for accurate reporting.
- Monitor saturation trends in benchmark reporting.
- Ensure diverse vendor contributions to benchmarks.
- Regularly update model cards to reflect new scores.
Full Summary
The analysis of 30 frontier model cards focuses on how benchmarks are reported by different organizations in the AI field. Each model card represents a unique benchmark, and the study tracks the timeline of when these benchmarks are first reported.
A notable observation is the saturation of reporting, indicated by long flat runs in the data, which suggests that many organizations are not publishing new benchmark scores. The methodology includes a cumulative count of benchmarks and a separate score tracking system that reflects the values reported verbatim from cited documents.
The results show that a diverse range of organizations contributes to benchmark reporting, but the impact of individual vendor contributions can vary significantly. This analysis underscores the importance of model card publication dates in understanding the landscape of AI benchmarks.