We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·Jun 30·5 min read
Featuring Every Eval Ever Results on Hugging Face Model Pages
TL;DR
AI evaluation results were previously scattered and inconsistent, making it difficult to compare model capabilities. The EvalEval Coalition and Hugging Face launched Every Eval Ever (EEE) to standardize reporting using a unified JSON schema.
✦ Why It Matters
Engineers and researchers can now access a standardized dataset of evaluation results, improving model comparison and selection.
Key Takeaways
- Utilize the unified JSON schema for consistent reporting.
- Access 229,000 evaluation results for informed decision-making.
- Contribute to the GitHub repository for ongoing improvements.
- Leverage standardized data to enhance model comparisons.
Full Summary
AI evaluation results are crucial for assessing model performance, yet they are often found in various formats across papers, leaderboards, and logs, leading to inconsistencies. To address this, the EvalEval Coalition introduced Every Eval Ever (EEE) in February 2026, alongside Hugging Face's Community Evals, which aims to decentralize benchmark score reporting.
EEE employs a single JSON schema for evaluation results, accommodating data from diverse sources like harness logs and leaderboard scrapes. Feedback from researchers shaped this schema, ensuring it meets the needs of the community.
Since its launch, the datastore on Hugging Face has amassed approximately 229,000 evaluation results from over 22,000 models and 2,200 benchmarks, sourced from 31 different reporting formats. This consolidation not only saves significant costs associated with reproducing evaluations but also fosters greater transparency and trust in AI model assessments.