NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
Data Science
Interconnects.ai·May 16·5 min read
Latest open artifacts (#21): Open model bonanza! Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1 & others. On CAISI's V4 assessment.
TL;DR
Open-source AI models from DeepSeek, Google, and others were released this month, but their capabilities relative to American frontier models remain unclear due to inconsistent evaluation methods. The Center for AI Standards and Innovation (CAISI) published an Elo-based assessment using Item Response Theory across nine benchmarks, showing open models lag by months.
✦ Why It Matters
Engineers should recognize that standardized benchmarks may underestimate open model practical capabilities compared to frontier models in real-world deployment scenarios.
Key Takeaways
- Open models show 3-7 month capability gap versus American frontier models per Epoch AI's ECI metric.
- Benchmark choice heavily influences Elo rankings; CTF-Archive-Diamond and PortBench drive large score differences.
- Standardized benchmark setups underestimate practical capabilities; real-world coding uses integrated harnesses, not fixed token budgets.
- Evaluate models using multiple frameworks and real-world task setups, not just standardized benchmarks, for accurate capability assessment.
Full Summary
This month saw major releases from open-source AI labs including DeepSeek V4, Gemma 4, and others, triggering a comparative assessment by CAISI. CAISI evaluated these models using Item Response Theory (IRT), a statistical technique that enables fair comparison across different benchmark sets by accounting for question difficulty.
They tested nine benchmarks including CTF-Archive-Diamond, PortBench, and ARC-AGI-2. The analysis revealed open models trail American frontier models, with the gap widening over time.
However, both CAISI and competing evaluation frameworks like Epoch AI's ECI use simplified testing environments that don't match real-world model deployment. For example, coding tasks use basic bash access with fixed token budgets rather than integrated development environments like Claude Code, potentially underestimating open model capabilities on practical tasks.