EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Sep 3·5 min read
GPT-6 Astra makes major gains in the Artificial Analysis Coding Agent Index
TL;DR
GPT-6 Astra achieves notable advancements in the Artificial Analysis Coding Agent Index, matching Fable 5's performance at a lower cost. However, its pricing is significantly higher in the Intelligence Index compared to its predecessor, GPT-5.6 Sol.
✦ Why It Matters
Evaluate your current AI model costs and consider switching to GPT-6 Astra for coding tasks.
Key Takeaways
- Consider GPT-6 Astra for cost-effective coding tasks.
- Be aware of the higher pricing in the Intelligence Index.
- Leverage the improved token efficiency for better performance.
- Monitor mixed results in various evaluations for informed decisions.
Full Summary
GPT-6 Astra has made notable strides in the Artificial Analysis Coding Agent Index, scoring 67, which is comparable to Fable 5 but at less than half the cost. It demonstrates a 70% improvement in token efficiency compared to its predecessor, GPT-5.6 Sol, using significantly fewer tokens for similar tasks.
In the Intelligence Index, it matches GPT-5.6 Sol with a score of 61 but is 75% more expensive per task due to a 2.5x price increase. Despite this, GPT-6 Astra shows a marked reduction in hallucination rates, dropping from 92% to 51%, while also increasing accuracy.
The model's performance in long-horizon knowledge work has improved, but it has regressed in some other evaluations, indicating mixed results across different capabilities. Overall, GPT-6 Astra sets new benchmarks for cost efficiency in coding tasks but faces challenges in the intelligence domain due to its pricing.