EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Sep 2·5 min read
Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
TL;DR
FrontierHarness Eval reveals significant cost variations in task execution, with a 17x difference in cost per pass across nine harnesses. The evaluation highlights the importance of understanding both quality and cost in software engineering tasks.
✦ Why It Matters
Evaluate your current harnesses to optimize for cost and performance in your software engineering tasks.
Key Takeaways
- Test your harness on Runta to understand cost implications.
- Consider both quality and cost when selecting harnesses.
- Evaluate cache hit rates to minimize execution costs.
- Be aware of the significant cost variations across different harnesses.
Full Summary
FrontierHarness v1.0 was evaluated using Runta, a platform for testing AI models, focusing on terminal-based software engineering tasks. The evaluation involved running nine different harnesses, revealing that while some harnesses could complete tasks efficiently, others incurred significantly higher costs, with a maximum of $18.34 per task.
Notably, the cost per task varied by a factor of 17, emphasizing the need for careful selection of harnesses based on both performance and cost. The evaluation excluded failed attempts, which, if included, would raise the average cost to $3.24 per task.
Additionally, the cache hit rate was highlighted as a critical factor, as cached failures could still lead to high costs. This analysis underscores the necessity for engineers to consider both quality and cost when selecting tools for their projects.