Announcing MCP Toolbox Java SDK v1.0: Agentic data access for the enterprise
cloud.google.com·6d ago
Cloud & Infrastructure
The New Stack·Aug 27·5 min read
Aider, Claude Code, and OpenClaw ran an identical model. Token use varied 70-fold.
TL;DR
Aider, Claude Code, and OpenClaw utilized the same AI coding model but exhibited a staggering 70-fold difference in token usage. This highlights the significant impact of the harness software on AI performance.
✦ Why It Matters
Evaluate your AI coding agent's harness to optimize token usage and improve performance today.
Key Takeaways
- Analyze the harness's prompt structure to minimize token overhead.
- Consider cache hit rates when evaluating AI coding agents.
- Focus on cost per successful task rather than raw token counts.
- Benchmark different harnesses to identify the most cost-effective option.
Full Summary
Three benchmarking studies revealed that the software harness, which manages AI coding agents, can greatly influence token usage and costs. In a June benchmark, token counts for solving tasks varied from 3,500 for Aider to 292,000 for OpenClaw, indicating that the harness's overhead significantly affects efficiency.
Composio's August study showed that while Claude Code had a high cost per successful task, it had a low cache hit rate, leading to higher expenses compared to competitors. The findings suggest that developers should focus on the harness's prompt structure and token management to optimize costs.
The results emphasize that harness choice can lead to cost differences comparable to those seen in model pricing, making it crucial for enterprises to evaluate both aspects when standardizing AI platforms.