NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
Stop Comparing LLM Agents Without Disclosing the Harness
TL;DR
Comparisons of large language model (LLM) agents often overlook the importance of the execution harness, which manages context and tool interactions. The authors propose the Binding Constraint Thesis, asserting that performance differences are more influenced by harness configuration than by the models themselves.
✦ Why It Matters
Engineers should ensure harness specifications are disclosed to accurately assess LLM agent performance.
Key Takeaways
- Disclose harness specifications in LLM evaluations.
- Recognize that harness changes can impact performance more than model changes.
- Adopt a harness-aware evaluation framework for accurate comparisons.
- Be cautious of misleading performance claims without harness context.
How It Works
The authors formalize the harness as a controller in a closed-loop system, where the LLM acts as a stochastic policy. This framework helps explain how small adjustments in the harness can lead to substantial performance changes, highlighting the need for careful evaluation.