Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 26·5 min read
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer
TL;DR
LLM agents often struggle with tool selection due to fixed shortlist sizes, which can either overwhelm or under-inform them. The authors developed a chance-corrected metric called Bits-over-Random (BoR) to evaluate the effectiveness of tool shortlists.
✦ Why It Matters
Engineers can implement adaptive tool selection strategies to improve LLM performance in real-world applications.
Key Takeaways
- Consider using adaptive shortlists for LLM agents.
- Evaluate tool selection effectiveness with the BoR metric.
- Optimize the number of tools shown based on query difficulty.
- Implement reinforcement learning to enhance tool selection strategies.
Full Summary
LLM (Large Language Model) agents require effective tool selection to perform tasks accurately, but existing systems typically use a fixed number of tools, which can lead to inefficiencies. The study introduces a new evaluation metric, Bits-over-Random (BoR), which assesses whether the success of tool selection is better than random chance at various shortlist depths.
By applying BoR across multiple benchmarks, the researchers developed a simple reinforcement learning (RL) agent to optimize the number of tools shown per query. Findings indicate that while a fixed shortlist of 5 tools performs well overall, it fails on harder queries, whereas the BoR agent, which adapts the shortlist size, achieves higher success rates.
For instance, it matched nearly the same coverage as showing 50 tools while only presenting an average of 7. These results suggest that adaptive tool selection can enhance LLM performance, particularly in complex scenarios.