NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 28·4 min read
Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
TL;DR
Researchers identified a gap in LLM evaluation benchmarks. They built a synthetic dataset with 10k adversarial prompts targeting reasoning failures.
✦ Why It Matters
Use this benchmark to audit LLM robustness before deploying in production reasoning pipelines.
Key Takeaways
- Utilize FeasiGen to identify critical tools for task completion.
- Implement feasibility-aware metrics to evaluate agent performance.
- Address high false continue rates in current agent models.
- Explore multi-agent architectures for better infeasibility recognition.
How It Works
FeasiGen constructs infeasible tasks by analyzing tool-calling traces from successful agent executions. It identifies critical tools that are essential for task completion and masks them, effectively creating tasks that agents cannot solve.
This method allows for a systematic evaluation of agents' awareness of task feasibility.