This week’s news from Zed, Anthropic, and OpenRouter shows why better harnesses matter more than better models
thenewstack.io·13h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents
TL;DR
Tool-using agents based on large language models (LLMs) face significant safety risks, especially during multi-turn interactions. To address this, researchers developed MT-AgentRisk, a benchmark for evaluating multi-turn safety, and ToolShield, a self-exploration defense mechanism.
✦ Why It Matters
Engineers can implement ToolShield to enhance the safety of multi-turn interactions in tool-using agents.
Key Takeaways
- Adopt the MT-AgentRisk benchmark for evaluating agent safety.
- Utilize ToolShield to autonomously test new tools.
- Monitor attack success rates in multi-turn interactions.
- Prioritize safety measures in AI tool usage.
How It Works
ToolShield operates by allowing agents to autonomously generate test cases when they encounter new tools. It executes these cases to observe the effects on safety, distilling experiences to improve future interactions.