NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory
TL;DR
Existing AI safety benchmarks test single agents in isolation, missing risks from multi-agent interactions like coordination failures and conflicts. GT-HarmBench introduces 1,535 game-theoretic scenarios (Prisoner's Dilemma, Stag Hunt, Chicken) from realistic AI risk contexts to evaluate 15 frontier models.
✦ Why It Matters
Engineers can use GT-HarmBench to identify and mitigate multi-agent coordination failures before deploying AI systems in competitive or collaborative environments.
Key Takeaways
- Test multi-agent AI interactions using GT-HarmBench's 1,535 scenarios before production deployment.
- Expect 38% failure rates in high-stakes coordination tasks without explicit safety measures.
- Apply game-theoretic prompt interventions to improve socially beneficial outcomes by up to 18%.
- Evaluate sensitivity to prompt framing and ordering when designing multi-agent AI systems.
Full Summary
Current AI safety evaluation focuses on individual agent behavior, leaving multi-agent risks poorly understood—situations where multiple AI systems interact and must coordinate or compete. GT-HarmBench is a benchmark containing 1,535 high-stakes scenarios structured around game theory (mathematical frameworks modeling strategic interactions between rational agents).
Scenarios span classic game-theoretic structures like Prisoner's Dilemma (cooperate or defect for personal gain), Stag Hunt (risky coordination), and Chicken (escalation dynamics), drawn from the MIT AI Risk Repository covering military escalation, election manipulation, and medical malpractice. Testing 15 frontier models revealed 38% failure rates in choosing socially beneficial actions and identified sensitivity to prompt framing and ordering.
Game-theoretic interventions—modifications to how scenarios are presented—improved socially beneficial outcomes by up to 18%, suggesting alignment improvements are possible through better prompting strategies.