NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
TL;DR
LLMs deployed in strategic settings (auctions, bidding, games) lack reliable evaluation methods—fixed benchmarks saturate and don't generalize to real deployments. GENSTRAT uses procedurally generated two-player card games with a capability-profile framework decomposing performance across six axes (state space, temporal depth, information sensitivity, opponent modeling, risk, brittleness).
✦ Why It Matters
Engineers deploying LLMs in competitive settings can now diagnose specific strategic weaknesses beyond aggregate scores to predict real-world failure modes.
Key Takeaways
- Use procedurally generated games instead of fixed benchmarks to avoid saturation and contamination in LLM evaluation.
- Decompose strategic competence across six axes to identify specific weaknesses masked by overall rankings.
- Measure jaggedness to detect when models fail unpredictably on strategically similar tasks.
- Expect models with identical scores to differ qualitatively in consistency and capability profiles before deployment.
Full Summary
Large language models are increasingly used as economic agents in competitive settings like auctions and marketplaces, but predicting their behavior remains difficult. Existing benchmarks use fixed canonical games that become less useful as models improve and don't reflect the complexity of real-world deployments.
GENSTRAT addresses this by generating a distribution of two-player zero-sum imperfect-information card games (games where players have incomplete knowledge and opposing interests). The framework decomposes model competence across six dimensions: state space complexity, temporal depth (planning horizon), information sensitivity, opponent modeling ability, risk tolerance, and brittleness (consistency across similar games).
A jaggedness metric quantifies unpredictable performance swings between strategically similar games. Evaluating nine frontier and open-weight LLMs across 36,000+ matches revealed that models with nearly identical overall tournament scores exhibit markedly different capability profiles and local volatility—notably, gpt-5 and claude showed higher jaggedness than gemini-3.1-pro despite comparable overall strength.