Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 25·8 min read
ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
TL;DR
Multi-agent reinforcement learning (MARL) struggles when environmental rewards are sparse, making it hard for multiple agents to learn simultaneously without destabilizing each other's strategies. ARMS (Automatic Reward-shaping in Multi-agent Systems) automatically generates dense reward signals from sparse rewards using trajectory ranking while preserving Nash equilibria—the stable strategic outcomes in game theory.
✦ Why It Matters
Engineers can apply ARMS to accelerate multi-agent learning in sparse-reward domains while maintaining strategic stability and equilibrium properties.
Key Takeaways
- ARMS automatically generates dense rewards from sparse signals using trajectory ranking without manual reward engineering.
- Game-theoretic equilibrium preservation ensures shaping rewards maintain stable multi-agent strategy profiles.
- Oscillatory learning failure occurs with limited exploration; increase exploration to stabilize convergence.
- Share reward-shaping parameters across agents to improve efficiency in multi-agent systems.
Full Summary
In multi-agent reinforcement learning, agents learn policies simultaneously in shared environments, but sparse rewards (infrequent feedback signals) combined with non-stationary opponents create severe learning bottlenecks. Reward shaping—adding intermediate reward signals to guide learning—helps single-agent systems but risks breaking game-theoretic equilibria (stable strategy profiles) in multi-agent settings.
ARMS addresses this by learning dense shaping rewards from sparse environmental signals through trajectory ranking, a self-supervised approach that ranks past experience sequences. The key innovation reformulates policy invariance using conditional best-response reasoning, proving that under certain conditions, shaping rewards preserve each agent's best-response set (optimal strategies against fixed opponents) and consequently preserve Nash equilibria.
ARMS alternates between policy learning and reward learning while sharing parameters across agents for efficiency. Experiments in partially observable multi-agent pathfinding demonstrated improved sampling efficiency under increasing sparsity and agent counts, generalization to unseen environments, and identified a MARL-specific oscillatory failure mode caused by limited exploration and coupled policy-reward dynamics, stabilized by increased exploration.