Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix
TL;DR
Deep multi-agent reinforcement learning (RL) in pricing markets faces issues like tacit cartel formation and actor-critic instability. The authors developed a continuous-time multi-agent RL benchmark to analyze these failures and proposed a partial fix through asynchrony and observation latency.
✦ Why It Matters
Engineers can apply insights on asynchrony and latency to improve multi-agent RL systems in pricing scenarios.
Key Takeaways
- Investigate failure modes in multi-agent RL systems.
- Implement asynchrony to reduce collusion in pricing models.
- Monitor trajectory-level diagnostics for signaling issues.
- Consider the trade-offs of latency in RL applications.
Full Summary
Deep multi-agent reinforcement learning (RL) can struggle in continuous-time pricing markets, leading to issues such as tacit cartel formation among agents and instability in actor-critic methods at high event rates. To investigate these problems, a continuous-time multi-agent RL benchmark was created, featuring Poisson-clocked price updates and observation latency.
The study identified two reproducible failure modes: collusion among synchronous DDPG (Deep Deterministic Policy Gradient) agents and critic divergence at specific parameters. A partial fix was proposed, where introducing asynchrony reduced the collusion index from 0.69 to 0.28, but this solution was not robust against all failure modes.
Additionally, trajectory-level diagnostics revealed significant signaling collapse during episodes. These findings highlight the complexities of multi-agent systems in dynamic environments and the need for careful design in RL applications.