Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 26·5 min read
When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs
TL;DR
Multi-agent workflows using large language models (LLMs) face challenges in stability during joint training with reinforcement learning (RL). The study compares Shared-Policy training, where all roles share one policy, with Isolated-Policy training, where each role has its own parameters.
✦ Why It Matters
Engineers can optimize multi-agent LLM workflows by choosing appropriate policy-sharing strategies based on specific tasks and workflows.
Key Takeaways
- Assess the specific workflow before implementing multi-agent RL.
- Consider isolated policies for potentially higher accuracy in LLM tasks.
- Monitor for performance drops when using isolated policy training.
- Understand the role of gradient dynamics in your model's training process.
How It Works
The study reveals that in Isolated-Policy training, parallel agents amplify gradients for the same role, leading to performance degradation in certain workflows. In contrast, Shared-Policy training can cause the dominant role to overshadow others, resulting in different failure patterns based on the task and workflow.