Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge
TL;DR
Multi-agent debate systems often neglect the quality of intermediate reasoning, focusing solely on final answers. This study introduces a framework that uses log-probability distributions and an LLM-as-judge to evaluate reasoning quality in debates.
✦ Why It Matters
Engineers can leverage confidence signals to enhance the evaluation of multi-agent debate systems and improve reasoning quality.
Key Takeaways
- Utilize log-probabilities to assess reasoning quality in debate systems.
- Implement LLM-as-judge for scoring agent arguments effectively.
- Focus on the Constructor agent for better confidence alignment.
- Adopt a four-phase confidence trajectory in evaluations.
How It Works
The framework pairs a two-agent debate architecture, where the Constructor generates arguments and the Auditor critiques them. An LLM-as-judge evaluates the reasoning quality based on instruction adherence, justification, and evidence grounding.
By analyzing log-probabilities of reasoning tokens, the system can track confidence levels and correlate them with external evaluations, revealing insights into the reasoning process.