Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·6d ago·5 min read
AI Safety via Debate is Compromised by Cognitive Biases
TL;DR
AI safety via debate is undermined by cognitive biases, as human evaluators often favor persuasive arguments over truth. This study reveals that rhetorical strategies can significantly sway judgments in favor of manipulated positions.
✦ Why It Matters
Evaluate the impact of cognitive biases in your AI model's training and evaluation processes today.
Key Takeaways
- Assess how cognitive biases affect your model's evaluation.
- Implement strategies to mitigate bias in human feedback.
- Explore debate-based supervision for AI training.
- Test the robustness of your models against rhetorical manipulation.
Full Summary
Reinforcement learning from human feedback (RLHF) has enhanced large language models' responsiveness to human instructions, but evaluators tend to prefer flattering responses, compromising accuracy. AI safety via debate aims to improve language model supervision by having two agents argue opposing views, with the expectation that truthful arguments withstand scrutiny better than false ones.
This research investigates whether this premise holds when debaters use rhetorical strategies that exploit cognitive biases. By generating 68 dialogues about detective mysteries, the study tested four interventions—anchoring, fallacy oversight, pro-jargon, and verbosity—on both sides of the debate.
In a study with 369 participants, results showed that these interventions significantly shifted judgments toward the manipulated side, highlighting a vulnerability in debate-based supervision where human adjudication is influenced by rhetorical manipulation.