Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 27·6 min read
TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews
TL;DR
LLM-generated peer reviews are fluent but often contain hidden deficiencies that existing detection methods miss because they either identify authorship without assessing quality or use features designed for human reviews. TADDLE, a tool-augmented agent, detects specific defect types in LLM-generated reviews by combining language models with external tools.
✦ Why It Matters
Conference organizers and editors can now identify specific quality failures in LLM reviews to maintain peer review integrity.
Key Takeaways
- Implement tool-augmented agents to detect domain-specific LLM failures beyond generic quality metrics.
- Design detection systems at defect-type granularity rather than binary classification for actionable insights.
- Apply TADDLE methodology to other fluent-but-flawed LLM outputs like summaries or technical documentation.
- Use structured reasoning with external tools to overcome LLM fluency masking underlying quality issues.
Full Summary
Academic peer review increasingly relies on large language models (LLMs), yet their generated reviews often contain subtle quality problems masked by fluent writing and proper structure. Existing approaches either classify whether a review was written by an LLM versus a human without evaluating quality, or apply quality metrics designed for human-authored reviews that miss LLM-specific failure modes.
TADDLE addresses this gap by implementing a tool-augmented agent—an AI system enhanced with access to external tools and structured reasoning—that identifies specific defect categories within LLM-generated reviews. The approach combines language model reasoning with targeted tools to decompose reviews into individual defect types rather than producing single quality scores.
By operating at the defect-type level, TADDLE enables reviewers and conference organizers to understand exactly what quality issues exist in LLM-generated content. This granular detection supports both automated filtering and targeted human review of problematic submissions.