Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 28·8 min read
Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Models
TL;DR
Large language models (LLMs)—AI systems trained on vast text data to generate human-like responses—lack systematic evaluation of how susceptible they are to persuasion attempts. Researchers built a framework to measure both persuasion effectiveness (how well arguments convince LLMs) and susceptibility (which models are easiest to persuade).
✦ Why It Matters
Engineers can use this framework to assess LLM vulnerability to persuasion before deployment in sensitive applications.
Key Takeaways
- Measure LLM susceptibility to persuasion before deploying in high-stakes domains.
- Test model variants systematically to identify which architectures resist manipulation better.
- Use framework results to inform safety guidelines and adversarial training strategies.
- Monitor persuasion vulnerability as a key robustness metric alongside accuracy metrics.
Full Summary
Large language models are increasingly deployed in high-stakes applications, yet their robustness against persuasion—deliberate attempts to change outputs through rhetorical or logical arguments—remains poorly understood. Researchers developed a comprehensive evaluation framework that systematically measures two dimensions: persuasion effectiveness (how successfully crafted arguments shift model outputs) and susceptibility (which LLM variants are most vulnerable).
The methodology involves constructing persuasive prompts across multiple domains and measuring changes in model responses before and after exposure. By testing across different model sizes, architectures, and training approaches, the work identifies which design choices correlate with resistance or vulnerability to persuasion.
Results reveal significant variation in susceptibility across models, with implications for deployment safety and adversarial robustness. This framework enables researchers to benchmark LLM resilience and informs defenses against manipulation.