Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 26·5 min read
PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection
TL;DR
Large language models (LLMs) used for vulnerability detection may perform inconsistently based on how they are prompted. PromptAudit is a framework that evaluates the sensitivity of LLMs to different prompting strategies while keeping other variables constant.
✦ Why It Matters
Engineers should consider prompt sensitivity when deploying LLMs for vulnerability detection to optimize performance.
Key Takeaways
- Use PromptAudit to evaluate prompt sensitivity in LLMs.
- Standard chain-of-thought prompting generally yields the best results.
- Be cautious with adaptive prompting strategies; they may reduce recall.
- Understand the model's sensitivity to prompts for better deployment outcomes.
Full Summary
Vulnerability detection using large language models (LLMs) is becoming more common, but their performance can vary significantly depending on how they are prompted. PromptAudit was developed as a controlled evaluation framework to assess this prompt sensitivity by fixing the dataset and parsing while varying the prompting strategies.
The study tested five prompting strategies across five open-weight models on 1,000 Common Vulnerabilities and Exposures (CVEs), which included 6,074 code samples from 16 programming languages. Key metrics evaluated included accuracy, recall, and effective F1 score.
Findings revealed that standard chain-of-thought prompting provided the best overall performance, while few-shot prompting showed benefits for certain models. Conversely, adaptive chain-of-thought prompting often reduced recall, and self-consistency methods led to excessive abstention, negatively impacting performance.
These insights highlight the importance of understanding prompt sensitivity as a critical factor in the deployment of LLMs for vulnerability detection.