Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 28·6 min read
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
TL;DR
Large language models (LLMs) often give confidence scores that don't match their actual accuracy—a problem called miscalibration. Researchers discovered that how you ask an LLM for confidence (the protocol) dramatically changes its calibration, meaning the same model gives different reliability estimates depending on phrasing.
✦ Why It Matters
Engineers must test multiple confidence elicitation protocols rather than assuming one approach works universally across deployments.
Key Takeaways
- Test multiple prompt phrasings when extracting confidence from LLMs; single protocols are unreliable.
- Calibration metrics vary significantly across different confidence elicitation methods on identical models.
- Document which protocol was used when reporting LLM confidence scores in production systems.
- Validate confidence estimates empirically for your specific use case rather than trusting defaults.
Full Summary
LLMs are increasingly used for high-stakes decisions, but their confidence estimates—how certain they claim to be about answers—often don't reflect true accuracy. This miscalibration (gap between stated confidence and actual correctness) creates risk when deploying these systems.
Researchers investigated whether the phrasing and structure of prompts asking for confidence affect calibration quality. They tested multiple elicitation protocols (different ways of requesting confidence scores) on the same LLM and measured calibration metrics like expected calibration error.
Results showed substantial performance variation across protocols, indicating that confidence calibration is highly sensitive to how questions are framed. This finding challenges the assumption that confidence extraction is protocol-agnostic and has direct implications for practitioners building reliable AI systems.