Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 28·5 min read
How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models
TL;DR
A systematic evaluation of verbal probability expressions across 19 large language models reveals that while they align closely with human benchmarks, they exhibit a consistent upward bias for negative terms like 'unlikely'. This study highlights the models' ability to mirror human uncertainty perception, with implications for human-model interactions in probabilistic contexts.
✦ Why It Matters
Evaluate how your models interpret negative probability expressions to improve communication accuracy.
Key Takeaways
- Test LLMs with verbal probability expressions to assess their accuracy.
- Be aware of biases in LLM responses, especially for negative terms.
- Use explanation elicitation to understand model reasoning better.
- Consider model divergence when interpreting probabilistic language outputs.
How It Works
The study employed a word-to-number mapping task to evaluate how LLMs interpret verbal probability expressions. By comparing model outputs to established human benchmarks, researchers assessed consistency and biases in responses.
The forced single-number response condition aimed to quantify model interpretations, while explanation elicitation sought to uncover reasoning behind those interpretations.