Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 28·5 min read
Refusal Is Not Robustness: Auditing Confident Fabrication in Large Language Models on a Provably Uninformative Clinical Pain Speech Transcript
TL;DR
Seven large language models were evaluated on a clinical pain speech transcript dataset, revealing that while some models abstained from unsupported predictions, others fabricated confident pain scores despite lacking evidence. Notably, Gemini 2.5 Flash and Llama 3.1 8B exhibited high rates of confident fabrication.
✦ Why It Matters
Evaluate your LLM's performance on clinical data to ensure reliable outputs in sensitive applications.
Key Takeaways
- Assess LLMs' performance on clinical datasets before use.
- Be cautious of models generating confident but unsupported predictions.
- Implement cooperative prompting to improve model abstention rates.
- Monitor fabrication rates in LLM outputs for clinical applications.
Full Summary
Large language models (LLMs) often struggle to distinguish between appropriate abstention from answering and making unsupported predictions, particularly in clinical settings. This study assessed seven LLMs using the TAME Pain speech corpus, which included 5,750 utterances without pain information and 1,294 with explicit pain ratings.
While acoustic features allowed for some pain recovery (AUC 0.622), transcript-based predictions were nearly random (AUC 0.489). Under cooperative prompting, most models abstained correctly on no-signal transcripts, but their performance varied significantly under authority-framed prompts.
Notably, Gemini 2.5 Flash and Llama 3.1 8B produced confident pain scores with fabrication rates of 0.53 and 0.76, respectively, compared to lower rates in other models. These findings highlight the challenges of relying on LLMs for clinical assessments, especially when they generate confident but unsupported outputs.