Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage
TL;DR
Patient safety event triage, a critical task for determining reportable clinical events, lacks reliable evaluation benchmarks. PSEBench, a new benchmark, utilizes a structured representation called clause cards to assess large language models (LLMs) in this context.
✦ Why It Matters
Engineers can leverage PSEBench to improve LLM evaluations in healthcare applications, enhancing patient safety outcomes.
Key Takeaways
- Utilize PSEBench for evaluating LLMs in healthcare contexts.
- Implement clause cards for clearer regulatory text analysis.
- Identify performance gaps in LLMs for targeted improvements.
- Enhance patient safety outcomes through better event triage.
Full Summary
Patient safety event triage involves assessing whether clinical events should be reported based on specific policies, a task traditionally performed by experts. Existing evaluation methods for large language models (LLMs) in this domain are insufficient, lacking benchmarks that capture nuanced policy reasoning and information gaps.
To address this, PSEBench was developed, employing a methodology centered on clause cards, which break down regulatory text into clear, auditable decision specifications. This approach allows for the generation of narratives with verified ground truth and supports the creation of missing or uncertain information.
PSEBench consists of 5,074 cases based on Minnesota's reportable adverse health events and provides a controlled evaluation environment for LLMs. Testing on 15 representative LLMs demonstrated consistent capability trends and highlighted actionable gaps in their performance.
These findings suggest that PSEBench can significantly enhance the reliability of LLMs in patient safety event triage.