Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 19·5 min read
LLMs for Medical Consultation Are Evaluated Too Late: The Preformulation Gap
TL;DR
Large language models (LLMs) for medical consultation are often assessed too late in the process, missing the initial vague concerns of patients. Evaluations should focus on first-contact behavior rather than just diagnostic accuracy.
✦ Why It Matters
Evaluate LLMs based on initial patient interactions to improve their effectiveness in medical consultations.
Key Takeaways
- Assess LLMs during initial patient interactions, not just after diagnosis.
- Focus on self-care advice in early consultation stages.
- Implement structured handoff summaries to improve communication.
- Use adaptive simulations to better evaluate LLM performance.
Full Summary
Large language models (LLMs) are increasingly used for medical consultations, but their evaluation typically occurs after a clinical problem is clearly defined. This study assessed three API models using four physician-authored, multi-turn vignettes under two conditions: baseline and entry-to-care instructions.
A total of 24 fixed-script transcripts were generated, along with 12 transcripts from adaptive standardized-patient simulations. Results showed that self-care advice was provided in 9 out of 12 baseline cases but in none of the instruction cases.
Additionally, structured handoff summaries were present in 10 out of 12 instruction cases but absent in baseline cases. The findings suggest that the sequencing and documentation of patient interactions can be improved, but they do not guarantee the elicitation of critical information.
Therefore, the evaluation of LLMs should prioritize observable first-contact behaviors to better understand their effectiveness in real-world scenarios.