Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 17·5 min read
Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions
TL;DR
Three AI chatbots—Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5—were evaluated for their ability to retrieve relevant medical studies. ChatGPT outperformed the others, retrieving 63.1% of relevant studies on average, highlighting the influence of user role and sample size on retrieval effectiveness.
✦ Why It Matters
Consider using ChatGPT for clinical study retrieval to enhance research accuracy and efficiency.
Key Takeaways
- Use ChatGPT for better retrieval of relevant medical studies.
- Be aware of user role impact on chatbot performance.
- Consider sample size when evaluating study relevance.
- Explore different LLMs for specific clinical inquiries.
Full Summary
Large language model (LLM) chatbots are increasingly utilized to answer clinical questions, but their effectiveness in retrieving relevant studies has not been thoroughly assessed. This study compared three LLMs—Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5—using clinical questions derived from the Cochrane Database of Systematic Reviews.
Each chatbot was prompted under three user roles: patient, clinician, and researcher, with a total of 720 responses analyzed. Results showed that ChatGPT had the highest recall of relevant studies at 63.1%, compared to 37.0% for Claude and 17.3% for Gemini.
Additionally, the researcher role yielded better retrieval rates than the clinician or patient roles. Notably, sample size was the only significant predictor of study retrieval, indicating a bias towards larger clinical trials.
These findings suggest that while LLMs can assist in study retrieval, their performance varies significantly based on the model used and the user's role.