Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·23d ago·5 min read
Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
TL;DR
A comparison was made between Retrieval-Augmented Generation (RAG) and long-context input methods for clinical reasoning using Electronic Health Records (EHRs). RAG was implemented to enhance the model's ability to retrieve relevant information dynamically.
✦ Why It Matters
Implementing Retrieval-Augmented Generation can improve the accuracy of clinical decision support tools in real-time applications.
Key Takeaways
- Explore RAG for clinical data extraction tasks.
- Evaluate the performance of different language models in your applications.
- Utilize fewer tokens for improved efficiency in processing EHRs.
- Implement findings to enhance clinical decision-making processes.
Full Summary
This research investigates the effectiveness of retrieval-augmented generation (RAG) compared to long-context input for clinical reasoning tasks involving electronic health records (EHRs). Three tasks were defined: extracting imaging procedures, generating antibiotic use timelines, and identifying key diagnoses during hospitalizations.
Using real clinical notes from a US academic health system, the study evaluated three large language models—GPT-5.4-mini, Mistral Medium 3, and DeepSeek V3.1—across varying context lengths. Results showed that RAG outperformed recent-note inputs and long-context methods, achieving an F1 score improvement of 0.17-9.83 across all models for imaging procedures.
Similar enhancements were noted for antibiotic timelines, indicating RAG's efficiency in processing clinical data with fewer than 8,000 tokens. These findings suggest that RAG can streamline clinical reasoning tasks, making it a valuable tool for healthcare professionals.