Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·12d ago·5 min read
Multi-Site Real-World Performance of Commercial AI for Pulmonary and Incidental Pulmonary Embolism Detection
TL;DR
Two FDA-cleared AI algorithms for detecting pulmonary embolism (PE) were evaluated across 17 facilities, revealing high specificity but lower sensitivity than expected, particularly for non-acute and peripheral cases.
✦ Why It Matters
Consider implementing standardized post-market surveillance for AI medical devices to ensure ongoing effectiveness.
Key Takeaways
- Evaluate AI performance against real-world data for accuracy.
- Monitor AI algorithms post-market for continued effectiveness.
- Understand limitations in detecting non-acute and peripheral emboli.
- Utilize validated LLM pipelines for accurate reference labeling.
Full Summary
Pulmonary embolism (PE) is a significant cause of cardiovascular death, yet the effectiveness of AI detection models in real-world settings is not fully understood. This study assessed two AI algorithms from Aidoc Medical BriefCase: one for triaging PE on CT pulmonary angiography (CTPA) and another for incidental PE detection on routine contrast-enhanced CTs.
The evaluation involved 30,678 and 37,191 scans, respectively, with reference labels obtained from radiology reports using a validated large language model (LLM) pipeline. The PE model demonstrated 86.8% sensitivity and 99.1% specificity, with sensitivity decreasing for smaller and non-acute emboli.
The incidental PE model showed 73.5% sensitivity and 99.8% specificity. Both models exceeded specificity benchmarks but fell short on sensitivity, highlighting the need for ongoing monitoring of AI medical devices post-approval.