Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 26·5 min read
Parameter-Efficient VLMs for Gastrointestinal Endoscopy: Medical Image Generation and Clinical Visual Question Answering
TL;DR
Gastrointestinal (GI) endoscopy AI systems face challenges due to limited annotated data and privacy concerns. A dual-pipeline Parameter-Efficient Fine-Tuning (PEFT) model was developed, utilizing the Florence-2 vision-language model for Visual Question Answering (VQA) and Low-Rank Adaptation (LoRA) for synthetic image generation.
✦ Why It Matters
Engineers can leverage this model to enhance AI applications in medical imaging while ensuring patient privacy.
Key Takeaways
- Implement PEFT to enhance model interpretability and reduce costs.
- Utilize Florence-2 for effective medical Visual Question Answering.
- Adopt LoRA for generating high-quality synthetic medical images.
- Focus on private datasets for better model performance.
Full Summary
Gastrointestinal (GI) endoscopy AI systems struggle with a lack of annotated data and strict privacy regulations, which hinder their effectiveness in clinical settings. To address these issues, a dual-pipeline Parameter-Efficient Fine-Tuning (PEFT) model was created, combining the Florence-2 vision-language model for medical Visual Question Answering (VQA) and Low-Rank Adaptation (LoRA) with Stable Diffusion 2.1 for generating synthetic GI images.
The research utilized the Kvasir-VQA dataset, achieving a ROUGE-1 score of 0.92 and a BLEU score improvement from 0.08 to 0.24. Fine-tuning on private datasets consistently outperformed public datasets, demonstrating the model's robustness.
The rank-4 LoRA synthesis achieved a fidelity score of 0.290 and reduced computational costs by almost 90%. Compared to existing models, this approach showed superior image-text coherence, indicating its potential to enhance VQA and synthetic data generation in clinical AI applications.