Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 3·5 min read
Disease Burden over Skin Tone: Decomposing the Dermatology-AI Generalization Gap
TL;DR
Dermatology AI models, primarily trained on light-skinned images, struggle with generalization to diverse skin tones and disease distributions. This study reveals that disease-distribution shift is a more significant factor than skin-tone underrepresentation in model performance.
✦ Why It Matters
Focus on improving model training with diverse datasets to enhance dermatology AI performance today.
Key Takeaways
- Evaluate AI models on diverse datasets to assess generalization.
- Consider disease distribution shifts when training dermatology AI.
- Utilize dermatology foundation models for better performance.
- Incorporate a minimum of ten labeled examples per category.
Full Summary
Many dermatology AI models are trained on datasets that predominantly feature light-skinned individuals and focus on specific diseases like cancer. This research investigates the generalization gap of these models when applied to diverse populations with varying skin tones and disease distributions.
The study evaluates several models, including a cancer-trained ResNet-50 and dermatology foundation models like DermLIP and MONET, using two distinct datasets: Diverse Dermatology Images (DDI) and Skin Condition Image Network (SCIN). Results indicate that the shift in disease distribution has a more substantial impact on model accuracy than skin tone, with a drop in balanced accuracy from 0.62 to 0.21 for the cancer baseline when faced with unfamiliar conditions.
Furthermore, the study shows that starting from dermatology foundation models, only about ten labeled examples per clinical category can significantly enhance performance. The authors provide evaluation protocols and code to facilitate reproducibility in dermatology AI research.