Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 21·5 min read
Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening
TL;DR
Medical image screening models trained on unlabeled data (self-supervised learning) are judged only on accuracy, but safety-critical tasks like diabetic retinopathy grading require models to recognize and reject uncertain predictions for human review. Researchers investigated how SSL pretraining duration affects calibrated confidence—a model's ability to accurately estimate when it should abstain (refuse to predict) rather than guess.
✦ Why It Matters
Engineers deploying medical screening systems can use SSL pretraining length as a tunable parameter to balance detection accuracy with safe abstention, reducing misdiagnosis risk.
Key Takeaways
- Measure calibrated confidence alongside accuracy when evaluating medical screening models.
- Vary SSL pretraining duration systematically to find optimal uncertainty quantification.
- Implement abstention mechanisms allowing models to defer uncertain cases to clinicians.
- Prioritize safety metrics over raw accuracy for safety-critical diagnostic applications.
Full Summary
Diabetic retinopathy screening requires automated models that not only classify disease severity accurately but also recognize their own uncertainty limits. Self-supervised learning (SSL)—a pretraining technique where models learn from unlabeled data by predicting parts of images—has become standard for medical imaging but is typically evaluated solely on downstream accuracy metrics.
This work examines a critical gap: how SSL pretraining duration affects calibrated confidence, the model's ability to express reliable uncertainty estimates and abstain from predictions on ambiguous cases. The researchers systematically varied SSL pretraining length and measured how this influences both prediction accuracy and confidence calibration on diabetic retinopathy grading tasks.
Results demonstrate that pretraining duration directly impacts whether models can safely defer uncertain cases to clinicians rather than making potentially harmful incorrect predictions. This finding has direct implications for deploying screening systems in clinical settings where safety and human oversight are paramount.