NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
CleanPatrick: A Benchmark for Image Data Cleaning
TL;DR
Current benchmarks for image data cleaning are limited by synthetic noise and narrow studies, hindering real-world application. CleanPatrick is introduced as a large-scale benchmark using the Fitzpatrick17k dermatology dataset, which includes 496,377 annotations to identify various data issues.
✦ Why It Matters
Engineers can leverage CleanPatrick to evaluate and improve their image data cleaning methods systematically.
Key Takeaways
- Utilize CleanPatrick for benchmarking image data cleaning methods.
- Focus on self-supervised techniques for near-duplicate detection.
- Explore classical methods for effective off-topic sample identification.
- Address challenges in detecting implausible labels in medical datasets.
Full Summary
Robust machine learning models require clean data, yet existing benchmarks for image data cleaning often rely on artificial noise or limited human studies, which restrict their applicability. CleanPatrick is a newly developed benchmark that utilizes the Fitzpatrick17k dermatology dataset, comprising 496,377 binary annotations collected from 933 medical crowd workers.
The study identifies off-topic samples (4%), near-duplicates (21%), and label errors (32%), employing an aggregation model based on item-response theory followed by expert review to establish high-quality ground truth. CleanPatrick formalizes the detection of data issues as a ranking task, using standard metrics that reflect real-world audit processes.
Benchmarking various methods, it was found that self-supervised representations are particularly effective for near-duplicate detection, while classical techniques perform competitively in off-topic detection under limited review budgets. However, detecting implausible labels remains a challenge, especially in fine-grained medical classification.
By providing both the dataset and evaluation framework, CleanPatrick facilitates systematic comparisons of image-cleaning strategies.