NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks
TL;DR
Public medical vision-language models (VLMs) may be contaminated by pretraining data, affecting their reported accuracy. A controlled audit was conducted using techniques like image-side near-neighbour overlap and canonical-order exchangeability.
✦ Why It Matters
Engineers should critically assess VLMs for pretraining contamination to ensure accurate model evaluations.
Key Takeaways
- Evaluate VLMs for potential pretraining data contamination.
- Use multiple detection methods for thorough audits.
- Be cautious of misleading signals from certain models.
- Ensure accurate reporting of model performance metrics.
Full Summary
Medical vision-language models (VLMs) are often evaluated on public benchmarks, but their reported accuracy assumes that the evaluation data was not included in pretraining. This study performed a controlled audit of several benchmarks, including SLAKE-En and OmniMedVQA, using four detection methods: image-side near-neighbour overlap, canonical-order exchangeability, cohort-relative Min-K%++ tail enrichment, and cross-model top-K overlap.
Findings revealed that 19.8% of images in SLAKE-En showed overlap, suggesting source or distributional contamination rather than direct memorization. Additionally, the Qwen2.5-VL model exhibited a persistent exchangeability signal, while other models like BLIP-2 showed misleading positive signals despite lacking medical exposure.
The study concludes that certain detection methods are unreliable for inferring membership in small medical VLM cohorts. These insights highlight the need for careful evaluation of VLMs to ensure their accuracy and reliability.