Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·1d ago
AI Research
arXiv CS.AI·Jun 10·5 min read
A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks
TL;DR
Public medical vision-language models (VLMs) may be contaminated by pretraining data, affecting their reported accuracy. A controlled audit was conducted using techniques like image-side near-neighbour overlap and canonical-order exchangeability.
✦ Why It Matters
Engineers should critically assess VLMs for pretraining contamination to ensure accurate model evaluations.
Key Takeaways
- Audit public benchmarks for pretraining contamination before evaluation.
- Use multiple detection methods to assess model reliability.
- Be cautious of reported accuracy in medical VLMs.
- Consider manual checks for overlap verification in datasets.
How It Works
The authors employed four detection methods to identify overlaps between benchmark images and pretraining datasets. These included image-side near-neighbour overlap and canonical-order exchangeability, which helped quantify the extent of contamination.