Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 20·5 min read
From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model
TL;DR
A new method called VLM TTA unifies inference and adaptation in vision-language models (VLMs) by using optimal transport principles, leading to improved performance. This approach provides robust pseudo-labels and enhances adaptation, achieving up to 7% better accuracy than existing methods.
✦ Why It Matters
Implement VLM TTA to enhance your vision-language model's performance in real-world applications today.
Key Takeaways
- Utilize VLM TTA for improved adaptation in VLMs.
- Leverage Wasserstein OT for robust pseudo-label generation.
- Apply contrastive learning to enhance image-text relationships.
- Achieve up to 7% better accuracy than current methods.
How It Works
VLM TTA formulates the zero-shot image classification task as a cross-modal alignment problem using Wasserstein optimal transport. This allows for the generation of reliable pseudo-labels at the sample level, which are then used in conjunction with a soft-label InfoNCE loss to adapt the VLMs.
The InfoNCE loss is reformulated as a Wasserstein OT formulation, creating a unified objective that enhances both inference and adaptation.