Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 28·5 min read
Subspace Alignment for Vision-Language Model Test-time Adaptation
TL;DR
SubTTA is a novel approach for adapting vision-language models (VLMs) at test time, addressing issues of modality gaps and visual noise. It improves zero-shot predictions, achieving an average performance boost of 2.24% over existing methods.
✦ Why It Matters
Implement SubTTA in your VLM projects to enhance performance under distribution shifts.
Key Takeaways
- Utilize SubTTA for improved adaptation of VLMs to new data.
- Align semantic subspaces to reduce modality gaps.
- Filter visual noise to enhance task-specific performance.
- Achieve an average performance boost of 2.24% over existing methods.
How It Works
SubTTA aligns the principal subspaces of visual and textual modalities by minimizing their chordal distance, effectively bridging the modality gap. It then projects the aligned visual features onto a task-specific textual subspace, filtering out irrelevant noise and refining decision boundaries through standard TTA.