Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·May 25·8 min read
Multimodal Distribution Matching for Vision-Language Dataset Distillation
TL;DR
Training vision-language models (systems that process both images and text) requires massive datasets, but compressing them into smaller synthetic versions while maintaining performance is difficult because prior methods ignore how images and text relate. Multimodal Distribution Matching (MDM) addresses this by matching statistical distributions across both modalities simultaneously using geometry-aware techniques at data, model, and loss levels.
✦ Why It Matters
Engineers can now compress vision-language training data efficiently while maintaining cross-modal alignment and reducing training costs across different model architectures.
Key Takeaways
- Utilize MDM for efficient vision-language dataset distillation.
- Leverage joint embedding space for synthetic data initialization.
- Adopt mixed teacher models for improved training outcomes.
- Apply geometry-aware loss functions to enhance cross-modal alignment.
How It Works
MDM operates by sampling synthetic image-text pairs from clusters in a joint embedding space, ensuring that the generated data reflects the original dataset's multimodal characteristics. It employs a mixed teacher model, which combines weights from multiple fine-tuned models based on their angular relationships to a pretrained anchor.
The loss function is designed to match joint distributions on a unit hypersphere, focusing on both agreement and discrepancies between modalities, thus enhancing the overall quality of the distilled dataset.