Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
TL;DR
Vision-Language Models (VLMs) struggle with reliably ranking multimodal items due to vulnerabilities in their cross-modal knowledge application. The authors developed Multimodal Generative Engine Optimization (MGEO), a technique that manipulates VLM rankings by crafting subtle image changes and coherent text.
✦ Why It Matters
Engineers should consider the vulnerabilities of VLMs when designing retrieval systems and explore defense strategies.
Key Takeaways
- Implement defenses against adversarial attacks on VLMs.
- Assess the robustness of your multimodal retrieval systems.
- Explore the implications of MGEO in your applications.
- Stay updated on advancements in knowledge grounding techniques.
How It Works
MGEO manipulates VLM rankings by creating subtle changes in images and text that align with the model's internal knowledge. This is achieved through an alternating optimization strategy that targets the interactions between visual and linguistic representations, allowing for effective rank manipulation.