NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
MirrorCheck: Efficient Adversarial Defense for Vision-Language Models
TL;DR
Vision-Language Models (VLMs)—AI systems combining image and text understanding—face adversarial attacks designed to fool them. MirrorCheck detects these attacks by regenerating images from model captions using Text-to-Image generators and comparing visual consistency.
✦ Why It Matters
Engineers can deploy MirrorCheck as a model-agnostic detection layer to protect VLM applications against sophisticated adversarial attacks without retraining.
Key Takeaways
- MirrorCheck detects attacks by comparing semantic consistency between original and regenerated images using embeddings.
- Randomized selection from multiple T2I generators and encoders prevents attackers from targeting a single defense mechanism.
- One-Time-Use perturbations applied to encoder embeddings reduce adaptive attack effectiveness without sacrificing model utility.
- Framework operates model-agnostically across both unimodal and multimodal settings, enabling broad deployment.
Full Summary
Vision-Language Models increasingly power real-world applications but remain vulnerable to adversarial attacks—maliciously crafted inputs designed to cause incorrect outputs. MirrorCheck addresses this by implementing a detection framework that works across single-modality and multi-modality scenarios.
The approach leverages Text-to-Image (T2I) models to reconstruct visual content from captions the target VLM produces, then compares embeddings (numerical representations capturing semantic meaning) between original and regenerated images to detect inconsistencies indicating attacks. To defend against adaptive attacks where adversaries specifically target the defense mechanism, MirrorCheck employs stochastic defense—randomly selecting from multiple T2I generators and image encoders—plus One-Time-Use perturbations that degrade attack effectiveness.
Experiments across multiple threat scenarios demonstrated consistent improvements over baseline methods while maintaining utility under strong adaptive adversarial conditions.