Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·Aug 19·5 min read
Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
TL;DR
ADAPT (Adversarial Disentangled Prompt Tuning) is a new framework that enhances the robustness of vision-language models by preventing overfitting on pseudo-robust features. It effectively improves performance on unseen classes by using a dual-prompt mechanism to disentangle robust features from non-generalizable shortcuts.
✦ Why It Matters
Implement ADAPT in your vision-language models to improve robustness against unseen adversarial examples.
Key Takeaways
- Use ADAPT to enhance model robustness against adversarial examples.
- Implement a dual-prompt mechanism to separate robust and pseudo-robust features.
- Monitor performance on unseen classes to evaluate model generalization.
- Apply orthogonal constraints in prompt tuning to prevent overfitting.
How It Works
ADAPT utilizes a dual-prompt mechanism where the target prompt learns robust features while decoy prompts capture pseudo-robust features. The orthogonal constraint ensures that the target prompt remains distinct from the decoys, effectively preventing the model from overfitting on non-generalizable shortcuts.