Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·23h ago
AI Research
arXiv CS.AI·May 25·6 min read
TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning
TL;DR
Multimodal large language models (MLLMs) fine-tuned via cloud services face backdoor attacks where poisoned training data embeds hidden triggers causing malicious behavior. TCAP (Tri-Component Attention Profiling) detects poisoned samples by analyzing how attention mechanisms allocate focus across system instructions, vision inputs, and user queries—revealing that backdoored data creates abnormal attention patterns.
✦ Why It Matters
Engineers deploying fine-tuned MLLMs can now detect poisoned training data automatically, reducing security risks in production systems without manual annotation.
Key Takeaways
- Adopt TCAP for unsupervised backdoor detection in MLLMs.
- Analyze attention distribution across system instructions, vision inputs, and user queries.
- Utilize Gaussian Mixture Model for profiling attention heads.
- Conduct extensive testing across various MLLM architectures.
How It Works
TCAP decomposes attention maps into three components, allowing for the identification of backdoor triggers by analyzing how attention is distributed across these components. By applying Gaussian Mixture Model (GMM) profiling, TCAP can detect which attention heads are influenced by poisoned samples, leading to effective isolation of these samples through an EM-based voting mechanism.