Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models
TL;DR
Existing methods for explaining Large Language Models (LLMs) primarily focus on text inputs, leaving a gap for multimodal models that process both text and audio. mllm-shap is an open-source Python framework that extends Shapley Value (SV) explainability to these Multimodal LLMs. By addressing challenges like modality-aware coalition masking, it enhances understanding of model decisions in joint text-audio contexts.
✦ Why It Matters
Engineers can leverage mllm-shap to enhance the interpretability of multimodal AI models in their applications.
Key Takeaways
- Utilize mllm-shap for explainability in text-audio models.
- Implement phonetic alignment for efficient token grouping.
- Leverage the GUI for visualizing model attributions.
- Explore five SV estimation strategies for better performance.
How It Works
mllm-shap employs modality-aware coalition masking to handle the simultaneous processing of text and audio inputs. It tracks conversation context through per-token metadata, ensuring that the model maintains awareness of roles and modalities.
The innovative phonetic alignment-based token grouping reduces the computational burden of SV estimation, making it practical for longer audio segments.