Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·24d ago·5 min read
Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Study
TL;DR
A novel framework, MuseDiag, was developed to analyze music hallucination in audio-language models, revealing universal weaknesses and model-specific vulnerabilities. Key findings include the identification of vocal misperception as a common issue and the effectiveness of two mitigation methods, though results vary by model.
✦ Why It Matters
Evaluate your audio-language models using MuseDiag to identify and mitigate hallucination issues effectively.
Key Takeaways
- Use MuseDiag to assess model performance on music descriptions.
- Focus on tonal perception to differentiate model architectures.
- Implement ADD-M and TPA to reduce hallucination in specific contexts.
- Monitor vocal misperception as a common weakness across models.
Full Summary
Audio-language models often generate inaccurate music descriptions, a phenomenon termed 'music hallucination.' This study introduces MuseDiag, a diagnostic framework that evaluates nine audio-language models across five perceptual layers: sound events, temporal properties, tonal attributes, style, and emotion.
The research found that vocal misperception is a consistent weakness across all models, while tonal perception differentiates model architectures. Audio-Flamingo-3 emerged as the most reliable model, with varying performance among others.
The study also identified factors like affirmative bias and generation-mode effects contributing to hallucination. Two training-free methods, Audio-Dependency-Aware Decoding for Music (ADD-M) and Taxonomy-Guided Perceptual Anchoring (TPA), were tested, showing potential to reduce hallucination but with inconsistent results across different models and tasks.