Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Jul 30·5 min read
VideoNorms: Benchmarking Cultural Awareness of Video Language Models
TL;DR
VideoNorms introduces a dataset to evaluate the cultural awareness of Video Large Language Models (VideoLLMs) using annotations from US and Chinese TV shows. Key findings reveal significant performance disparities between cultures, highlighting the need for culturally grounded training.
✦ Why It Matters
Incorporate cultural context into your VideoLLM training to enhance model performance across diverse audiences.
Key Takeaways
- Utilize the VideoNorms dataset for culturally aware model training.
- Focus on improving norm adherence predictions for underrepresented cultures.
- Incorporate both verbal and non-verbal evidence in model evaluations.
- Consider the limitations of model size in enhancing classification performance.
Full Summary
As Video Large Language Models (VideoLLMs) gain global traction, assessing their cultural reasoning capabilities becomes crucial. VideoNorms is a newly developed dataset that includes cultural norm annotations from popular US and Chinese TV shows, marked with adherence or violation labels and supporting evidence.
The dataset was created through a collaboration where a VideoLLM initially annotated items, followed by reviews from trained annotators familiar with the respective cultures, resulting in over 3,000 human judgments. Analysis of seven open-weight VideoLLMs showed that these models performed worse in predicting norm adherence for Chinese content compared to US content.
Additionally, they struggled more with providing non-verbal evidence than verbal evidence for norm predictions. Ablation studies confirmed that video modality is essential for accurate performance, while simply increasing model size did not improve classification scores.
These findings emphasize the importance of culturally aware training and evaluation for video models.