Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 12·5 min read
Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech
TL;DR
A switch aware evaluation of ASR and audio language models on English Yoruba code-switched speech reveals that traditional metrics like word error rate (WER) can obscure performance differences. The study highlights significant shortcomings in recognizing Yoruba tokens, especially at switch points, and introduces new metrics for better assessment.
✦ Why It Matters
Evaluate your ASR and audio LM systems using switch-aware metrics to improve performance on code-switched speech.
Key Takeaways
- Adopt switch-aware metrics for evaluating ASR and audio LMs.
- Recognize the limitations of traditional WER in code-switched contexts.
- Focus on improving Yoruba token recognition in ASR systems.
- Utilize provided tools for reproducible benchmarking in your projects.
How It Works
The study introduces new metrics such as switch entry token error rate (SETER) and windowed switch point error rates to better evaluate code-switching performance. These metrics allow for a more nuanced understanding of how well systems handle transitions between languages.