Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·23h ago
AI Research
arXiv CS.AI·May 28·8 min read
On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
TL;DR
Researchers identified a critical flaw in how spoken language models are evaluated using global token perplexity—a single aggregate metric measuring prediction uncertainty across all tokens. They demonstrated that this metric masks important performance differences across phonetic contexts and token types.
✦ Why It Matters
Engineers should evaluate spoken language models using stratified metrics by phonetic context and token frequency, not global perplexity alone.
Key Takeaways
- Transition from global token perplexity to new evaluation methods.
- Utilize likelihood-based metrics for better performance assessment.
- Focus on human-rated scores to gauge model quality.
- Recognize the reduced gap between models and human performance.
How It Works
The study critiques the use of global token perplexity for evaluating spoken language models, highlighting its inadequacy in capturing speech-specific features. It introduces new evaluation methods that leverage likelihood and generative principles, which align more closely with human perceptions of speech quality.
By correlating these new metrics with human ratings, the authors demonstrate that they provide a more accurate reflection of model performance.