NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
TL;DR
Generative perplexity (gen-PPL), a common metric for evaluating language models, fails to assess text quality beyond predictability. The authors developed naive samplers that achieved state-of-the-art gen-PPL scores while producing incoherent text.
✦ Why It Matters
Engineers and researchers should adopt distributional metrics for more accurate evaluations of language model outputs.
Key Takeaways
- Rethink reliance on generative perplexity for model evaluation.
- Explore distributional metrics for better text quality assessment.
- Test your models against new evaluation suites for accuracy.
- Be cautious of models achieving high gen-PPL but producing low-quality text.
How It Works
The authors demonstrate that generative perplexity only measures predictability based on an autoregressive model, neglecting the actual quality of generated text. They constructed naive samplers that, despite being simple, achieved high gen-PPL scores, highlighting the metric's limitations.