NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
When Mean CE Fails: Median CE Can Better Track Language Model Quality
TL;DR
Mean cross-entropy (CE) is commonly used to evaluate language model quality but can misrepresent performance during training. The authors propose using median CE as a more reliable metric, demonstrating its effectiveness in two scenarios: synthetic fact-learning and top-K distillation.
✦ Why It Matters
Engineers should consider median CE alongside mean CE for more accurate model evaluation during training.
Key Takeaways
- Use median CE for better tracking of model quality.
- Report both mean and median CE during evaluations.
- Monitor distribution changes in CE metrics.
- Consider task performance correlation when selecting models.
Full Summary
Mean cross-entropy (CE) is a standard metric for assessing language model quality, but it can fail to accurately reflect model performance during training. This study introduces median CE as a more effective alternative, particularly in two scenarios: the Qwen2.5-1.5B model's synthetic fact-learning and top-K distillation on TinyStories.
In the first case, mean CE increased while fact-recall accuracy plateaued, indicating a disconnect. In the second case, reducing the top-K parameter improved median CE but worsened mean CE, with the best-performing model achieving a higher score despite lower mean CE.
The analysis reveals that training reshapes the CE distribution, affecting how these metrics correlate with actual task performance. The authors recommend reporting both mean and median CE to better track model quality and distribution changes.