Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·May 26·5 min read
When Mean CE Fails: Median CE Can Better Track Language Model Quality
TL;DR
Mean cross-entropy (CE) is commonly used to evaluate language model quality but can misrepresent performance during training. The authors propose using median CE as a more reliable metric, demonstrating its effectiveness in two scenarios: synthetic fact-learning and top-K distillation.
✦ Why It Matters
Engineers should consider median CE alongside mean CE for more accurate model evaluation during training.
Key Takeaways
- Use median CE for a more accurate assessment of model quality.
- Monitor both mean and median CE during training.
- Report percentile CE summaries to track distribution changes.
- Adjust K in top-K distillation to optimize median CE.
How It Works
The study analyzes how the distribution of per-token CE changes during training. In top-K distillation, a smaller K leads to a CE distribution with more mass at both extremes, which lowers the median and raises the mean.
In contrast, during Qwen SFT, the bulk of the CE saturates quickly while the tail extends, indicating that the model's performance is more sensitive to the bulk than the tail.