Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 12·5 min read
Output Embedding Centering for Stable LLM Pretraining
TL;DR
Output embedding centering (OEC) is introduced as a new strategy to stabilize large language model (LLM) pretraining by addressing output logit divergence. Both deterministic and regularization implementations of OEC outperform existing methods in training stability.
✦ Why It Matters
Implement output embedding centering to enhance the stability of your LLM training processes today.
Key Takeaways
- Adopt output embedding centering to improve LLM training stability.
- Consider using μ-loss for less sensitivity to hyperparameter tuning.
- Evaluate the effectiveness of μ-centering in your training pipeline.
- Compare OEC methods against existing strategies like z-loss.
How It Works
Output embedding centering (OEC) addresses the geometric distribution of output embeddings. By centering these embeddings, either through a deterministic approach (μ-centering) or a regularization method (μ-loss), the model reduces anisotropic behavior, leading to more stable training outcomes.