NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 25·5 min read
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
TL;DR
Muon, a method for training large language models (LLMs), struggles with heavy-tailed weight distributions that hinder performance. HTMuon was developed to enhance Muon by allowing for heavier-tailed updates while maintaining parameter interdependencies.
✦ Why It Matters
Engineers can implement HTMuon to improve LLM training efficiency and model performance significantly.
Key Takeaways
- Consider using HTMuon for LLM training to enhance performance.
- Implement heavy-tailed updates to improve model robustness.
- Evaluate the impact of HTMuon on your existing Muon variants.
- Analyze convergence properties under Schatten-q norm constraints.
Full Summary
Muon has recently emerged as a promising technique for training large language models (LLMs), but it faces challenges due to its orthogonalized update rule, which can lead to heavy-tailed weight spectra and excessive focus on noise. To address this, HTMuon was introduced, leveraging the Heavy-Tailed Self-Regularization (HT-SR) theory.
HTMuon modifies the update mechanism to produce heavier-tailed updates while preserving the ability to capture interdependencies among parameters. Experimental results demonstrated that HTMuon consistently outperformed state-of-the-art baselines in both LLM pretraining and image classification tasks.
For instance, during LLaMA pretraining on the C4 dataset, HTMuon achieved a perplexity reduction of up to 0.98 compared to the original Muon. Theoretical analysis indicates that HTMuon aligns with steepest descent under the Schatten-q norm constraint, providing insights into its convergence in smooth non-convex settings.
This advancement offers a practical enhancement for researchers and engineers working with LLMs.