Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization

TL;DR

Muon, a training algorithm for neural networks, achieves guaranteed convergence to a target loss using a finite-step Newton-Schulz orthogonalization method. This approach ensures effective training even with momentum accumulation and without exact orthogonalization.

Key Takeaways

• Implement Muon for guaranteed convergence in neural network training.
• Use a learning rate proportional to (1-μ)√ε for effective training.
• Ensure network width is sufficient but independent of target accuracy.
• Leverage numerical experiments to validate theoretical findings in practice.