NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·Jun 11·5 min read
Redesign Mixture-of-Experts Routers with Manifold Power Iteration
TL;DR
Existing Mixture-of-Experts (MoE) routers face inefficiencies in selecting expert models for tasks. The authors introduced a new method called Manifold Power Iteration to optimize these routers.
✦ Why It Matters
Engineers can enhance the efficiency and accuracy of their Mixture-of-Experts models using Manifold Power Iteration.
Key Takeaways
- Implement Manifold Power Iteration for router design in MoE models.
- Align router rows with principal singular directions of experts.
- Utilize the 'Power-then-Retract' method for stability.
- Pretrain models across various scales to validate improvements.
How It Works
The proposed method uses Manifold Power Iteration (MPI) to align router rows with the principal singular direction of the expert matrix. This alignment captures the most expressive features of the expert, enhancing the router's ability to determine token-expert affinity.
The 'Power-then-Retract' approach first applies power iteration to optimize router weights, followed by a retraction step to enforce a norm constraint, ensuring both efficiency and stability in the learning process.