Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Post-Trained MoE Can Skip Half Experts via Self-Distillation
TL;DR
Existing dynamic Mixture-of-Experts (MoE) models require extensive pre-training, limiting their efficiency. The Zero-Expert Self-Distillation Adaptation (ZEDA) framework allows post-trained MoE models to dynamically skip unnecessary experts during inference.
✦ Why It Matters
Engineers can leverage ZEDA to enhance the efficiency of existing MoE models without extensive retraining.
Key Takeaways
- Implement ZEDA to optimize existing MoE models.
- Utilize self-distillation for efficient model adaptation.
- Achieve significant inference speedup with minimal accuracy loss.
- Explore zero-output experts for dynamic model efficiency.
Full Summary
Dynamic Mixture-of-Experts (MoE) models enhance language model efficiency by activating only a subset of experts based on input, but they typically need to be trained from scratch or adapted for specific tasks. The Zero-Expert Self-Distillation Adaptation (ZEDA) framework addresses this by transforming fully trained static MoE models into dynamic ones without extensive retraining.
ZEDA introduces parameter-free zero-output experts into each MoE layer and employs a two-stage self-distillation process, where the original MoE serves as a frozen teacher. This method stabilizes the conversion and applies a group-level balancing loss to optimize performance.
Testing on models like Qwen3-30B-A3B and GLM-4.7-Flash across various benchmarks showed that ZEDA can eliminate over 50% of expert floating-point operations (FLOPs) while maintaining accuracy. Additionally, it outperformed existing dynamic MoE methods by 6.1 and 4.0 points on the respective models, resulting in a significant end-to-end inference speedup of about 1.20 times.