NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
TL;DR
Sparse Mixture-of-Experts (MoE) models face deployment challenges due to high memory requirements for expert routing. SHAPE is a new pruning framework that optimizes expert selection by modeling expert cooperation using a Shapley-style attribution method.
✦ Why It Matters
Engineers can leverage SHAPE to optimize expert selection in MoE models, improving efficiency and reducing memory usage.
Key Takeaways
- Implement SHAPE for efficient expert pruning in MoE models.
- Use Shapley-style attribution to evaluate expert contributions.
- Aim for 20-40% expert pruning without sacrificing accuracy.
- Reduce peak GPU memory usage in large language models.
Full Summary
Sparse Mixture-of-Experts (MoE) large language models are effective but struggle with memory limitations because all experts must be available for routing decisions. SHAPE, a novel pruning framework, addresses this by considering the cooperative nature of expert outputs, using a Shapley-style attribution to evaluate expert contributions based on their interactions in top-k combinations.
This method identifies essential experts for high-utility collaborations rather than just frequently used ones. SHAPE also implements a quality-coverage selection rule to ensure that a minimal subset of experts is retained while meeting a global pruning budget.
Testing on models like Qwen3-30B-A3B and GPT-OSS-20B demonstrated that SHAPE consistently outperforms traditional pruning methods, achieving competitive accuracy with 20% to 40% expert pruning and significantly lowering peak GPU memory usage. These findings suggest that SHAPE can enhance the efficiency of deploying MoE models in resource-constrained environments.