Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 27·5 min read
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
TL;DR
Large language models (LLMs) run poorly on devices with limited memory and compute. Dense2MoE converts dense neural networks into mixture-of-experts (MoE) architectures—where different input types activate different specialized sub-networks—via pruning and retraining.
✦ Why It Matters
Engineers can deploy larger, more capable LLMs on mobile and edge devices by adopting MoE conversion techniques that reduce runtime memory and compute requirements.
Key Takeaways
- Convert dense LLMs to MoE architectures to reduce active parameters during inference on edge devices.
- Apply unified pruning and upcycling to maintain model quality while shrinking computational footprint.
- Measure deployment success via latency, memory usage, and accuracy tradeoffs specific to your hardware.
- Test MoE routing efficiency; poorly calibrated routers can negate sparsity gains from selective activation.
Full Summary
Running large language models on edge devices (phones, embedded systems) requires balancing model size, latency, and accuracy—a constraint known as the Pareto frontier. Dense2MoE addresses this by transforming standard dense LLMs into mixture-of-experts (MoE) models, where a router network selectively activates only relevant expert sub-networks per input token, reducing computation.
The approach combines unified pruning (removing less important weights) and upcycling (repurposing pruned weights into expert modules). This two-stage process maintains model quality while dramatically reducing active parameters during inference.
Results demonstrate improved efficiency metrics—faster inference speed and lower memory footprint—compared to standard pruning or quantization alone, enabling practical deployment on resource-constrained devices without sacrificing output quality.