NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
TL;DR
Local Mixture-of-Experts (MoE) models struggle to match cloud-level service quality due to limitations in throughput and concurrency. A CPU-GPU hybrid system was developed, incorporating techniques like stream-loading prefill and distributed expert parallelism.
✦ Why It Matters
Engineers can leverage this hybrid design to enhance local AI model performance without extensive infrastructure.
Key Takeaways
- Implement stream-loading techniques to boost model throughput.
- Utilize distributed expert parallelism for improved performance.
- Adopt intra-node disaggregation to enhance concurrency.
- Focus on original-precision inference for cost-effective AI deployment.
Full Summary
Local deployment of large Mixture-of-Experts (MoE) models often fails to provide the same service quality as cloud environments, particularly under low-concurrency workloads. Key issues include reliance on reduced-capacity models and inadequate throughput and latency.
To address these, a CPU-GPU hybrid system was created, featuring stream-loading prefill (SLP) to boost throughput to 1,200 tokens per second and enabling 32,000 prompts in under 30 seconds. Additionally, distributed SLP (DSLP) with SmallEP expert parallelism achieved 1,800 tokens per second on consumer GPUs.
The system also implemented intra-node prefill-decode disaggregation, which maintained concurrency with minimal latency increase. Evaluations demonstrated that this approach allows for original-precision inference, reshaping local deployment and making high-quality AI access more cost-effective.