Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 21·6 min read
GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
TL;DR
Mixture-of-Expert (MoE) models speed up inference by activating only a subset of specialized neural network components per input token, but distributing these experts across GPUs creates synchronization bottlenecks—faster GPUs idle waiting for slower ones. GEM is a mapping algorithm that assigns experts to GPUs based on measured hardware performance variation, reducing lock-step synchronization delays.
✦ Why It Matters
Engineers deploying MoE models can reduce serving latency by accounting for GPU performance variance during expert assignment.
Key Takeaways
- Account for GPU performance differences when assigning MoE experts to reduce synchronization bottlenecks.
- Lock-step batching creates idle time; balance workload completion times across heterogeneous hardware.
- GEM algorithm optimizes expert placement for mixed-generation or variable-performance GPU clusters.
- Measure and model actual GPU performance variance before deploying MoE serving systems.
Full Summary
Mixture-of-Expert systems improve inference efficiency by activating only a subset of specialized expert modules per input token rather than running all parameters. During serving, tokens are batched and routed to appropriate GPUs hosting activated experts.
However, lock-step synchronization requires all tokens in a batch to complete processing before advancing to the next layer, creating idle time when some GPUs finish faster than others due to hardware variability. GEM addresses this by developing a GPU-variability-aware expert-to-GPU mapping algorithm that optimizes expert placement considering actual GPU performance differences.
The approach models GPU heterogeneity and uses it to assign experts such that workload completion times align more closely across devices. Results demonstrate reduced inference latency compared to naive expert distribution strategies, with implications for cost-effective MoE deployment on mixed-generation or heterogeneous GPU clusters.