Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·23h ago
AI Research
arXiv CS.AI·May 28·5 min read
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
TL;DR
Mobile neural processing units (NPUs) are often assumed to be faster for running large language models (LLMs), but this is not always the case. A stage-level analysis was conducted to evaluate the performance of NPUs compared to traditional CPUs and GPUs during LLM inference.
✦ Why It Matters
Engineers should assess hardware performance based on specific LLM tasks rather than defaulting to NPUs for mobile applications.
Key Takeaways
- CPUs may outperform NPUs in compute-intensive tasks.
- NPUs show limited benefits in memory-bound operations.
- Increased NPU offloading can raise energy consumption.
- Design guidelines for NPUs should focus on practical efficiency.
How It Works
The study employs an OPMASK-based controlled pipeline decomposition methodology to isolate various overheads in the NPU execution path, allowing for a detailed performance analysis across different stages of LLM inference.