We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 5·5 min read
Train and run transformers directly on Apple's Neural Engine
TL;DR
Engineers faced challenges with slow inference times for transformers on Apple devices. Espresso was developed to compile Machine Intermediate Language (MIL) programs directly to Apple’s Neural Engine (ANE) without using CoreML.
✦ Why It Matters
Engineers can significantly enhance transformer model performance on Apple devices using Espresso for direct ANE inference.
Key Takeaways
- Use Espresso for faster transformer inference on Apple devices.
- Leverage direct ANE compilation to improve performance.
- Avoid CoreML for reduced latency in AI applications.
- Integrate with minimal code for quick deployment.
Full Summary
Transformers, a type of neural network architecture, often struggle with inference speed on Apple devices due to reliance on CoreML, which can introduce latency. To address this, Espresso was created to compile Machine Intermediate Language (MIL) programs directly to the Apple Neural Engine (ANE) using reverse-engineered private APIs.
This method eliminates the need for per-token recompilation and utilizes IOSurface buffers and fused multi-layer kernels, allowing for two verified tokens to be processed per decode step. As a result, engineers can achieve inference speeds that are 4.76 times faster than those using CoreML.
This significant performance boost opens up new possibilities for deploying transformer models on Apple hardware. The streamlined process also simplifies integration into existing projects, making it easier for developers to leverage advanced AI capabilities.