BerriAI / litellm
github.com·3d ago
Developer Tools
GitHub Trending·Jul 29·5 min read
maderix / ANE
TL;DR
A research project has successfully demonstrated training neural networks directly on Apple's Neural Engine (ANE) using reverse-engineered private APIs, bypassing CoreML's limitations. This breakthrough highlights the ANE's potential for training tasks, previously restricted to inference only.
✦ Why It Matters
Explore the potential of ANE for your own neural network training projects using reverse-engineered APIs.
Key Takeaways
- Experiment with the ANE for training small models.
- Contribute to the project by sharing benchmarks on different hardware.
- Utilize the provided research code for custom AI development.
- Be aware of the current limitations in performance and operation.
How It Works
The project utilizes reverse-engineered private APIs, specifically _ANEClient and _ANECompiler, to enable training on the ANE. It constructs custom compute graphs that include both forward and backward passes for neural networks, allowing for gradient calculations directly on the ANE hardware.
The implementation leverages shared memory (IOSurface) for efficient data transfer between the GPU and ANE, optimizing performance.
⚠ The Catch
Despite successful training, the ANE's utilization is low, around 5-9% of its peak capabilities, and many operations still require CPU fallback, limiting its effectiveness for larger models.