We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·May 16·5 min read
AI and compute
TL;DR
Since 2012, the compute power used for training large AI models has been increasing rapidly, with a doubling time of just 3.4 months. This exponential growth has led to a more than 300,000-fold increase in compute usage, far surpassing the 7-fold increase predicted by Moore's Law over the same period.
✦ Why It Matters
Engineers should anticipate the need for increased computational resources to support future AI developments.
Key Takeaways
- Invest in scalable hardware solutions for AI training.
- Optimize algorithms to make better use of available compute.
- Stay informed about trends in compute usage for AI.
- Prepare for rapid advancements in AI model complexity.
Full Summary
In recent years, the demand for compute power in AI training has surged, highlighting a significant gap in resources needed for advanced models. An analysis reveals that since 2012, the compute used in the largest AI training runs has been doubling every 3.4 months, compared to Moore's Law, which predicted a 2-year doubling period.
This rapid increase has resulted in a staggering 300,000-fold growth in compute usage, emphasizing the critical role of hardware advancements in AI progress. The methodology involved tracking compute usage across major AI training runs, providing a clear picture of the trend.
These findings suggest that as AI models become more complex, the need for enhanced computational resources will only intensify. Engineers and researchers must adapt to this trend by investing in more powerful hardware and optimizing algorithms for efficiency.