We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·23d ago·5 min read
Flash-MSA: Accelerating Million-Token Training with Sparse Attention Kernels
TL;DR
Sparse attention mechanisms have accelerated inference in large models, but efficient training has lagged. Flash-MSA introduces performant open-source training kernels for Minimax Sparse Attention, utilizing blockwise sparsity for improved caching.
✦ Why It Matters
Engineers can implement Flash-MSA to train large models with sparse attention more efficiently today.
Key Takeaways
- Explore Flash-MSA for efficient training of large AI models.
- Utilize blockwise sparsity to enhance caching in attention mechanisms.
- Consider SpheronH100 and B200 for implementing new training kernels.
- Investigate the implications of sparse attention on model accessibility.
Full Summary
Sparse attention has been a game-changer for speeding up inference in large models, yet efficient training implementations have been lacking. Flash-MSA presents the first open-source training kernels for Minimax Sparse Attention (MSA), designed for Hopper and Blackwell GPUs.
By employing blockwise sparsity, where attention is computed in blocks of 128 using max-pooling over proxy scores, the new kernels enhance caching efficiency. Developed on SpheronH100 and B200 rentals, this implementation addresses a gap in training methodologies for sparse attention, which has not been widely adopted in Western labs.
The introduction of these kernels allows for training models like GLM-5.2 and DSv4, which were previously limited by their reliance on more complex sparse attention formulations. This innovation not only streamlines the training process but also opens up new avenues for research and development in large-scale AI models.