EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·11d ago·5 min read
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
TL;DR
Olmo-core 3 introduces a redesigned open mixture-of-experts (MoE) training system, enabling scalable training of large language models with up to one trillion parameters while maintaining efficiency.
✦ Why It Matters
Explore Olmo-core 3 to enhance your large language model training capabilities today.
Key Takeaways
- Utilize Olmo-core 3 for efficient large model training.
- Experiment with expert and pipeline parallelism techniques.
- Leverage MXFP8 for improved training throughput.
- Explore the open framework for custom MoE architectures.
Full Summary
Olmo-core 3 is a significant upgrade to the framework for developing large language models, specifically designed to enhance the training of mixture-of-experts (MoE) models. By increasing the expert pool from 8 to 128 while keeping active parameters per token stable, the total parameter capacity surged from 4.6 billion to 47 billion with minimal impact on training throughput.
The new system employs distributed data parallelism (DDP) to optimize GPU memory usage and routing efficiency, achieving a throughput of 52,000 tokens per second per GPU on a 47-billion-parameter model. Additionally, it incorporates techniques like expert parallelism and pipeline parallelism to further enhance scalability.
The implementation of MXFP8, a lower-precision number format, improved training throughput by 21% compared to the previous BF16 format. These advancements not only facilitate the development of larger models but also provide researchers with an open framework to experiment with their own MoE architectures.