NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
Data Science
Sebastian Raschka Blog·20d ago·5 min read
Inkling: A New Open-Weight 975B MoE with a Few Surprises
TL;DR
Thinking Machines Lab has introduced Inkling, a 975 billion parameter open-weight model utilizing a sparse mixture of experts (MoE) design. It features short convolutions and embedding RMSNorm, enhancing its performance on benchmarks.
✦ Why It Matters
Engineers can implement Inkling's sparse MoE design to optimize resource usage in large-scale AI applications.
Key Takeaways
- Evaluate Inkling's performance on general tasks for your applications.
- Consider fine-tuning Inkling for specialized use cases.
- Investigate the impact of small convolution layers on model performance.
- Explore the learned relative-position bias for handling longer sequences.
How It Works
Inkling employs a sparse Mixture-of-Experts architecture, activating only a fraction of its parameters per token. It integrates small convolution layers after key and value projections to enhance local token mixing, while a learned relative-position bias improves performance on longer sequences.
This design allows for efficient handling of context within its extensive token window.