EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·7d ago·5 min read
Dust: Pretraining Transformers Without Backpropagation
TL;DR
Dust introduces a novel zeroth-order method for pretraining transformer language models that competes with backpropagation, achieving significant efficiency gains. It demonstrates that larger models can be more population-efficient, potentially surpassing traditional methods in compute-rich environments.
✦ Why It Matters
Consider experimenting with Dust for pretraining transformer models to enhance efficiency in your projects.
Key Takeaways
- Explore Dust for pretraining transformer models to improve efficiency.
- Utilize activation perturbations instead of weight perturbations for gradient estimation.
- Investigate the benefits of larger models for population efficiency.
- Consider the implications of compute-rich environments on model training strategies.
How It Works
Dust operates by adding Gaussian noise to the activations of each token independently during a forward pass. This approach allows the model to evaluate a large virtual population of perturbations without needing to materialize them in the weights.
The method calculates the loss reduction for each token, using these rewards to estimate gradients. This innovative mechanism enables Dust to efficiently scale with population size, outperforming traditional methods.