EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 5·5 min read
An SLM trained on $8 ESP32-S3
TL;DR
A new model was successfully trained from scratch on an $8 ESP32-S3 microcontroller, demonstrating that powerful AI training can occur on low-cost hardware. This breakthrough challenges the notion that only high-end GPUs or data centers are necessary for model training.
✦ Why It Matters
Consider experimenting with AI model training on low-cost microcontrollers to explore new possibilities.
Key Takeaways
- Experiment with training small models on microcontrollers.
- Utilize compact and clean datasets for effective learning.
- Consider edge AI applications in remote environments.
- Explore the potential of low-cost hardware for innovative solutions.
How It Works
The ESP32-S3 microcontroller trains a transformer model by initializing random weights and processing a structured corpus of Klingon text. It performs forward passes to compute losses and updates weights through backpropagation, all within the device's limited memory.