EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·21d ago·5 min read
Mini-AGI – dynamic continual learning model trained from scratch on 8GB VRAM
TL;DR
Mini-AGI is a continual learning language model that can be trained from scratch on an 8 GB VRAM GPU, dynamically adjusting its architecture and learning from a single data stream. This experiment demonstrates the feasibility of continual learning without catastrophic forgetting on modest hardware.
✦ Why It Matters
Consider experimenting with Mini-AGI to explore continual learning on your own hardware setup.
Key Takeaways
- Train a language model from scratch on consumer hardware.
- Utilize continual learning to prevent knowledge loss.
- Store model weights on disk to optimize VRAM usage.
- Customize the model to evolve with your data.
How It Works
Mini-AGI employs a unique architecture that allows it to read and learn from data continuously. It processes input one character at a time, adjusting its internal structure dynamically based on the data it encounters.
The model uses a pool of experts, where each expert can be activated based on the current input, allowing for efficient learning and retention of knowledge. This design enables the model to adapt its capacity as needed, ensuring it can handle new information without forgetting previous learning.