EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Lobsters·12d ago·5 min read
We used a database as a message queue. Now we use Kafka
TL;DR
Using FoundationDB as a message queue has worked well for scaling async tasks, but it has led to performance issues and complexity. Transitioning to Kafka could alleviate these challenges.
✦ Why It Matters
Consider evaluating your current message queue solution for potential performance bottlenecks and complexity.
Key Takeaways
- Assess the scalability of your current database as a message queue.
- Explore Kafka for improved task management and performance.
- Simplify onboarding for new team members with standard implementations.
- Monitor read load impacts when using databases for queuing.
Full Summary
FoundationDB, a key-value store, was initially used to manage various data types and as a message queue for asynchronous tasks, similar to Apple's QuiCK system. This approach allowed for seamless transaction management, avoiding the dual-write problem.
However, the team encountered significant challenges, including increased read load from scheduling tasks, expensive multi-write operations for task management, and a steep learning curve for new team members due to custom implementations. As the number of tasks grew with new features, these issues became more pronounced.
The decision to switch to Kafka, a distributed event streaming platform, aims to streamline task handling and reduce the burden on FoundationDB, ultimately improving system performance and maintainability.