EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Lobsters·Aug 20·5 min read
20× the CI traffic without getting slower: How we rebuilt Git serving at Datadog
TL;DR
Datadog developed gitretriever, a Git mirror that efficiently serves code to their CI system, achieving 20× traffic growth while maintaining low latency. This innovation significantly reduced CPU usage on their previous Git backend.
✦ Why It Matters
Consider implementing a Git mirror to optimize CI performance in your own development environment.
Key Takeaways
- Implement a Git mirror to reduce fetch times in CI.
- Use independent pods to avoid CPU bottlenecks during high traffic.
- Consider a read-only API for specific Git queries to minimize load.
- Monitor and optimize your CI infrastructure regularly to prevent outages.
How It Works
Gitretriever operates with independent pods that maintain local copies of repositories, allowing for efficient read operations without the need for consensus or multi-writer replication. Each pod polls for changes and serves requests directly, minimizing CPU-intensive operations.
This design reduces the load on the main GitHub repository and allows for quick responses to CI jobs.