EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 20·5 min read
The August 17 outage, and the work ahead
TL;DR
On August 17, GitHub faced a 7-hour outage due to a critical infrastructure component failing to scale with peak traffic, disrupting services globally. This incident highlighted the urgent need for improved capacity management as monthly commits surged from 1.4 billion to 2.9 billion since April.
✦ Why It Matters
Evaluate your own system's capacity management to prevent similar outages in your projects.
Key Takeaways
- Assess your infrastructure's capacity to handle peak traffic.
- Implement consistent retry limits to avoid cascading failures.
- Isolate critical systems to minimize outage impact.
- Invest in monitoring and alerting for better observability.
How It Works
GitHub's recovery involved rerouting traffic and isolating affected infrastructure, which allowed for staged restoration of services. They implemented consistent retry limits and variable timeouts to prevent retry storms during recovery.