EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Lobsters·Jul 29·5 min read
Lorenz and Little: How Much Does Your Tail Cost?
TL;DR
Marc discusses the significance of tail latency in cost optimization, emphasizing its impact on mean latency. He introduces the empirical Lorenz Curve to quantify contributions of latency percentiles to overall performance.
✦ Why It Matters
Evaluate your latency percentiles today to better understand their impact on overall system costs.
Key Takeaways
- Focus on tail latency to enhance customer experience.
- Use the empirical Lorenz Curve to quantify latency contributions.
- Implement the OneMinusL function for latency analysis.
- Regularly assess latency percentiles for cost optimization.
How It Works
The empirical Lorenz Curve calculates the contribution of latency percentiles to mean latency. By sorting latency samples and applying a power law for interpolation, it quantifies how much requests below a certain percentile impact overall performance.