NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
Security
Cloudflare Blog·May 14·5 min read
Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse
TL;DR
Cloudflare's billing pipeline, which processes millions of daily queries on ClickHouse (an analytics database), suddenly slowed after a migration despite normal I/O and memory metrics. Engineers discovered a hidden bottleneck buried in ClickHouse's internal query processing logic rather than obvious resource constraints.
✦ Why It Matters
Engineers should investigate internal subsystem bottlenecks when standard metrics appear normal during performance degradation.
Key Takeaways
- Standard performance metrics (I/O, memory, rows scanned) can mask internal bottlenecks in complex systems.
- Investigate query engine internals when obvious resource constraints appear normal during slowdowns.
- Post-migration performance regressions warrant deep-dive analysis beyond surface-level diagnostics.
- Patches to database internals can resolve bottlenecks that standard tuning cannot address.
Full Summary
Cloudflare relies heavily on ClickHouse, an open-source OLAP (online analytical processing) database designed for fast analytical queries, to process petabyte-scale billing data across dozens of clusters. Their daily aggregation jobs determine customer charges and power fraud detection systems worth hundreds of millions in revenue.
After a migration, these jobs inexplicably slowed despite standard performance diagnostics showing normal I/O, memory usage, rows scanned, and data partition reads. Rather than resource exhaustion, the team identified a hidden bottleneck in ClickHouse's internal query execution engine.
They developed and deployed three targeted patches addressing the root cause within ClickHouse's core processing logic. This discovery demonstrates that performance problems in complex distributed systems can hide behind normal-looking metrics, requiring deep investigation of internal subsystems.