Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
Software Engineering
Cloudflare Blog·May 28·5 min read
How we built Cloudflare's data platform and an AI agent on top of it
TL;DR
Cloudflare processes over a billion events per second across distributed systems, but analysts struggled to query data scattered across dozens of databases, data warehouses, and message queues with different credentials and query languages. The company built Town Lake, a unified SQL interface consolidating all data sources, and Skipper, an AI agent enabling plain-English queries.
✦ Why It Matters
Engineers can adopt unified data platforms and AI agents to democratize analytics access and accelerate decision-making in data-intensive organizations.
Key Takeaways
- Consolidate fragmented data sources behind a single SQL interface to reduce query complexity.
- Layer AI agents on top of unified platforms to enable natural language data access for non-technical users.
- Design systems to track data freshness and sampling status to ensure query result reliability.
- Implement audit trails in AI-driven analytics to maintain accountability and traceability of insights.
Full Summary
Cloudflare's rapid growth created data sprawl: analytics data lived in ClickHouse clusters (columnar databases optimized for analytics), Kafka streams (real-time message queues), BigQuery datasets (cloud data warehouse), production Postgres databases, and Google Cloud buckets, each with separate credentials and query languages. Answering basic questions required knowing which system held relevant data and whether it was sampled or stale.
To solve this, Cloudflare built Town Lake, a unified SQL query layer abstracting all underlying data sources into a single interface. On top of Town Lake, they deployed Skipper, an AI data agent that translates natural language questions into SQL queries and returns auditable answers within seconds.
This approach eliminates context-switching between systems and reduces barriers to data access across the organization.