tailwindlabs / tailwindcss
github.com·23h ago
Developer Tools
Databricks Blog·Jun 11·5 min read
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
TL;DR
Data ingestion from high-volume sources like IoT sensors and satellites often requires complex infrastructure management. Databricks developed Zerobus Ingest, a serverless streaming API that simplifies this process by automatically scaling resources.
✦ Why It Matters
Engineers can leverage Zerobus Ingest to simplify data ingestion processes and focus on analytics rather than infrastructure.
Key Takeaways
- Utilize Zerobus Ingest for serverless data ingestion.
- Eliminate infrastructure management with push-based API.
- Achieve high throughput for large datasets.
- Focus on data analysis instead of tuning systems.
Full Summary
High-volume data streams from sources such as IoT devices and satellites present challenges in terms of fast and reliable ingestion into queryable databases. Databricks created Zerobus Ingest, a fully managed, serverless streaming API that allows users to push data directly into Delta tables without needing to manage infrastructure.
It utilizes dynamic partitioning to automatically scale compute resources based on data volume, eliminating the need for pre-configuration. In benchmarking with NASA's NEOWISE dataset, Zerobus Ingest ingested 1 petabyte of data in less than 24 hours, achieving a throughput of 12 GB/s to a single table.
Key design features include a zero-copy protobuf decoder and a latency-optimized write-ahead log. These innovations enable continuous data ingestion without the complexities typical of traditional streaming architectures.
This advancement allows engineers to focus on data analysis rather than infrastructure management.