Google is a leader in The Forrester Wave™: Public Cloud Platforms, Q3 2026
cloud.google.com·4d ago
Cloud & Infrastructure
Google Cloud Blog·1d ago·5 min read
The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud
TL;DR
Pine59 upgraded its data pipeline orchestration to Apache Airflow 3 on Google Cloud, achieving significant improvements in processing speed and reliability. This transition enhanced their MLOps capabilities and developer workflow.
✦ Why It Matters
Consider upgrading your orchestration tools to improve data pipeline performance and reliability.
Key Takeaways
- Assess your current orchestration tools for scalability.
- Consider migrating to Managed Airflow for better performance.
- Implement custom plugins to enhance developer workflows.
- Optimize ML workloads with dedicated infrastructure.
Full Summary
Pine59, a provider of location intelligence data, faced challenges with its data pipelines as workloads increased. To improve performance, the company migrated from a monorepo setup to Managed Airflow (Gen 3) with Airflow 3, which significantly enhanced processing speed and stability.
The transition involved optimizing machine learning (ML) inference workloads by integrating a dedicated Google Kubernetes Engine (GKE) cluster. As a result, the Daily Foot Traffic pipeline's processing time decreased from nearly 38 minutes to less than 26 minutes, a 32% improvement.
Additionally, custom plugins were developed to enhance the developer experience, allowing for faster debugging and troubleshooting. Overall, the migration has positioned Pine59 for future growth and efficiency in handling large data volumes.