tailwindlabs / tailwindcss
github.com·23h ago
Developer Tools
Databricks Blog·Jul 1·5 min read
Granular Usage Attribution for dbt Pipelines with Query Tags
TL;DR
Data teams faced challenges in tracking costs and performance across multiple dbt (data build tool) models, leading to confusion in accountability. The introduction of Query Tags allows for automatic tagging of dbt models, enabling better tracking and optimization of queries.
✦ Why It Matters
Engineers can implement Query Tags to improve cost tracking and performance analysis in their dbt projects.
Key Takeaways
- Implement Query Tags for better cost tracking in dbt models.
- Use auto-injected tags to simplify performance debugging.
- Create custom tags for enhanced query categorization.
- Leverage SQL queries for efficient workload monitoring.
Full Summary
Data teams often struggle with understanding the costs and performance of their dbt (data build tool) models, especially when multiple models generate similar queries, making it hard to identify responsible teams. To address this, Databricks introduced Query Tags, which automatically inject tags like dbt_model_name into every query generated by dbt pipelines.
Users can also create custom tags for better categorization, such as team or environment. These tags are stored in system.query.history, allowing teams to perform straightforward SQL queries for cost attribution and performance monitoring.
An open-source dbt project is provided to demonstrate the end-to-end implementation of Query Tags, from configuration to creating cost attribution dashboards. This approach not only clarifies accountability but also enhances performance debugging and workload monitoring.
Overall, it empowers data teams to optimize their resources effectively.