Scaling and Operating a Large dbt Project on Databricks: IFCO's Data Team on Performance, Visibility, and Debugging

This post details the introduction of granular usage attribution for dbt pipelines by leveraging query tags. This enhancement allows for more detailed tracking and analysis of pipeline performance and resource consumption, enabling better cost allocation and performance monitoring by associating specific query tags with dbt pipeline executions.