Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data
4/9/2025
This post details Discord's custom solutions for scaling dbt to handle petabytes of data and a large number of developers. It highlights challenges with default dbt configurations like slow re-compilation times (20+ minutes) and issues with incremental materialization and test table overwrites. The custom solutions aim to improve developer productivity, prevent breaking changes, and streamline complex calculations, making dbt a more robust and scalable data transformation platform.