Running out of runway: migrating Temporal Cloud's billing store to ClickHouse

This post details the migration of Temporal Cloud's billing store from a previous warehouse solution to ClickHouse. The migration focused on improving data processing latency, reducing steady-state costs, and increasing available headroom for future growth. Key technical contributions include: implementing a new data pipeline leveraging ClickHouse's efficient deduplication at read time, re-implementing customer-facing API queries and a scheduled aggregation workflow to read from ClickHouse, and developing a rigorous validation strategy involving parallel runs and a comparison framework to identify and categorize differences between the old and new systems. A significant defect caught was an integer overflow in ClickHouse's aggregation of large integer columns, which was resolved by casting before aggregation and adding a specific test case. The rollout strategy prioritized reversibility, migrating read-only APIs first and the critical aggregation path last, with a phased, account-by-account approach.