Engineering the Forge Billing Platform for Reliability and Scale

This post details the engineering of Atlassian's Forge Billing Platform, focusing on the reliable measurement, attribution, and billing of usage for the Forge serverless app development platform. It outlines the architecture from Forge services emitting usage events through Streamhub to a shared usage pipeline (UTS) that normalizes and verifies data before handing it off to internal billing systems. Key technical challenges addressed include handling duplicates and late events for counter-style and gauge-style metrics (like invocations and storage), ensuring idempotency and determinism in data processing, and translating billing state into entitlements and enforcement. The system leverages Plato (Databricks/S3) for data lake storage and StarRocks for query-optimized hot tier tables, with Flink jobs for data refinement. The goal is to provide developers with visibility and control over their spend through the Developer Console and APIs.