TechPrufer
Get started
Atlassian
Atlassian logoAtlassian · Feature trail

Atlassian Webhook Delivery Platform

6 posts · Mar 2018 – Jul 2026

What this trail covers

Atlassian's webhook delivery platform, the 'events rail', has been architected to handle over 10 billion events per month, serving hundreds of thousands of customer sites across multiple production regions. It reliably delivers events from Atlassian products to Forge apps and other subscribed systems. The platform is designed for extensibility, allowing new event sources and destinations to be added without rebuilding the core infrastructure. Key architectural decisions focus on optimizing subscription management and delivery reliability. This post details the migration of StreamHub from Amazon Kinesis to Apache Kafka (via AWS MSK) to handle 150 billion events per day, leveraging Kafka Tiered Storage for cost reduction, and the resilience challenges encountered at scale, including broker networking limits, S3 request rate behavior, and managed service control plane dependencies. Design patterns for hardening StreamHub are also discussed.

All Atlassian trails
  1. The Events Rail: Inside Atlassian's Webhook Delivery PlatformThis post details the architecture of Atlassian's webhook delivery platform, referred to as the 'events rail'. It highlights the challenges of selective subscriptions, multi-tenant fairness, and the design philosophy of prioritizing delivery over latency. The architecture is a two-stage pipeline with an Events Queue and a Webhooks Queue, separating CPU-bound event matching from network-bound event delivery. Key improvements include layered fairness mechanisms and a per-surface configuration model for extensibility.
  2. Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily EventsThis post details the migration of StreamHub from Amazon Kinesis to AWS Managed Streaming for Apache Kafka (MSK) to handle a 150 billion daily event volume. It outlines the limitations of Kinesis at scale, including API reliability, cost of high partitioning, retention limits, and consumer throughput exhaustion. The adoption of Kafka Tiered Storage with a two-tier approach (local EBS for hot data, S3 for historical data) is highlighted as a key cost-saving and retention strategy. Significant challenges encountered at scale with MSK are discussed, such as broker network limits, instance sizing, tiered storage offload throughput, S3 request rate limits during retention changes, storage scaling cooldowns during incidents, managed control plane availability issues, and the slow, irreversible nature of scaling operations. Hardening strategies for StreamHub are also presented.
  3. Building a Multi Region Compliant Customer Data Lake at ScaleThis post details the architecture and implementation of Atlassian's customer data lake, focusing on the challenges of data replication and streaming ETL. It describes the decision to use logical replication over physical replication for data capture, the design of a 'replication bus' using Protobuf for event standardization and S3 offloading for large payloads, and the use of Apache Flink and Delta Lake for building a streaming ETL pipeline. The system aims to provide instant analytics, a data export path for cloud customers, and offload analytical workloads from operational databases.
  4. Lithium: distributed, ephemeral streaming pipelinesThis post introduces the Lithium Platform, a new system for building dynamic and ephemeral streaming data pipelines. It details the concept of 'Workplans' which are provisioned and deprovisioned at runtime, built on Kafka and Kafka Streams. The 'Bring your own Host' model allows consumers to integrate an SDK and host processors closer to their data. The architecture is divided into a control plane (managing workplan lifecycle and resource providers) and a data plane (executing data movement). Key features include tenant isolation, pipeline isolation for pause/resume/remediation, and a workplan specification for defining pipeline parameters. This extends the capabilities of traditional ETL systems by enabling on-demand, short-lived data movement.
  5. Introducing enhanced webhook securityIntroduced webhook secrets for Bitbucket Cloud, enabling HMAC signature verification of webhook payloads using SHA256 and a user-defined secret token. This enhances security by ensuring payload authenticity and protecting against unauthorized access. Provided example code in Java and Python for signature verification and detailed the process of adding secrets to webhooks.
  7. Why we chose Kafka for the Trello socket architectureThis post details Trello's migration from RabbitMQ to Kafka for their socket architecture. It explains the previous RabbitMQ implementation, its limitations (especially with network partitions and expensive queue/binding operations), and the problems encountered. The post then compares Kafka with other queuing providers and Kafka drivers, justifying the choice of Kafka and the `node-rdkafka` driver. The new architecture features a master-client model for socket servers, where the master handles filtering of delta updates locally. The migration resulted in improved reliability, reduced costs (5x reduction), and better observability metrics, with a drop in memory usage and an increase in CPU usage due to local filtering. A minor outage during an upgrade due to an improperly set `max_open_file` limit on Kafka servers is also noted.