Data Pipeline Deployment Capability

2 posts · Feb 2019 – Dec 2025

What this trail covers

Atlassian's data platform has evolved to include an opinionated deployment capability inspired by Kubernetes and Micros. This capability provides a declarative, metadata-driven, self-serve system for provisioning data pipelines, an extensible abstraction over execution capabilities, and a single governed deployment experience. It aims to address challenges of data discovery, trust, and user experience by modeling the user's natural workflow and collecting rich metadata for data discovery purposes.

All Atlassian trails

Dec 2025Building a Multi Region Compliant Customer Data Lake at ScaleThis post details the implementation of logical replication for change data capture (CDC) across Atlassian's diverse product ecosystem, utilizing a Protobuf-based event protocol and a 'replication bus' infrastructure. It describes the use of AWS Kinesis for real-time event streaming and Amazon S3 for initial data bootstrapping. The post further elaborates on the streaming ETL pipeline built with Apache Flink to process these events into Delta Lake tables, organized into raw and refined layers, enabling near real-time analytics and data exports for customers.
Nov 2024Lithium: distributed, ephemeral streaming pipelinesThis post introduces the Lithium Platform, an event-driven system built on Kafka and Kafka Streams for dynamic and ephemeral streaming data pipelines called 'Workplans'. It details the Workplan architecture (Source, Transform, Sink), the 'Bring Your Own Host' model where consumers integrate an SDK to provide processors, and the heterogeneous distribution of these processors across services. The post emphasizes pipeline isolation, tenant-level data separation, and the control plane/data plane architecture for managing workplans and resource providers. It also describes the Workplan specification and the role of Kafka Streams for state management.
May 2024Evolve your data platform with a Deployment CapabilityThis post introduces a novel deployment capability for Atlassian's data platform, inspired by Kubernetes and Micros but tailored for the data domain. This capability offers a declarative, metadata-driven, self-serve system for provisioning data pipelines, an extensible abstraction over execution capabilities, and a single governed deployment experience. It addresses the challenges of data discovery, trust, and user experience by modeling user workflows and collecting rich metadata. The system uses configuration metadata to decouple users from pipeline execution capabilities, allowing for easier platform upgrades. It also emphasizes upfront metadata capture for data discovery and governance, moving away from post-deployment inspection. The model includes entities like pipelines, steps, outputs (tables), databases, columns, and data products, with a schema for each. The implementation interfaces with execution capabilities like dbt via keywords in configuration files, providing an extensible mechanism for adding new capabilities.
Feb 2019Meet Bitbucket Pipes. 30+ ways to automate your CI/CD pipelineThis post announces the launch of Bitbucket Pipes, a new feature for Bitbucket Pipelines that allows developers to automate CI/CD workflows by selecting and configuring pre-built integrations (pipes) with external services. It highlights the benefits of reduced manual effort, simplified configuration, and maintained integrations. The post also encourages community contributions for new pipes and provides links to get started.
Advertisement