Data Mesh Architecture and Practices

4 posts · Sep 2019 – Sep 2024

What this trail covers

Confluent Cloud continues to evolve data mesh architectures and their implementation using streaming data platforms. This iteration focuses on leveraging generative AI to automate the creation of metadata for data governance, thereby improving data quality and enabling the development of more accurate AI-powered applications. This complements existing efforts in decentralizing data ownership, federated governance, and delivering data products as APIs, with a focus on cloud-native principles, scalability, and real-time data processing.

All Confluent trails

Sep 2024How Developers Can Use Generative AI to Improve Data QualityThis post introduces the application of generative AI to automate the creation of metadata for data governance. It explains how LLMs can be presented with data examples or production code to induce schemas and other machine-readable and human-readable metadata. The post emphasizes the need for human review of AI-generated metadata and highlights the role of data streaming platforms in facilitating the application of metadata at the time of data production and the creation of reusable data products. It also touches upon the emerging concept of AI governance and its dependence on data governance.
Oct 2023Telco Use Cases With Streaming Data Mesh at DISH WirelessThis post details how DISH Wireless is building a programmable 5G network using data mesh principles on Confluent Cloud. It highlights the adoption of cloud-native data streaming capabilities to decentralize data ownership, implement federated governance, and democratize access to data products through telco APIs. The architecture leverages Confluent Cloud's Kora engine for a flexible data platform that allows developers to customize connectivity and ensures performance and reliability. The post outlines how streaming data pipelines connect apps, networks, and infrastructure, enabling event-driven architectures for real-time monitoring, predictive analytics, and Industry 4.0 use cases in retail, manufacturing, and automotive sectors. It also discusses the benefits of a distributed architecture across heterogeneous environments and the use of Cluster Linking for replicating data from edge locations to cloud regions for processing and model training.
Mar 2023Benefits of Event-Driven Architecture (EDA) and Data MeshThis post introduces the Data Product Descriptor Specification (DPDS) and the Open Data Mesh (ODM) Platform as tools for defining and automating the lifecycle of streaming data products. It demonstrates a track-and-trace logistics example using Confluent Cloud components like Kafka topics, Schema Registry, and ksqlDB, highlighting the concepts of data ownership, scope, and interfaces for data products, and the role of a self-serve platform in managing their lifecycle.
Oct 2022Data Mesh - 7 Effective Practices to Get StartedThis post outlines seven effective practices for adopting a data mesh architecture: 1. Everything is evolvable, emphasizing adaptable governance and self-determinism. 2. Define a domain hierarchy for easier data storage and coordination. 3. Continuously manage complexity by isolating integrations and designing for resilience. 4. Refine ownership with a RACI model and the single-writer principle. 5. Define default implementations for a 'happy path' with idempotent producers and tolerant consumers. 6. Think deeply about time and data product lifecycles, including eventual consistency and SLAs. 7. Institute feedback loops with new KPIs and a technology adoption lifecycle for continuous improvement.
Jul 2022Modern Data Flow: The Key to Modern Data PipelinesThis post introduces the concept of treating data as a product, emphasizing the shift to decentralization and federation for better sharing and reuse. It highlights the adoption of agile and DevOps practices in data management, such as unit testing, CI/CD, and continuous observability. The post also discusses declarative transformation models, exemplified by dbt, which allow data transformation logic to be packaged into discrete, expressive, testable, and composable units. It emphasizes the benefits of modularity, reusability, and testability in data pipelines, contrasting them with traditional, monolithic approaches. The emergence of developer-friendly tooling that aligns with modern software delivery practices is also noted.
Advertisement
Jun 2021Best Practices for Distributed Domain-Driven Architecture on the Data MeshThis post details Saxo Bank's approach to implementing a data mesh architecture, focusing on distributed domain-driven design, a self-serve platform, and the convergence of data and product thinking. It outlines their operating model for data domains, the role of enterprise data architects and governance, and the use of Confluent as the foundational layer for their data fabric. The self-service platform is managed via GitOps, with separate repositories for operational configurations (topics, connectors, data quality rules) and schemas (Protobuf validation, code binding generation). DataHub is used to surface metadata and metrics for discoverability and lineage. The post emphasizes best practices for designing effective schemas, including choosing appropriate formats, employing consistent style guides, strong typing, documentation, and capturing information classification.
Jan 2021Powering Microservices with Event Streaming at SEIThis post details SEI Investments' transition from a monolithic architecture to an event-driven architecture using Confluent Platform. It outlines their phased approach, starting with CDC from Oracle GoldenGate to capture raw data events, then transforming these into business events by joining related table updates based on System Change Numbers (SCNs). The post emphasizes the role of Confluent Schema Registry for Avro schema normalization and evolution, enabling forward and backward compatibility for producers and consumers. It also describes using Confluent Replicator for delivering near-real-time data streams to clients. Finally, it touches upon their ongoing move to containerized microservices on Kubernetes, with Kafka as the communication backbone and Schema Registry for payload governance.
Sep 2019Introducing Derivative Event Sourcing for KafkaIntroduces the concept and implementation of derivative event sourcing, a pattern for deriving events from observed sources like database changes (CDC) rather than direct emission from services. It details the 'Observe -> Transform -> Emit' flow, using Oracle GoldenGate and Debezium as observation tools, and Kafka Streams for transformation. The post provides concrete examples of defining event profiles for different services and transforming observed database operations into standardized 'order created' events, ultimately emitting them to a unified Kafka topic.