TechPrufer
Sign inGet started

Blogs›Confluent›Data Mesh Architecture and Practices

Data Mesh Architecture and Practices

Data Mesh Architecture and Practices

4
posts
2019–2022

This post details the implementation of derivative event sourcing using Change Data Capture (CDC) and Kafka Streams to handle multiple, disparate order services. It addresses the challenge of integrating legacy systems by observing database changes (Oracle GoldenGate, Debezium for MySQL) and transforming them into a unified event stream. The approach simplifies event sourcing in complex environments by decoupling event emission from the original services, allowing for centralized transformation and emission to Kafka topics. This enables a more manageable and scalable event-driven architecture, especially when dealing with older or acquired applications.

2022

Data Mesh - 7 Effective Practices to Get Started

10/11/2022

This post defines data mesh as a conceptual framework for data architecture, focusing on organizational agility, simplicity, and flexibility. It details seven key practices for implementing a data mesh: 1. Ensuring evolvability through governance and self-determinism. 2. Defining a domain hierarchy for data storage and coordination. 3. Continuously managing complexity by isolating integrations and designing for resilience. 4. Refining ownership models, advocating for the single-writer principle. 5. Establishing default implementations for data consumers and producers, promoting idempotency and tolerance for out-of-order delivery. 6. Deeply considering the temporal aspects of data products, including lifecycles and consistency models. 7. Instituting feedback loops with new KPIs and technology adoption lifecycles to drive continuous improvement.

2021

Best Practices for Distributed Domain-Driven Architecture on the Data Mesh

6/23/2021

This post details Saxo Bank's practical implementation of a data mesh architecture, building upon the conceptual framework. It highlights the use of Confluent as the data fabric foundation, GitOps for operational and schema management (including Protobuf and Great Expectations), and DataHub for metadata and discoverability. Key technical aspects include federated domain ownership, self-service platform design, and the convergence of data and product thinking with an emphasis on effective schema design and metadata utilization.

Powering Microservices with Event Streaming at SEI

1/22/2021

This post details SEI Investments' transition from a monolithic architecture to an event-driven architecture using Confluent Platform. It outlines their phased approach, starting with CDC from Oracle GoldenGate to capture raw data events, then transforming these into business events by joining related table updates based on System Change Numbers (SCNs). The post emphasizes the role of Confluent Schema Registry for Avro schema normalization and evolution, enabling forward and backward compatibility for producers and consumers. It also describes using Confluent Replicator for delivering near-real-time data streams to clients. Finally, it touches upon their ongoing move to containerized microservices on Kubernetes, with Kafka as the communication backbone and Schema Registry for payload governance.

2019

Introducing Derivative Event Sourcing for Kafka

9/6/2019

Introduces the concept and implementation of derivative event sourcing, a pattern for deriving events from observed sources like database changes (CDC) rather than direct emission from services. It details the 'Observe -> Transform -> Emit' flow, using Oracle GoldenGate and Debezium as observation tools, and Kafka Streams for transformation. The post provides concrete examples of defining event profiles for different services and transforming observed database operations into standardized 'order created' events, ultimately emitting them to a unified Kafka topic.