Event-Driven Architectures

5/11/2022

This post details Walmart's implementation of Apache Kafka for real-time omnichannel replenishment. It describes their micro-batch architecture, which processes tens of billions of messages daily for nearly 100 million SKUs. Key technical aspects include using Kafka for ingesting data via Change Data Capture (CDC), denormalizing it for accessibility, and processing it in a planning engine. The system publishes plans through various Kafka topics for different consumers. The post highlights the scale of their Kafka deployment (18 brokers, 20+ topics, 500+ partitions per topic). Architectural considerations include real-time input streaming, multi-tenancy, enabling diverse downstream consumers, and ensuring tight contracts between inputs and the replenishment engine. Platform decisions like active-passive data replication, automatic VM recovery, Kafka retries, and an in-house audit/replay mechanism are discussed. Significant tuning and optimizations are detailed for both producers (custom partitioning strategy using murmur hash, linger.ms, batch size, acks='all') and consumers (max.poll.records, max.poll.interval.ms, disabling enable.auto.commit, session.timeout.ms, heartbeat.interval.ms).

11/8/2021

This post details Confluent's strategies for scaling Apache Druid for real-time cloud analytics. It introduces splitting data sources by cardinality, implementing data tiering with hot/default tiers and Kubernetes StatefulSets, and query laning with a 'slow-lane' configuration. The post also covers optimizing MiddleManager and Historical node instance types (e.g., c5.9xlarge, i3.2xlarge) for better resource utilization and cost-effectiveness, and discusses compaction strategies to reduce segment counts for improved query performance. It highlights the use of Druid for customer-facing monitoring, billing, and internal operations.

10/21/2020

This post details the technical evolution of Tesco's Risk Engine from a JMS-based system to an event-driven architecture using Apache Kafka and Kafka Streams. It outlines the initial challenges with JMS, including lack of scalability, persistence, and concurrency. The rebuild introduced a new event-driven design with finer-grained topics (handshakes, identifications, verification, session results) and leveraged Kafka Streams for stream processing. The post details the initial 'false start' with a Session Aggregator, highlighting issues with high memory demand, state management (RocksDB cache), frequent rebalancing due to lack of cooperative rebalancing, and data skew caused by large 'elephant' aggregates. Remediation efforts included limiting aggregate size and scaling, but ultimately a redesign was necessary. The improved architecture emphasizes multiple independent and parallel pipelines, avoiding a monolithic aggregate, and focusing on specific goals like blocking malicious IP addresses. The post discusses the technical trade-offs and lessons learned during the migration and redesign process.

9/10/2020

This post explains why Apache Kafka does not natively support message prioritization, contrasting its event streaming architecture (commit log, immutability, partitions) with traditional messaging systems. It details the challenges of implementing prioritization directly within Kafka due to its design. The post then proposes and explains the 'bucket priority pattern' as a practical solution. This pattern involves producing messages to different topics (buckets) based on their priority, allowing consumers to process higher-priority topics first. It highlights how this approach avoids buffering and sorting complexities, enabling effective prioritization by leveraging Kafka's topic and consumer group mechanisms.

6/13/2019

This post details the use of Apache Kafka for real-time data streaming in astronomical sky surveys, specifically the Zwicky Transient Facility (ZTF). It highlights the challenges of handling millions of transient object detections per night and the need for a scalable, real-time alert distribution system. The post discusses the selection of Apache Kafka for its consumer offset management and rewind capabilities, and Avro as a compact and schema-evolvable serialization format. It details the alert data pipeline, including image differencing, Avro serialization, Kafka production, and mirroring to other Kafka clusters. The system is designed to handle up to 2 million alerts per night, with each alert packet around 60 KB, resulting in up to 70 GB of data. The post also touches upon the ease of use for scientists, with Python clients and the ability to deploy custom filters in Docker containers. The system enables near real-time science by distributing alerts within approximately 20 minutes of image capture and four seconds from the main Kafka hub to external consumers.

3/19/2019

This post contrasts Service-Oriented Architecture (SOA) with Event-Driven Architecture (EDA), highlighting the benefits of EDA for real-time data processing, decoupling, and deriving insights from data in flight. It emphasizes the role of event streaming platforms as intelligent analytical engines and contrasts the request-response model of SOA with the publish-subscribe model of EDA. Key advantages of EDA discussed include improved service availability, reduced coupling, better support for consistency through event sourcing, and the ability to perform streaming analytics.

10/15/2018

This post defines and categorizes modern event-driven architectures into four patterns: Global event streaming platform, Central event store, Event-first and event streaming applications, and Automated data provisioning. It explains how these patterns leverage streaming systems like Apache Kafka for real-time data processing, storage, and self-service data provisioning, moving beyond traditional messaging systems.

9/6/2018

This post details Stitch Fix's decision to build a self-service, centralized data integration platform using Apache Kafka, moving away from database snapshots. It highlights the challenges of manual data integration and the benefits of an event-driven architecture for Data Scientists. The post explains the design philosophy of self-service tooling and introduces the concept of 'The Data Highway' as a unified platform for data access. It also discusses the technical considerations for choosing Kafka over Kinesis and evaluates alternative data integration solutions like Apache Beam, Apache Storm, Logstash, and Kafka Connect, ultimately favoring Kafka Connect for its ecosystem and extensibility.

7/18/2018

This post introduces the analogy of microservices to functional programming within the context of event-driven architectures. It proposes a model where actions are written to an immutable log (like Kafka), and functions react to process these events to form aggregations representing current state. It details the use of CQRS and Event Sourcing, and categorizes microservices into 'Functional Core' (pure transforms) and 'Imperative Shell' (shared infrastructure).

3/13/2018

This post introduces event sourcing as a pattern that can be implemented using Apache Kafka. It explores different strategies for storing events in Kafka (single topic, topic-per-entity-type), discusses basic operations like reading the current state by folding events, and details how Kafka Streams and state stores can be used to efficiently manage and query entity states. The post also covers fault tolerance through changelog topics and standby replicas, and discusses consistency guarantees (at-least-once and exactly-once) achievable with Kafka transactions. Finally, it touches upon handling side-effects triggered by events.

11/29/2017

This post introduces a functional programming analogy for microservices, contrasting it with the object-oriented analogy. It proposes an event-driven architecture where microservices act as reactive functions processing immutable events from a durable log. Key concepts like CQRS, Event Sourcing, and the 'single writer' principle are discussed. Microservices are categorized into 'Functional Core' and 'Imperative Shell' (materialized views and side-effect execution). Kafka, Kafka Streams, and Kafka Connect are presented as foundational technologies for this architectural style. A simplified banking use case is used to illustrate the concepts.

9/6/2017

This post details The New York Times' implementation of a log-based architecture using Apache Kafka as the 'Monolog' to store all published content. It contrasts this with previous API-based approaches, highlighting the benefits of a log as the source of truth for creating materialized views, simplifying schema evolution, and enabling immutable deployments of stateful systems. The post also explains why Kafka's log retention and ordered consumption capabilities are crucial for this use case, differentiating it from message brokers like Google Pub/Sub and AWS SNS/SQS/Kinesis.

8/17/2017

This post introduces the concept of a 'database unbundled' by leveraging Kafka as a durable log and Kafka Streams for stream processing. It explains how this architecture enables the creation of decentralized, continuously updated materialized views that can be embedded within services. The benefits highlighted include improved read performance, reduced coupling, avoidance of shared mutable state, and enhanced agility. The post contrasts this with traditional databases and service APIs, illustrating how this approach can solve challenges in data sharing and manipulation within microservices.

8/2/2017

This post introduces Event Sourcing as a pattern where events are the single source of truth, contrasting it with traditional database-centric approaches. It explains how Kafka's log-structured nature and features like retention-based and compacted topics make it suitable as an event store. The post details two primary methods for ingesting events into Kafka: direct event sourcing (writing events as they occur) and using Change Data Capture (CDC) from databases to stream inserts, updates, and deletes. It highlights the benefits of CDC for migrating legacy systems and achieving consistency by writing through a database into Kafka. The post also discusses the trade-offs in event modeling (whole events vs. deltas) and the use of Kafka transactions for handling multiple output events.

7/19/2017

This post introduces Apache Kafka as a message backbone for service architectures, contrasting it with traditional message brokers. It details Kafka's log-structured design, linear scalability, load segregation via quotas, strong ordering guarantees (using keys and single-partition topics), durability through replication (with configuration recommendations like `min.insync.replicas`), and load balancing of services through partition assignment. It also discusses producer configuration for retries (`max.inflight.requests.per.connection`) and broker configurations for durability (`log.flush.interval.messages`).

5/31/2017

This post introduces the concept of building microservices using an event-driven architecture with Apache Kafka as the backbone. It contrasts this with traditional request-response models, highlighting the challenges of synchronous interactions in distributed systems. The post defines Commands, Events, and Queries, and illustrates how an event-driven flow can reduce coupling and improve pluggability. It introduces the 'Query by Event-Carried State Transfer' pattern to avoid remote queries and internalize them by caching event data within services, leading to better decoupling, autonomy, and efficient joins. The 'Single Writer Principle' is also presented as a way to manage consistency and validation by assigning responsibility for event propagation to a single service. The post uses examples of customer orders and stock management to demonstrate these concepts.

12/13/2016

This post introduces the concept of the 'Data Dichotomy' in microservices, where services aim to encapsulate data while data systems aim to expose it. It argues that traditional approaches to sharing data between microservices (service interfaces, messaging, shared databases) are insufficient. The post proposes event-driven architectures, specifically leveraging distributed logs like Kafka, as a solution. It highlights Kafka's ability to provide retentive, scalable streams and enable stateful stream processing within consuming services, thereby addressing the data dichotomy and facilitating independent service evolution.

10/25/2016

This post details the architectural pattern of integrating streaming data (via Kafka and Confluent Platform) with at-rest data (from traditional databases and file systems) to provide context for real-time event processing. It uses a fraud detection use case as a concrete example, illustrating how ATM transactions streamed through Kafka can be enriched with static customer data from at-rest stores to identify fraudulent activity. It also mentions other industry use cases like hotel inventory management and healthcare data analysis, emphasizing the versatility of this combined architecture.

10/24/2016

This post details Walmart Labs' adoption of Apache Kafka for their 'Item Setup' process, which involves normalizing, matching, classifying, and processing product data, offers, pricing, inventory, and logistics. They model changes as event streams recorded in Kafka, processed by technologies like Apache Storm, Spark, plain Java, Reactive Microservices, and Akka Streams. Kafka serves as a company-wide data hub, enabling hundreds of microservices and teams to operate autonomously. The post highlights the growth in Kafka usage, with billions of updates processed daily, and the development of operational tools for monitoring SLAs, latencies, and backlogs. Challenges include managing service topologies, schema management, multi-DC data balancing, and shielding consumers from business-impacting changes. The adoption has enabled agile development and faster onboarding of sellers and product listings, with Kafka also powering a near real-time search index.

9/7/2016

This post introduces the connection between Event Sourcing, CQRS, stream processing, and Apache Kafka. It explains how Kafka acts as a backbone for Event Sourcing by storing immutable event logs. It further details how Kafka Streams can be used to implement the event handler component in CQRS architectures, either by writing to an external datastore or by modeling application state as local, partitioned, and durable state within Kafka Streams itself. This local state is exposed via KTables and benefits from Kafka's fault tolerance, availability, and scalability.

6/14/2016

This post introduces a streaming analytics stack architecture using Apache Kafka and Druid. It explains how Kafka acts as a buffer for event delivery and how Druid consumes these events for real-time analytical queries. A tutorial is provided for setting up and visualizing data from Wikipedia edits using Kafka and Druid.

8/27/2015

This post explains the role of distributed consensus services like Apache ZooKeeper in distributed systems, focusing on how Apache Kafka uses ZooKeeper for metadata management, replica set coordination, and leader election. It clarifies that ZooKeeper provides a 'coordination kernel' rather than an explicit consensus API, and that its atomic broadcast capabilities are fundamental to implementing replicated state machines, which Kafka uses for data replication. The post also highlights potential pitfalls in reasoning about agreement in distributed systems and provides a concrete example of how Kafka leverages ZooKeeper's sequential nodes to break ties and achieve agreement on values.

8/1/2015

This post draws parallels between the Unix philosophy of composability (programs doing one thing well and expecting output to be input for another program) and the design of Apache Kafka and Apache Samza. It uses examples of Unix command-line tools (awk, sort, uniq) to illustrate the concept of pipelines and how they can process large amounts of data efficiently. The post argues that Kafka, with its stream processing capabilities, embodies this philosophy by enabling the chaining of data processing stages, similar to Unix pipes, facilitating the creation of robust and scalable distributed data systems.

6/16/2015

This post explains the value of Apache Kafka in the Big Data ecosystem, highlighting its role as a scalable pub/sub system for ingesting high volumes of data in real-time. It discusses the limitations of traditional monolithic databases for handling large, diverse datasets and the emergence of specialized distributed systems. Kafka is presented as a central stream data platform that feeds these specialized systems (e.g., Hadoop, key/value stores, search engines, stream processors) by persisting data to disk and supporting multiple subscriptions for both real-time and batch consumers. The post also touches upon the importance of data curation early in the ingestion process.

5/29/2015

This post introduces the fundamental concept of logs (append-only, totally ordered sequences of records) as a core building block for robust, scalable, and maintainable data infrastructure. It contrasts this with the problematic 'dual writes' approach for data integration, explaining how dual writes lead to race conditions and inconsistencies. The post advocates for an event-driven architecture based on logs, drawing parallels to how databases, replication, and distributed algorithms like Raft utilize logs. It highlights the benefits of this approach for managing search indexes, caches, and enabling richer data analysis, ultimately leading to more robust and scalable systems.

4/13/2015

This post details the combination of Luwak and Apache Samza for implementing real-time full-text search on streams. It explains the concept of streaming search, contrasting it with after-the-fact search, and introduces Luwak as a library for efficient streaming search by indexing queries. The post also highlights Apache Samza as a stream processing framework based on Kafka, enabling distributed real-time computations.

3/1/2015

This post introduces Apache Samza as a distributed stream processing framework, highlighting its potential to revolutionize database architecture by treating data as immutable facts and processing them in real-time. It discusses replication and secondary indexes as examples of database operations that can be re-imagined through a stream-processing lens, emphasizing the benefits of simpler code, better scalability, robustness, and lower latency.

2/25/2015

Announces Confluent Platform 1.0, which includes Apache Kafka 0.8.2.0, a schema management layer (Schema Registry), Java and REST clients, and Camus for Kafka to Hadoop ETL. This marks the initial release of a comprehensive stream data platform.

11/1/2014

This post announces the formation of Confluent, a company dedicated to Apache Kafka. It details the origin of Kafka at LinkedIn, driven by the need for scalable distributed data systems and real-time data processing. The post explains Kafka's core concepts as a horizontally scalable messaging system that acts as a replacement for traditional brokers, a data synchronization tool, and a foundation for real-time analytics. It highlights Kafka's design for durability, persistence, and scalability, and its evolution into a central nervous system for companies. The post also touches upon the limitations of existing open-source Kafka and the need for surrounding capabilities to build complete real-time data pipelines, which Confluent aims to address.