TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Event-Driven Architectures

32 posts · Oct 2014 – May 2026

What this trail covers

Confluent Cloud continues to evolve event-driven architectures by enabling real-time alerting and automated response systems. This iteration focuses on designing and implementing intelligent alerts using Apache Kafka, moving beyond simple notifications to create systems that keep business operations resilient and ahead of critical events. This includes threshold-based, anomaly detection, composite event alerts, and automated action triggers, emphasizing the use of stream processing for real-time data.

All Confluent trails
  1. InfiniteWatch + Confluent: Turning Customer Interaction Data into Real-Time IntelligenceThis post details how InfiniteWatch integrated Confluent's event streaming platform as the backbone for their AI-native customer interaction intelligence platform. It highlights the rapid integration from joining the Confluent for Startups program to production deployment within one quarter. The reference architecture emphasizes Confluent's role in ingesting, buffering, processing, replaying, enriching, and routing customer interaction events. Key capabilities enabled by Confluent include burst absorption, durable event preservation, service decoupling, replayable intelligence for AI systems, and multi-destination routing. The five-stage deployment process (Capture → Stream → Enrich → Understand → Act) is described, with Confluent facilitating the 'Stream' and 'Enrich' stages. The post underscores the shift towards treating customer interaction data as a continuous, real-time operational system powered by event streaming.
  2. Streaming Data Integration with Apache Kafka® vs ETLThis post contrasts traditional ETL data integration with streaming data integration using Apache Kafka. It details the advantages and disadvantages of ETL, particularly its limitations in data reuse, batch processing, and siloed data. The post then highlights the benefits of streaming data integration with Kafka, including real-time and batch support, one-to-many and many-to-one source/destination patterns, reusable data products, and unified real-time and historical data analysis. It advocates for adopting ETL best practices like user-friendly interfaces and metadata catalogs within a streaming data integration framework.
  3. How to Build Real-Time Alerts to Stay Ahead of Critical EventsThis post details the design and implementation of real-time alerting systems using Apache Kafka. It introduces four key alerting patterns: threshold-based, anomaly detection, composite event alerts, and automated action alerts. For each pattern, it outlines typical use cases, architectural considerations (e.g., using Flink or Kafka Streams for correlation), and the benefits and drawbacks. The post also emphasizes the importance of designing trustworthy alerts by addressing alert fatigue through filtering and prioritization, illustrating an alert prioritization pyramid. It highlights how organizations can achieve faster response times (seconds vs. minutes/hours) by leveraging event-driven alerts for critical events.
  4. How to Build Real-Time, Event-Driven Automation for Business InsightsThis post elaborates on event-driven workflows as a means to achieve real-time business insights and automation. It emphasizes the benefits of immediate responsiveness, scalability, and operational efficiency by leveraging data streaming platforms like Kafka and stream processing engines like Flink or Kafka Streams. The post details how these workflows can power AI agents for automated decision-making and action execution, reducing the latency between data arrival and actionable insight. It also provides architectural overviews and real-world use cases in fraud detection, customer churn prevention, and inventory optimization, highlighting the transition from traditional BI to event-driven approaches.
  5. Why Microservices Need Event-Driven Architectures for Agility and ScaleThis post argues for the adoption of event-driven architectures (EDA) for microservices, contrasting it with synchronous API-based communication. It details the problems with synchronous microservices, including API dependency chains leading to cascading failures, latency, blocking, and scaling issues, and tight coupling that limits autonomy. The post then explains how EDA, powered by Apache Kafka, solves these challenges through asynchronous communication, decoupling services, using event streams as a reliable source of truth, enabling smarter on-demand scaling, and providing governance. It highlights business outcomes like faster innovation, lower risk, and better customer experiences, and provides industry use cases in banking, retail, and telecommunications. The core technical contribution is the conceptual framework and justification for using Kafka-based EDA to enhance microservice agility and scale.
  7. How to Build Real-Time Apache Kafka® Dashboards That Drive ActionThis post details how to build real-time Apache Kafka dashboards that drive action by leveraging data streaming. It explains the architecture for ingesting data into Kafka, processing it with stream processors like Flink, Kafka Streams, or ksqlDB to create materialized views, and then connecting these to BI tools or custom applications. It emphasizes data modeling for real-time consumption, including denormalization and partitioning strategies, and discusses delivery patterns such as read-optimized views, search-optimized indexes, and app-embedded state using Kafka Streams interactive queries.
  8. Integrate Kafka with AWS EventBridge Using ConfluentIntroduces the Amazon EventBridge Sink Connector for Confluent Platform, enabling seamless integration of Kafka event streams with AWS EventBridge. Details use cases in financial services, manufacturing, and healthcare. Provides step-by-step instructions for setting up AWS resources (CLI, Event bus, rule) and Confluent Platform (connector installation and configuration). Outlines best practices for production, including aligning service quotas, implementing dead-letter queues, enforcing least-privilege IAM, handling large events with S3 offloading, and leveraging logging for debugging.
  9. The Future of AI Agents Is Event-Driven | ConfluentThis post argues that the future of AI agents is event-driven, positioning EDA as the critical infrastructure for their scalability and interoperability. It contrasts agentic systems with traditional control logic and microservices, emphasizing how EDA's asynchronous, event-based communication pattern facilitates loose coupling. This allows agents to access diverse data sources, use tools, and share outputs with other agents and enterprise systems (CRMs, CDPs) without creating rigid dependencies. The post uses historical examples like early social networks to underscore the importance of scalable architecture for adoption and survival, framing agent scaling as a distributed systems problem best solved by EDA.
  10. How to build a multi-agent orchestrator using Flink and KafkaThis post introduces the concept of building an event-driven orchestrator for multi-agent systems using Apache Flink and Apache Kafka. It details a reference architecture where Flink interprets and routes messages, and Kafka serves as short-term shared memory. A hands-on implementation for an AI-driven sales development representative (SDR) system is presented, showcasing how Flink SQL with external model inference can dynamically route messages to specialized agents based on LLM decisions. The post emphasizes the benefits of event-driven architectures for agent communication, decoupling, and scalability, and highlights the orchestrator pattern for managing complex agent interactions.
  11. Why Google’s Agent2Agent Protocol Needs Apache KafkaThis post introduces Google's Agent2Agent (A2A) protocol as a standard for inter-agent communication, analogous to HTTP for the web. It highlights the limitations of A2A's default point-to-point communication model (HTTP, JSON-RPC, SSE) for enterprise-scale agent ecosystems due to connection explosion, tight coupling, limited visibility, and orchestration challenges. The post argues for adopting an event-driven architecture, specifically using Apache Kafka, as a communication layer for A2A. It proposes using Kafka topics for publishing A2A task requests and status updates, enabling multiple consumers, decoupled communication, durable storage, and real-time orchestration. Three potential implementation patterns are discussed: Kafka as a transport layer, Kafka for task routing and fan-out, and a hybrid orchestration pattern.
  12. Building Real-Time Multi-Agent AI With ConfluentThis post details the technical architecture of Agent Taskflow, a multi-agent orchestration platform built on Confluent's data streaming platform. It explains how Confluent enables real-time communication, observability, and fault tolerance for AI agents. Key technical contributions include: an event-first architecture where every agent action, thought, and decision is an event flowing through Kafka topics; integration of Confluent connectors (PostgreSQL Sink, Apache Iceberg Sink, custom webhook source) for data persistence and external triggers; utilization of Schema Registry for strict event schema compatibility and validation; and the application of RAG by vectorizing Kafka events for context retrieval. The post emphasizes how this event-driven approach allows for independent scaling of agent teams and observability, enabling real-time responsiveness and autonomous agent collaboration.
  13. The AI Silo Problem: How Data Streaming Can Unify Enterprise AI AgentsThis post introduces the concept of using a Data Streaming Platform (DSP) as an event-driven backbone for inter-agent communication to solve the 'AI silo problem'. It proposes an agent registry for structured governance, dedicated Kafka topics for each agent, and stream processing with an LLM-based event mapper for intelligent routing of insights. This approach aims to enable AI agents to communicate and coordinate actions in real-time across heterogeneous platforms, moving beyond static mappings found in frameworks like AutoGen.
  14. Real-Time Toxicity Detection in Games: Balancing Moderation and Player ExperienceThis post details a real-time toxicity detection system for games, combining Confluent Cloud and Databricks. It uses Kafka for message transport, Apache Flink for real-time triage with a lightweight ML model, and Databricks for deeper AI analysis of high-risk messages. Tableflow is used to integrate Kafka streams with Delta tables for Databricks. The system balances moderation with player experience by analyzing context (e.g., friend relationships) and continuously learning from new data and human moderator input.
  15. Your GenAI Project Needs a Data Streaming PlatformThis post argues that Generative AI projects are often hindered by the "data liberation problem," where data is siloed, duplicated, and accessed via slow, complex batch processes (ELT/reverse ETL). It posits that a data streaming platform is the solution, enabling real-time data flow, decoupling AI systems from infrastructure, making unstructured data AI-ready, improving data quality, and connecting hybrid ecosystems. The core technical argument is that streaming platforms facilitate a "shift-left" approach to data processing, making data available at the source for immediate use by AI.
  16. Four Design Patterns for Event-Driven, Multi-Agent SystemsThis post introduces four event-driven design patterns (orchestrator-worker, hierarchical agent, blackboard, market-based) for building multi-agent systems, drawing parallels to event-driven microservices and Apache Kafka. It details how to adapt these patterns using Kafka topics and consumer groups for communication and coordination, simplifying operations and improving fault tolerance compared to traditional request/response or direct communication models. The post also emphasizes a reactive agent model (input, processing, output) and the shift from request/response to event-driven interactions.
  17. Revolutionizing Computational Chemistry with Data StreamingThis post details how Atomic Tessellator uses Confluent and Apache Kafka to build a scalable platform for computational chemistry research, leveraging generative AI. They moved from a batch-based ETL system (RabbitMQ, Celery) to a data streaming architecture to handle large, siloed molecular databases. The new architecture allows for flexible scaling of pipelines, real-time adjustment of parameters like 'entropy injection rate', and improved visibility into pipeline status via JMX metrics. They plan to explore Apache Flink for pipeline templates.
  18. Busie: Building a Charter Platform From the Ground Up with Confluent CloudBusie built their Charter Platform on Confluent Cloud, migrating from a monolithic and then RESTful microservices architecture to an event-driven architecture. They leveraged Confluent Cloud's managed connectors to ingest data from 13 PostgreSQL and MongoDB databases, breaking down data silos. Their core application, the 'Charter Experience Finder' (ChEF), aggregates and enriches data from disparate sources within Confluent Cloud to create a unified data model optimized for search, pagination, and filtering, reducing load times by nearly 300%. They plan to further utilize Confluent Cloud's stream processing with Apache Flink for real-time analytics dashboards, yield-based pricing, and real-time vehicle location reporting.
  19. Enhanced Cybersecurity with Real-Time Log Aggregation and Analysis | ConfluentThis post details the implementation of a real-time log aggregation and analysis system for cybersecurity using Confluent Cloud. It highlights the challenges of managing large data volumes and ensuring real-time processing, and proposes a solution leveraging Confluent's data streaming platform. Key technical aspects include: real-time data ingestion from heterogeneous sources (login logs, firewall logs, IPS logs, web logs) using FilePulse source connector; stream processing with Flink SQL for data enrichment and correlation (e.g., window aggregation for suspicious logins, joining IDS and login streams for custom alerts); and downstream consumption via Elasticsearch and Google BigQuery sink connectors for dashboards and AI/ML modeling. Sample Flink SQL queries are provided for detecting multiple login attempts from the same source IP and identifying login attempts from known attack source IPs. The post emphasizes the benefits of a shift-left approach with stream processing, enabling data reuse and reducing latency for threat detection and response.
  20. How Moniepoint Transformed Financial Services for 2 Million Businesses in Africa with ConfluentThis post details how Moniepoint migrated from a legacy database replication system with significant lag issues to Confluent Cloud to achieve near real-time data replication for their financial services. Key technical contributions include: elimination of replication lags by adopting a horizontal scaling solution with Confluent, enabling real-time migration of transaction data to a reporting store without additional infrastructure costs, and decoupling services by logging transactions on Confluent's managed streaming layer to expose events for independent consumption and decision-making. The architecture shift allowed Moniepoint to scale from hundreds of thousands to millions of requests, resolving issues with database locks, low throughput, and scalability. The use of Confluent Cloud for transaction and notification management provided a reliable and scalable foundation, preventing downtime and enabling efficient handling of millions of requests.
  21. How Network Automation with Confluent Helps BT Stream TV Signals Across the UKThis post details how British Telecom (BT) built a new network automation system called Vena to stream TV signals across the UK. Vena leverages Confluent Platform and Apache Kafka as its underlying messaging system to reliably distribute and respond to messages between physical equipment, microservices, and user interactions. The system automates site provisioning, service ordering, and service assurance monitoring. A key use case involves event-driven microservices that automatically provision network links when both ends are established, triggered by LLDP events streamed via Confluent. The post emphasizes the importance of data streaming for real-time data, scalability, event-driven architectures, efficiency, and high availability (99.999%) required for critical broadcast services.
  22. Real-Time Risk Analysis with Stream ProcessingThis post argues that event-driven design and Confluent Cloud's real-time capabilities offer a superior approach to risk aggregation and analysis compared to traditional batch processing. It highlights the limitations of batch processing in financial risk management, citing historical failures and the inefficiencies of micro-batching. The author advocates for adopting event-driven architectures to ingest and process data continuously, enabling timely risk assessments and proactive decision-making. The post contrasts the 'data at rest' paradigm with the benefits of real-time data streaming for dynamic and complex risk calculations.
  23. How Cloud Services Are Helping NASA Take Kafka Farther, FasterThis post details NASA's adoption of cloud computing and data streaming, specifically Apache Kafka, to support various mission teams and projects. It highlights the challenges of a distributed IT organization and how a managed cloud service (Confluent Cloud) lowered the barrier to entry for cloud adoption. The post discusses specific projects like GCN (General Coordinates Network) for real-time astronomy alerts and DAPHNE for petabyte-scale science data processing using cloud-based serverless computing. It also touches upon NASA's future initiative, the Data and Reasoning Fabric, aimed at enabling future air mobility through massive data streaming. The importance of immediacy in scientific missions and the benefits of managed Kafka services over self-managed or other managed solutions are emphasized.
  24. Maximizing the Power of AI with Data StreamingThis post discusses how data streaming, powered by Apache Kafka, enables AI by providing continuous training, efficient data synchronization, real-time model application, and high-volume data processing. It highlights the benefits of a real-time data mesh for democratizing data access and accelerating AI ROI. The post also emphasizes the importance of data governance, security (RBAC, ABAC), and transforming data into actionable intelligence through Confluent's platform.
  25. With Confluent Cloud, Loggi boosted productivity and cut costsThis post details Loggi's journey in building an event-driven architecture using Confluent Cloud. It highlights the challenges of a monolithic system, the decision to adopt Kafka and Confluent Cloud for scalability and ease of management, and the lessons learned. Key technical contributions include the implementation of the Outbox Pattern for transactional guarantees, the design of an Event Broker microservice for event consumption via gRPC, and the use of Protocol Buffers for event schemas. The post also discusses the importance of simplicity, user needs, potential bottlenecks (like the Event Broker as a single point of failure), and the essential role of observability and performance tuning.
  26. Delivery Guarantees in Messaging SystemsThis post explains the concepts of at-most-once, at-least-once, and exactly-once delivery guarantees in messaging systems, using the analogy of teleportation and the Two Generals Problem to illustrate the challenges of achieving reliability over unreliable networks. It highlights that true exactly-once delivery is impossible over unreliable channels and discusses the trade-offs involved in handling duplicates or accepting message loss. The post concludes by mentioning Confluent's transactional guarantees in Apache Kafka as a solution for developers.
  27. Build Streaming Data Pipelines Visually with Stream DesignerIntroduces Stream Designer, a visual interface for building streaming data pipelines on Kafka. Stream Designer translates pipeline logic into ksqlDB code, allowing users to drag and drop components for filters, joins, and aggregates. It integrates with Confluent's managed connectors and supports Single Message Transforms. Existing ksqlDB pipelines can be imported and edited visually. The tool provides an end-to-end view, real-time alerts, and RBAC for security. Pipelines can be exported as SQL code for sharing and CI/CD integration. A demo illustrates building a product recommendation pipeline by joining SQL Server CDC data with Kafka clickstream data and sending it to MongoDB Atlas.
  28. Serverless Stream Processing with Apache Kafka, Azure Functions, and ksqlDB | Confluent Serverless Stream Processing with Apache Kafka and AzureThis post details two methods for integrating ksqlDB with Azure Functions: using the Confluent Azure Function sink connector and using the Azure Functions Kafka extension. It explains the architecture and trade-offs of each approach, including how ksqlDB queries can trigger Azure Functions and how the Azure Functions Kafka extension can use Kafka topics as direct event sources. It also discusses the internal workings of the Azure Function sink connector, including record grouping, batching, and asynchronous POST requests, and briefly mentions implementing a .NET producer within Azure Functions for responding to events.
  29. Stream Processing vs Batch ProcessingThis post introduces the fundamental concepts of batch vs. stream processing, highlighting the benefits of stream processing for real-time insights and its application in various industries. It details how Apache Kafka enables stream processing through its duality of streams and tables, real-time data pipelines, and the availability of Kafka Streams and ksqlDB for building streaming applications. It also provides an overview of common use cases like streaming ETL, anomaly detection, predictive analytics, and real-time gaming.
  30. How Walmart Uses Apache Kafka for Real-Time Replenishment at Scale | Confluent How Walmart Uses Apache Kafka for Real-Time Omnichannel ReplenishmentThis post details Walmart's implementation of Apache Kafka for real-time omnichannel replenishment at scale. It outlines the challenges of managing inventory and replenishment across a vast global retail and e-commerce network, emphasizing the need for cycle-time reduction, accuracy, speed, reduced complexity, elasticity, scalability, and resiliency. The article describes their micro-batch architecture, where Kafka serves as the central nervous system, processing tens of billions of messages daily from millions of SKUs. Key architectural considerations include real-time input ingestion, multi-tenancy, diverse downstream consumer needs, and tight contracts between inputs and the replenishment engine. Platform decisions like active-passive data replication, automatic VM recovery, Kafka retries, and in-house audit/replay mechanisms are highlighted for resiliency. Specific tuning and optimizations are discussed, including a custom producer partitioning strategy using a murmur hash function, producer configurations like linger.ms, batch size, and acks='all', and consumer configurations such as max.poll.records, max.poll.interval.ms tuning, disabling auto-commit, session.timeout.ms, and heartbeat.interval.ms.
  31. How Instacart Used Confluent to Scale 10 Years of Growth in 6 WeeksInstacart migrated client streams from web and mobile clients to Kafka, leveraging Confluent Cloud for scalability and durability. This enabled real-time data processing for use cases like Customer 360, product catalog, and inventory management, supporting predictive models. The company is also developing a data mesh strategy to improve data discoverability and usability, treating data as products to enhance recommendations and fraud detection.
  32. Serverless Event Stream ProcessingThis post defines serverless stream processing, stateless vs. stateful processing, FaaS, and serverless. It details ksqlDB's capabilities for both stateless and stateful processing, and AWS Lambda's suitability for stateless tasks. It proposes a division of labor where ksqlDB handles stateful processing and Lambda handles stateless tasks, demonstrating a two-way communication scenario between ksqlDB and Lambda via Kafka topics. A GitHub repository with a full end-to-end example is provided, illustrating a ksqlDB application enriching stock trade data with user information and then triggering a Lambda function for further processing.
  33. Scaling Apache Druid for Real-Time Cloud Analytics at ConfluentThis post details Confluent's strategies for scaling Apache Druid for real-time cloud analytics. It introduces splitting data sources by cardinality, implementing data tiering with hot/default tiers and Kubernetes StatefulSets, and query laning with a 'slow-lane' configuration. The post also covers optimizing MiddleManager and Historical node instance types (e.g., c5.9xlarge, i3.2xlarge) for better resource utilization and cost-effectiveness, and discusses compaction strategies to reduce segment counts for improved query performance. It highlights the use of Druid for customer-facing monitoring, billing, and internal operations.
  34. Modern Messaging and Real-Time Data Integration with ConfluentThis post details how Confluent's platform, including Kafka Connect, Schema Registry, and ksqlDB, can be used to modernize legacy messaging architectures by providing standardized connectivity, metadata integration, and stream processing capabilities. It explains how to integrate with existing message queues (RabbitMQ, ActiveMQ, IBM MQ, TIBCO EMS) using Kafka Connectors, manage data schemas with Schema Registry, and perform real-time stream processing with ksqlDB. An example walkthrough is provided to demonstrate these capabilities.
  35. Real-Time Data Streams in the Cloud, On-Prem, or Both with ConfluentThis post provides a tutorial demonstrating how to build a real-time data pipeline using Confluent Cloud and Confluent Platform. It covers using the Datagen connector to simulate data, ksqlDB for stream joining and enrichment, and a MongoDB sink connector to persist the results. It also shows how to replicate data between Confluent Cloud and a self-managed Confluent Platform deployment using Confluent Replicator. The tutorial highlights the flexibility of Confluent for both managed and self-managed environments.
  36. Data Streams Are Nothing Without Actionable Insights Leading to Real-Life ActionsThis post details a real-life use case of transforming a stream of vehicle sensor data into actionable insights that trigger workflows in a Camunda BPMN engine. It highlights the challenges of duplicate event handling, time windowing for continuous insights, and semantic aggregation of events to correctly start, influence, or cancel workflow instances. A custom Java connector using Kafka and Camunda clients was developed to manage these complexities and maintain a history for stream replay and tuning.
  37. Using Kafka, ksqlDB, and Quarkus for Real-Time Sports TrackingThis post details the engineering of a real-time sports tracking application ('Töggelomat') using Kafka, ksqlDB, and Quarkus. It demonstrates event sourcing for player data, ksqlDB for managing player scores and point changes, and Quarkus for application logic. The application uses Kafka topics for game results, score calculations, and player updates, showcasing a practical implementation of event-driven principles for a real-time system.
  38. How Apache Kafka Enables Podium to "Ship It and See What Happens"This post details Podium's implementation of a real-time search index for conversations using Kafka, Kafka Connect, and ksqlDB. It covers using Debezium PostgreSQL CDC Connector to capture changes from Postgres tables (users, conversations, conversation_items), transforming and joining these streams with ksqlDB to calculate conversation 'step' status, and then using the Elasticsearch Sink Connector to index the data. It also explains the use of tombstone records to delete closed conversations from Elasticsearch and discusses considerations for restreaming and data versioning in Kafka pipelines.
  39. Powering Microservices with Event Streaming at SEIThis post details SEI Investments' multi-year journey of migrating from a monolithic database-oriented architecture to a containerized, event-driven architecture using Confluent Platform. It outlines their progression through 'days' of maturity: Day 1 (Proof of concept with CDC from Oracle GoldenGate into Kafka for auditing), Day 2 (Converting raw CDC events into business events by assembling related table operations based on SCNs and denormalizing them), and Day 3 (Integrating business events for new product offerings and delivering near-real-time event streams to clients using Confluent Schema Registry and Avro for governance and efficient serialization, and Confluent Replicator for delivery). The ongoing development focuses on containerization in Kubernetes, using Kafka for inter-microservice communication, and leveraging Confluent Schema Registry for payload governance and schema evolution.
  40. Spring Microservices into Production with Kubernetes and GitOpsThis post details the refactoring of a Kafka Streams based microservice into a Spring Boot application. It highlights the use of Spring Boot for standardized development tasks such as testing, configuration, and integrations with Apache Kafka. The post demonstrates how to leverage Spring for Kafka clients and Kafka Streams DSL, externalize configuration using properties files and profiles, manage dependencies with the Spring dependency management plugin, and expose REST services using declarative annotations. It also covers integration testing with embedded Kafka and deployment to Kubernetes using a GitOps approach with Kustomize.
  41. 4 ksqlDB Techniques That Make Stream Processing Easier Than EverThis post introduces four ksqlDB techniques that simplify stream processing: 1. Modeling and accessing JSON structures (homogeneous and heterogeneous) using SQL types and STRUCTs. 2. Extracting nested field values from escaped JSON strings stored as VARCHARs using EXTRACTJSONFIELD, with examples of handling different data types and casting. 3. Dynamically masking sensitive fields in real-time using the MASK function to create new streams with obfuscated data, while preserving the original unmasked data. 4. Updating active stream processing queries using CREATE OR REPLACE syntax, demonstrating how to modify masking logic for new events without affecting historical data.
  42. Apache Kafka and Spring Cloud Data Flow TutorialThis post introduces a tutorial on integrating Spring Cloud Data Flow with Confluent Cloud for building event-driven applications. It covers setting up Spring Cloud Data Flow locally using Docker Compose, deploying a simple 'ticktock' stream composed of 'time' and 'log' applications, and configuring it to use Kafka as the messaging broker. The tutorial highlights the use of Confluent Schema Registry and ksqlDB for seamless integrations and emphasizes the benefits of event streaming for real-time data processing.
  43. Digital Transformation in Style: How Boden Modernizes Retail Using KafkaBoden's digital transformation journey with event streaming and Apache Kafka is detailed. They migrated from a batch-oriented monolith to a microservices architecture using Confluent Cloud to address integration challenges and the need for near-real-time data. Initial projects included a retail stock service POC and a product domain service pilot, which exposed product information changes as events for downstream consumption and compatibility via Schema Registry. They also integrated ksqlDB with Snowflake for near-real-time data analytics, enabling analysis of customer journeys and buying preferences. Future plans involve establishing business event standards and architecture patterns for domain services, and building broker services to abstract legacy system complexities.
  44. Announcing Pull Queries in Confluent Cloud ksqlDBIntroduces pull queries as a preview feature in Confluent Cloud ksqlDB. Pull queries allow for traditional request-response lookups against materialized views, complementing the existing push-based persistent queries. The post details CLI configuration for authorization, API key creation, and provides an example of creating a stream, a materialized view, and issuing a pull query. It also outlines limitations during preview, including throughput rate limiting, eventual consistency, potential performance impact on push queries, and SLA exclusion.
  45. How to Build an Intrusion Detection System Using ksqlDBThis post demonstrates building an intrusion detection system (IDS) using ksqlDB and Kafka. It details the process of capturing network packets using TShark, ingesting them into Kafka via a SpoolDirSourceConnector, and then using ksqlDB to structure, flatten, and analyze the network traffic data. Specific ksqlDB streams are created to process raw packet data, flatten it into a more manageable format, and then identify potential port scan and Slowloris attacks by creating separate topics for alerts. The post provides setup instructions using Vagrant and Docker Compose, along with example ksqlDB queries and connector configurations.
  46. Real-Time Risk Management Using Apache Kafka and Event StreamingThis post details the migration of Tesco PLC's Risk Engine from a monolithic, in-memory JMS-based system to an event-driven architecture using Apache Kafka and Kafka Streams. It highlights the challenges of the previous version, including lack of persistence, inability to scale horizontally, data loss, and performance bottlenecks due to chained joins and large stateful aggregates. The new design introduces finer-grained Kafka topics (Handshakes, Identifications, Verification, Session results) and initially uses Kafka Streams for stream processing, later incorporating ksqlDB for specific tasks. The post also discusses the 'false start' with a Session Aggregator approach that led to memory issues and rebalancing problems, and the subsequent architectural improvement focusing on multiple independent and parallel pipelines with single logical responsibilities to address these issues.
  47. How Real-Time Materialized Views Work with ksqlDBThis post details how ksqlDB builds real-time materialized views using stateful stream processing. It explains the concept of incremental updates, the use of RocksDB for local storage and indexing, and the role of changelog topics for durability and recovery. The post also covers automatic repartitioning (shuffling) for GROUP BY operations and introduces the LATEST_BY_OFFSET aggregation for materializing the last value per key.
  48. How to Prioritize Messages in Apache KafkaThis post explains why Apache Kafka does not natively support message prioritization, contrasting its event streaming model with traditional messaging systems. It details how Kafka's commit log and partition-based storage prevent direct message reordering. The post then proposes the 'bucket priority pattern' as a workaround, where messages are produced into different topics (buckets) based on their priority, allowing consumers to process them in a prioritized order without complex sorting logic. This pattern aims to provide a 'good enough' solution for prioritization needs within Kafka's architecture.
  49. Stream, Analyze, and Visualize Wi-Fi Data with Kafka and FriendsThis post details a practical implementation of an event-driven architecture for processing Wi-Fi packet capture (pcap) data. It demonstrates a pipeline that ingests raw pcap data into Kafka, processes it using ksqlDB for aggregation and transformation (including creating materialized views and routing to PostgreSQL), and visualizes the data using Elasticsearch/Kibana and Neo4j. Specific technical contributions include: setting up Kafka Connect for Elasticsearch sink, using ksqlDB for stream creation, aggregation (tumbling windows), table materialization, and creating sink connectors to PostgreSQL; and exploring graph analysis with Neo4j.
  50. Leverage Real-Time Clickstream Data Using ksqlDB and ElasticsearchThis post demonstrates how to leverage ksqlDB, Kafka Connect, and Elasticsearch to build an event-driven application for analyzing clickstream data. It details specific ksqlDB statements for creating streams and tables to track user activity, errors, and user sessions using hopping and session windows. It also shows the configuration for the Kafka Connect Elasticsearch Sink Connector to export processed data for real-time analysis and visualization.
  51. SIEM with Anomaly Detection Using Machine Learning and ksqlDBThis post introduces the use of Latent Dirichlet Allocation (LDA) for behavioral anomaly detection in osquery logs. It details a Lambda Architecture with a batch pipeline for training the LDA model and a streaming pipeline that uses a ksqlDB UDF to score incoming logs in real-time. The scored events are then routed to 'GOOD', 'BAD', and 'UGLY' topics based on their anomaly score, enabling SIEM integration and model retraining. The post provides a practical demonstration with code snippets and setup instructions.
  52. How Netflix Uses Kafka for Distributed StreamingThis post details how Netflix uses Apache Kafka for its eventing, messaging, and stream processing needs within its microservices architecture. It explains the benefits of moving from synchronous HTTP requests to asynchronous event streams for improved decoupling, traceability, and availability. The post outlines three major sub-components: Producers (publishing state changes as events, including CDC), Enrichers (using stream processing engines like Flink and ksqlDB for data enrichment, windowing, and sessionization, with a focus on delayed materialization to ensure event ordering), and Consumers (implementing consuming microservices with Spring Boot, leveraging Schema Registry and Avro for schematized streams, and ensuring idempotent behavior for message delivery guarantees). It also touches upon monitoring infrastructure service levels with Atlas.
  53. Pipeline to the Cloud - On-Premises Data Streaming for Cloud AnalyticsThis post introduces a pattern for offloading data from on-premises OLTP databases to cloud analytics platforms (Snowflake, S3/Athena) using Apache Kafka and Kafka Connect. It details the benefits of this approach over traditional batch ETL, including workload isolation, optimization, data federation, and cost optimization. The post outlines a multi-stage implementation: 1) Streaming data from RDBMS (SQL Server) to Snowflake using Debezium CDC and Snowflake Kafka Connector. 2) Adding reference data from another RDBMS (MySQL) using a similar pattern. 3) Implementing stream processing with ksqlDB to enrich and denormalize data before loading it to Snowflake. The core technical contribution is the architectural pattern and the specific connectors and tools used to achieve real-time data streaming from transactional systems to cloud analytics platforms.
  54. Event Streaming at Deutsche Bahn with Apache Kafka & Confluent PlatformDeutsche Bahn developed the RI-Plattform system to provide consistent passenger information by processing 180 million events daily. They initially evaluated Kafka with Storm but found Storm complex to operate, leading them to adopt Kafka Streams. Kafka Streams improved their development lifecycle and reduced complexity, enabling a 90% reduction in schedule import processing time through local state stores. They collaborated with Confluent for optimized Kafka configuration on AWS, achieving 99.9% availability. The platform uses ~100 microservices communicating via Confluent Platform, with a CI/CD workflow deploying 4-8 times daily. They are exploring ksqlDB and Confluent Cloud for future enhancements.
  55. Build Streaming ETL Solutions with Apache Kafka & Rail DataThis post details building a streaming ETL solution using Apache Kafka and Confluent Platform for UK train data. It covers ingesting data from ActiveMQ, REST endpoints, and web pages using Kafka Connect and CLI producers. Data wrangling is performed using ksqlDB for schema application, Avro serialization, key derivation, data enrichment, and joining with reference data. The processed data is used for real-time analysis and visualization with Elasticsearch/Kibana, advanced analytics in PostgreSQL and S3, and an event-driven alerting service. The alerting mechanism involves joining train movement data with configuration data in ksqlDB to trigger notifications via Kafka Connect and Telegram.
  56. Building an Event-Driven Stock Platform at EuronextEuronext built a new market infrastructure and event-driven trading platform called Optiq using Confluent Platform. They leveraged Apache Kafka as a reliable, scalable persistence layer for market orders with millisecond latencies, replacing their market data gateway to handle billions of messages per day. Confluent Platform also enables applications for clearinghouse interfacing, market latency monitoring, disaster recovery replication, and regulatory-compliant data warehousing. The platform is now in production on all Euronext cash markets, resulting in a tenfold increase in message ingestion capacity and an average latency as low as 15 microseconds.
  57. Real-Time Analytics and Monitoring Dashboards with Kafka and RocksetThis post details how to integrate Apache Kafka with Rockset for real-time interactive dashboards and analytics. It explains Kafka's role as an event streaming platform, the use of Kafka Connect for ingesting data into Rockset, and Rockset's converged indexing for low-latency SQL queries. The post provides a step-by-step example of connecting Kafka to Rockset using the Rockset Kafka Connector, creating a collection from a Kafka data source, querying Twitter data, joining with other datasets (e.g., company information), and generating a real-time monitoring dashboard using Redash. It highlights the ability to perform interactive analytics on scalable, event streaming data without upfront ETL, enabling quick answers and acting while data is 'hot'.
  58. Introducing Derivative Event Sourcing for KafkaIntroduces the concept of Derivative Event Sourcing, a pattern for deriving events from observed sources (CDC, logs) rather than direct emission. Details a practical implementation using Kafka Streams to transform database changes (from Oracle GoldenGate and Debezium) into unified 'order created' events. Demonstrates filtering and mapping logic in Kafka Streams to handle different event profiles from multiple services writing to the same database. Emphasizes the benefits for integrating legacy systems and complex environments.
  59. Kafka Graph Processing: Visual Stream Analytics with Neo4jThis post introduces the integration of Neo4j, a graph database, with Apache Kafka for advanced stream analytics. It details how to use the Neo4j Streams plugin to ingest Kafka data (friend relationships) into Neo4j, transforming event streams into a property graph. It also demonstrates using Neo4j's graph algorithms, specifically Common Neighbors, to perform link prediction (suggesting potential friends) and send the results back to Kafka. The setup involves configuring Neo4j with Kafka bootstrap servers, security protocols, and a Cypher query for data ingestion. This extends event-driven architectures to incorporate graph-based analysis for relationship-centric use cases.
  60. Building Systems with Transactions Using Apache Kafka | ConfluentThis post details how to achieve ACID-like properties in Apache Kafka by modeling transactions as single, self-contained events. It explains how to ensure atomicity by treating events as the smallest unit of work, consistency by having event producers enforce data constraints, isolation by leveraging Kafka's partition-level ordering within consumer groups, and durability through producer acknowledgements and manual consumer commits. It also addresses API consistency by having producers return committed event data and discusses the importance of idempotent event processing to handle duplicate requests or events. The post highlights the role of Kafka Streams in managing state during rebalances to maintain transactional integrity.
  61. Microservices, Apache Kafka, and Domain-Driven Design | ConfluentThis post details how Apache Kafka serves as a backbone for microservice architectures, enabling decoupled communication through event streaming. It explains how Domain-Driven Design (DDD) principles, specifically bounded contexts and ubiquitous language, are applied to model business domains and how Kafka facilitates this by decoupling producers and consumers. The post highlights Kafka's ability to handle messaging, storage, and real-time processing, contrasting it with traditional middleware and emphasizing its role in building event-driven microservices using Kafka Streams, ksqlDB, and Kafka Connect.
  62. Streaming Data from the Universe with Apache Kafka | ConfluentThis post details the implementation of an event-driven architecture for astronomical transient object detection using Apache Kafka. It describes the data pipeline from image capture to alert distribution, including the use of Avro for serialization and Python clients for consumer interaction. The system handles millions of alerts per night, enabling near real-time scientific analysis. Key technical aspects include Kafka's consumer offset management for reliable delivery, Avro's efficiency for scientific data, and the design of downstream filtering algorithms for object classification. The architecture is presented as a prototype for future, larger-scale projects like LSST.
  63. Spring for Apache Kafka Deep Dive – Part 4: Continuous Delivery of Event Streaming Pipelines | ConfluentThis post details how Spring Cloud Data Flow can be used to build and continuously deploy event streaming pipelines. It explains common topologies like named destinations, parallel pipelines, partitioned streams, and function composition. It also demonstrates how to integrate Kafka topics as named destinations and how Spring Cloud Data Flow automatically creates Kafka topics for stream connections. The post provides examples of using the Spring Cloud Data Flow shell and Stream DSL for defining and deploying these pipelines, including a Kafka Streams application example.
  64. Spring for Apache Kafka Deep Dive – Part 3: Apache Kafka and Spring Cloud Data Flow | ConfluentThis post introduces Spring Cloud Data Flow as a toolkit for developing, deploying, and orchestrating event streaming pipelines based on Apache Kafka. It details the Spring Cloud Data Flow ecosystem, including its dashboard GUI, command-line shell, Java DSL, and RESTful APIs. The post explains how Spring Cloud Skipper is used for orchestrating deployments to platforms like Cloud Foundry and Kubernetes. It also covers the development of event streaming applications using Spring Cloud Stream, defining source, processor, and sink application types, and illustrates the creation of a linear event pipeline (http | transform | log) using the Stream DSL. The setup involves a Docker Compose configuration that includes Kafka, Spring Cloud Data Flow, Spring Cloud Skipper, Prometheus, and Grafana, and demonstrates how Kafka topics are automatically created for inter-application communication.
  65. Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation | ConfluentThis post details the implementation of an event-driven architecture for inventory management, integrating Oracle WMS Cloud with on-premises systems using Apache Kafka and Confluent Platform. It highlights the use of Oracle GoldenGate for relational data change events into Kafka and MuleSoft for producing payloads from other APIs to Kafka. The core contribution is the architectural pattern of using KSQL for stream transformations (joining, filtering, aggregating) and Kafka Streams for final message packaging to meet specific API contracts (e.g., nested messages for shipments/orders), leveraging Kafka Streams' built-in state store. It also discusses the challenges of integrating KSQL into DevOps processes and the need for Gradle plugins for managing KSQL implementations and UDFs.
  66. Putting Events in Their Place with Dynamic Routing | ConfluentThis post introduces KStream#branch() for static routing of events based on predicates, creating separate Kafka topics for different event subsets. It then details dynamic routing using KIP-303 and the TopicNameExtractor interface, allowing output topic names to be determined at runtime based on event data. The post also demonstrates how to dynamically route events by joining an order stream with a GlobalKTable of customer data (populated via JDBC Source Connector) to extract customer loyalty levels and route orders accordingly.
  67. SOA vs. EDA: Is Not Life Simply a Series of Events? | ConfluentThis post contrasts Service-Oriented Architecture (SOA) with Event-Driven Architecture (EDA), highlighting the shift from request-response patterns to event-based communication. It explains how EDA enables decoupled, asynchronous processing through event brokers and processors, offering advantages in resilience, scalability, and consistency compared to traditional SOA. The post emphasizes that EDA does not replace SOA but complements it, advocating for a hybrid approach to achieve optimal solutions.
  68. CloudBank's Journey from Mainframe to Streaming with Confluent Cloud | ConfluentThis post details CloudBank's migration from a mainframe-based core banking system to a cloud-native SaaS product (Genesis) using Apache Kafka and Confluent Cloud. It explains the architectural shift from traditional databases to a streaming-first approach to handle real-time data across multiple channels and diverse data types. The post highlights the challenges of self-managing Kafka in the cloud and the benefits of Confluent Cloud for simplifying operations, achieving multi-cloud support via Terraform, Docker, and Kubernetes, and enabling a single source of truth for data.
  69. Sysmon Security Event Processing in Real Time with KSQL and HELK | ConfluentThis post details the integration of Sysmon security event logs with Confluent Cloud and HELK for real-time processing. It demonstrates how KSQL can be used to join Sysmon Event ID 1 (ProcessCreate) and Event ID 3 (NetworkConnect) in real time to detect lateral movement techniques, such as a process spawning another process that makes an external network connection. The post highlights the advantages of event-driven alerting at the pipeline level over query-time SIEM analysis and explains the concepts of streams and tables in KSQL, along with different join types. It also provides a Sysmon data model based on ProcessGUID for correlating events and shows how to integrate Sysmon data into the HELK platform.
  70. Stream Processing Tutorial Part 1: Building Event Streaming ApplicationsThis post introduces a tutorial for developers on building real-time streaming applications using Kafka Streams. It details a microservices ecosystem for order management, demonstrating event persistence, event-driven service interaction, stream enrichment via joins (including CDC integration), filtering and branching, stateful operations with windowing and aggregation, state stores for querying, and enrichment with KSQL. The tutorial provides hands-on exercises to learn Kafka Streams API patterns.
  71. Noise Mapping with KSQL, a Raspberry Pi and a Software-Defined Radio | ConfluentThis post demonstrates how to ingest ADS-B signals from aircraft using a Raspberry Pi and a software-defined radio, relaying the data to Kafka topics via the Confluent REST Proxy. It then utilizes KSQL to process these streams, enriching location data with aircraft identification and flight details by joining with reference data topics. The processed data is then pushed to Elasticsearch via Kafka Connect for visualization in Kibana dashboards, enabling real-time noise mapping and aircraft identification.
  72. Event Driven 2.0 | ConfluentThis post introduces the concept of 'Event Driven 2.0' and outlines four key patterns for modern event-driven architectures: Global event streaming platform, Central event store, Event-first and event streaming applications, and Automated data provisioning. It emphasizes the unification of events, storage, and processing into a single platform, moving beyond traditional messaging systems and databases to enable data to be provisioned as a service. It highlights the use of Apache Kafka and ksqlDB as core technologies for these patterns.
  73. Data Wrangling with Apache Kafka and KSQL | ConfluentThis post introduces the use of Apache Kafka and KSQL for data wrangling tasks, specifically focusing on ingesting data from REST sources, applying schemas, filtering, masking, and transforming data structures. It demonstrates how to unify data from multiple sources into a single output topic and then stream it to Google Cloud Storage and BigQuery for analysis. The technical contribution lies in showcasing the practical application of Kafka Connect with a community REST source connector and KSQL for schema declaration and data transformation, addressing real-world data integration challenges with varying JSON structures.
  74. Putting the Power of Apache Kafka into the Hands of Data Scientists | ConfluentThis post details Stitch Fix's decision to build a self-service event delivery infrastructure using Apache Kafka, moving away from database snapshots. It highlights the challenges of manual administration in legacy systems and the benefits of a centralized log pipeline for data integration. The authors discuss their research into message brokers (Kinesis vs. Kafka) and data integration solutions (Beam, Storm, Logstash, Kafka Connect), ultimately choosing Kafka for its ecosystem, control, scalability, and cost-effectiveness. The post introduces the concept of 'The Data Highway' as a self-service platform built on Kafka to provide data scientists with easy access to company-wide event data.
  75. Apache Kafka vs. Enterprise Service Bus (ESB) | ConfluentThis post introduces Apache Kafka as a superior alternative to Enterprise Service Buses (ESBs) and ETL tools for enterprise integration. It details Kafka's event-driven architecture, emphasizing its scalability, flexibility, and ability to decouple applications. The post outlines the benefits of a streaming platform, including elastic scalability, architectural flexibility, event-driven microservices, openness, independent business services, multi-tenancy, and industrialized deployment. It also presents a streaming maturity model to guide enterprises in adopting event-driven platforms.
  76. We ❤️ syslogs: Real-time syslog processing with Apache Kafka and KSQL – Part 3: Enriching events with external dataThis post details the process of enriching real-time syslog data streams with external reference data from MongoDB. It demonstrates how to ingest syslog data using Kafka Connect, process it with KSQL, and then join it with data streamed from MongoDB using Debezium. The technical contributions include: 1. Ingesting Ubiquiti syslog data into Kafka. 2. Filtering and creating a KSQL stream for Ubiquiti-specific logs. 3. Streaming MongoDB device and user metadata into Kafka using Debezium. 4. Creating KSQL streams for the MongoDB data and transforming it to key on IP addresses. 5. Joining the Ubiquiti syslog stream with the device metadata stream in KSQL to enrich the syslog events with device names and IPs. 6. Persisting the enriched data to a new Kafka topic for downstream processing (e.g., Elasticsearch, Kibana, push notifications).
  77. Introducing Self-Service Apache Kafka for DevelopersThis post announces the launch of self-service Confluent Cloud and its availability on Google Cloud Platform (GCP). It addresses the developer challenge of self-managing Kafka infrastructure by providing a fully managed, low-cost, consumption-based Kafka service. This enables developers to focus on building applications rather than operating the platform, and allows integration with cloud-native services like BigQuery and TensorFlow on GCP. The post highlights self-service provisioning, elastic scaling, global availability, open-source Kafka API compatibility, and cloud independence as key benefits for cloud-first, event-streaming developers.
  78. We ❤️ syslogs: Real-time syslog Processing with Apache Kafka and KSQL—Part 2: Event-Driven Alerting with SlackThis post details the implementation of an event-driven alerting system using Apache Kafka, KSQL, and the Confluent Kafka Python client. It demonstrates how to subscribe to a KSQL-derived Kafka topic containing filtered syslog data (specifically 'Invalid user' login attempts) and use the Confluent Kafka Python client to send push notifications to Slack via its API. The post provides code examples for installing the Slack client, testing API keys, installing the Confluent Kafka Python client, consuming messages from a Kafka topic, and combining Kafka consumption with Slack notifications to alert on suspicious login activities.
  79. We ❤️ syslogs: Real-time syslog Processing with Apache Kafka and KSQL—Part 1: FilteringThis post introduces the real-time processing of syslog data using Apache Kafka and KSQL. It details how to ingest syslog data into Kafka via Kafka Connect and then use KSQL to filter these messages. The post provides a practical example of creating a KSQL stream from the 'syslog' Kafka topic and demonstrates inspecting the data. The core contribution is the initial step in building an event-driven architecture for log processing, specifically focusing on the filtering aspect.
  80. Event Sourcing Using Apache Kafka | ConfluentThis post introduces event sourcing as a pattern that can be implemented using Apache Kafka. It outlines three strategies for storing events in Kafka (single topic, topic-per-entity-type, topic-per-entity) and discusses the trade-offs. It details how Kafka Streams and its state stores can be used to efficiently manage and query the current state of entities derived from event streams. The post also covers interactive queries for cross-node state access, changelog topics and standby replicas for fault tolerance, and the use of Kafka's transactional capabilities for exactly-once processing guarantees.
  81. Apache Mesos and Kafka Streams for Highly Scalable MicroservicesThis post details how to build highly scalable microservice infrastructure using Apache Kafka, Kafka Streams, and Apache Mesos (via Mesosphere's DC/OS). It explains the benefits of running Kafka and Confluent components on DC/OS, including automated provisioning, unified management, and elastic scaling. It also describes how to deploy and scale Kafka Streams microservices on DC/OS using Marathon, highlighting automatic load balancing and fault tolerance. The post also provides guidance on using newer Kafka clients and offers resources for setting up DC/OS and deploying Kafka Streams applications.
  82. Toward a Functional Programming Analogy for MicroservicesIntroduces a functional programming analogy for microservices, contrasting it with the synchronous, object-oriented analogy. Advocates for an event-driven architecture based on immutable logs (Kafka) and reactive dataflows (Kafka Streams). Details CQRS, Event Sourcing, and the single writer principle in this context. Categorizes microservices into 'Functional Core' and 'Imperative Shell' (materialized views and side-effect executors). Provides a banking use-case example using Kafka Streams.
  83. Building a Microservices Ecosystem with Kafka Streams and KSQLThis post introduces building microservices ecosystems using Kafka Streams and KSQL. It details how to leverage Kafka Streams for stateful stream processing, including materializing local data for snappy UI responses and managing state with standby replicas and disk checkpoints. It also explains how KSQL can be used for stateful stream processing in separate processes, enabling stateless application logic. The post provides an example of a simple email service built with Kafka Streams and discusses the single-writer principle for managing consistency in distributed systems.
  84. Ranking Websites in Real-time with Apache Kafka’s Streams APIThis post details how Zalando implemented a real-time website ranking system using Apache Kafka's Streams API and a modified HITS algorithm. It explains the HITS algorithm's concepts of Hubs and Authorities, the iterative matrix calculation, and the rationale for choosing Kafka Streams over MapReduce/Hadoop/Spark due to team expertise and infrastructure. The architecture involves three components: Domain Link Extractor, Domain Reducer, and HITS Calculator, deployed in containers on AWS. The Domain Link Extractor uses KStream's flatMap to process s3 URLs of web documents, extract links, and output domain-to-domain mappings. The Domain Reducer aggregates these mappings into a KTable using groupByKey and reduce. The HITS Calculator then uses this KTable to perform the iterative HITS calculation and expose ranked Hubs and Authorities via a REST API. The post highlights the flexibility of Kafka Streams, demonstrating how intermediate outputs can be captured for further processing without disrupting the main calculation.
  85. Publishing with Apache Kafka at The New York Times | ConfluentThis post introduces The New York Times' 'Publishing Pipeline' architecture, which uses Apache Kafka as a central log ('The Monolog') for all published content. It contrasts this with a previous API-based approach that suffered from schema inconsistencies and difficulty accessing historical data. The log-based architecture allows for materialized views of the data, enabling systems like search engines and personalization services to consume content efficiently. It highlights Kafka's suitability for retaining all events forever and ordered consumption, which are crucial for recreating data stores and enabling immutable deployments of stateful systems. The post also explains why services like Google Pub/Sub and AWS Kinesis are not suitable for this specific log-based use case due to their different design goals.
  86. Leveraging the Power of a Database 'Unbundled' | ConfluentThis post introduces the concept of an 'unbundled database' where Kafka acts as a durable log and stream processing (like Kafka Streams) enables materialized views within services. It contrasts this with traditional shared databases, highlighting the benefits of loose coupling, independent service deployability, and ephemeral, service-specific views. The post details how this architecture avoids the pitfalls of shared mutable state and amplifying interfaces, enabling services to be more nimble and agile by composing data-enabled components.
  87. Messaging as the Single Source of Truth | ConfluentIntroduces the concept of events as a "single source of truth" by leveraging event sourcing and Apache Kafka as a durable event store. Details how retained events act as a version control system for data, enabling recovery and auditing. Explores patterns for getting events into Kafka, including event sourcing and Change Data Capture (CDC) from databases, highlighting Kafka's suitability for long-term event storage with retention-based and compacted topics.
  88. Chain Services with Exactly-Once Guarantees | ConfluentThis post introduces Kafka's exactly-once semantics (EOS) as a solution to the problem of duplicate message processing in distributed systems. It explains how EOS is achieved through producer idempotence (preventing duplicates in the log) and transactional APIs (atomically committing reads and writes). The article details the underlying mechanisms of Kafka transactions, including marker messages, transaction coordinators, and two-phase commit, and discusses the performance implications. It also touches upon using transactions to tie stream processing and state persistence together atomically.
  89. Apache Kafka as an Event-Driven Backbone for Service ArchitecturesThis post introduces Apache Kafka as a unique message backbone for service architectures, contrasting it with traditional messaging systems. It details how Kafka's log-structured approach, sequential reads/writes, and direct data copying from disk buffer to network buffer contribute to high performance and efficient message retention. The post highlights Kafka's linear scalability, enabling it to handle massive message volumes and multi-tenancy. It also discusses features for segregating load in multi-service ecosystems through throughput control (quotas), maintaining strong ordering guarantees via partitioning and producer configurations, ensuring message durability through replication, and enabling highly available, load-balanced services through partition re-routing.
  90. Build Services on a Backbone of Events | Confluent for MicroservicesThis post introduces the core concepts of event-driven architectures, contrasting them with traditional request-driven microservices. It defines commands, events, and queries, and illustrates how event-driven patterns (like receiver-driven routing and 'Query by Event-Carried State Transfer') offer better decoupling, autonomy, and efficiency compared to synchronous request-response models. The 'Single Writer Principle' is also introduced as a strategy for managing event propagation consistency.
  91. Microservices: Rethinking the Way We Treat Data and Services | ConfluentThis post introduces the concept of 'data-on-the-outside' as a solution to the 'data dichotomy' in microservices, where services need to hide internal state (encapsulation) but also require access to shared data. It argues that traditional approaches like service interfaces, messaging (without historical reference), and shared databases are inadequate. The post proposes using a distributed log (like Kafka) as a streaming platform, enabling event-driven architectures where services can join and operate on shared streams with embedded stream processing. This allows for greater autonomy and scalability by pushing flow control to the receiver and leveraging the log for both transport and long-term storage.
  92. Confluent & Syncsort: An Architecture for Streaming and At-Rest DataThis post introduces a partnership between Confluent and Syncsort to combine streaming data from Kafka with at-rest data from traditional databases and file systems. It highlights use cases like fraud detection, real-time reporting for hotels, and healthcare analytics, emphasizing how Kafka acts as the central nervous system for data flow and Syncsort provides the context from historical data. The architecture enables real-time decision-making by integrating batch and streaming data.
  93. Apache Kafka for Real-Time Retail at Walmart Labs | ConfluentThis post details Walmart's adoption of Apache Kafka for their 'Item Setup' process, which involves managing products, offers, pricing, inventory, and logistics. It describes how Kafka was used as a central data hub to connect hundreds of microservices and teams, modeling changes as event streams. The implementation leverages stream processing technologies like Apache Storm and Spark, plain Java programs, reactive microservices, and Akka Streams. The post highlights the scale of operations, processing billions of updates daily, and the challenges encountered, such as managing service topologies, schema management, multi-DC data balancing, and shielding consumers from business-impacting changes. It also mentions the development of operational tools for tracking flows, SLA metrics, and latency, and the potential for reprocessing catalog data, sharing data between environments, A/B testing, and analytics.
  94. Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection? | ConfluentThis post introduces Event Sourcing and CQRS as application architecture patterns and explains how Apache Kafka and Kafka Streams can be used to implement them. It details how Kafka serves as a durable log for immutable events and how Kafka Streams can act as an event handler to transform events and update read stores. The post also explores two approaches for modeling application state: using an external datastore or leveraging Kafka Streams' local, partitioned, and durable state, which is exposed via KTables. It highlights the benefits of Kafka Streams for load balancing, failover, and elastic scaling in CQRS-based applications.
  95. Building a Streaming Analytics Stack with Apache Kafka and Druid | ConfluentThis post introduces a streaming analytics stack by combining Apache Kafka for high-throughput event delivery with Druid for real-time analytical queries and visualization. It details the setup of both Kafka and Druid, loading Wikipedia edits data, and visualizing it using Pivot. The architecture leverages Kafka as a central repository for buffering events before they are consumed by Druid's real-time workers, enabling event replayability and delivery to other systems.
  96. Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Apache Kafka | ConfluentThis post explains the role of distributed consensus services like Apache ZooKeeper in distributed systems, focusing on how Apache Kafka uses ZooKeeper for metadata management, replica set coordination, and leader election. It clarifies that ZooKeeper provides a 'coordination kernel' rather than an explicit consensus API, and that its atomic broadcast capabilities are fundamental to implementing replicated state machines, which Kafka uses for data replication. The post also highlights potential pitfalls in reasoning about agreement in distributed systems and provides a concrete example of how Kafka leverages ZooKeeper's sequential nodes to break ties and achieve agreement on values.
  97. Apache Kafka, Samza, and the Unix Philosophy of Distributed Data | ConfluentThis post draws parallels between the Unix philosophy of small, composable tools and the design principles of Apache Kafka and Samza. It argues that by adhering to the Unix philosophy (each program does one thing well, and expects output to be input for another unknown program), distributed data systems can achieve greater flexibility, extensibility, and rapid prototyping, similar to how Unix pipes enable complex data processing pipelines. The core technical takeaway is the architectural benefit of loosely coupled, single-purpose components in building scalable systems.
  98. The Value of Apache Kafka in Big Data Ecosystem | ConfluentThis post introduces Apache Kafka as a central component in the Big Data ecosystem, addressing the limitations of monolithic databases and the emergence of specialized distributed systems. It highlights Kafka's role in feeding these specialized systems by acting as a scalable, multi-subscription pub/sub system that persists data to disk, enabling both real-time and batch consumption. The post positions Kafka as a stream data platform that facilitates the integration of diverse data types into various specialized systems like key/value stores, search engines, and stream processors.
  99. Using logs to build a solid data infrastructure (or: why dual writes are a bad idea) | ConfluentThis post introduces the fundamental concept of logs as a core component of robust data infrastructure, explaining how they are used in databases, replication, distributed algorithms, and stream processing systems like Kafka. It highlights the challenges of data integration in complex systems with multiple storage services and argues against dual writes due to race conditions and inconsistencies. The post advocates for using logs as a single source of truth to simplify data synchronization and build more reliable and scalable applications.
  100. Real-time full-text search with Luwak and Samza | ConfluentThis post describes the combination of Luwak and Apache Samza for building a scalable streaming search engine. Luwak, a Lucene-based library, efficiently handles thousands of queries over a single document, while Samza, a stream processing framework based on Kafka, distributes real-time computations across a cluster. The approach involves registering queries in advance and matching them against incoming documents, optimizing the process by indexing queries themselves to quickly identify non-matching queries. This enables real-time full-text search on streams, exemplified by use cases like Twitter's firehose search and Google Alerts.
  101. Turning the database inside-out with Apache Samza | ConfluentThis post introduces Apache Samza as a distributed stream processing framework that offers an alternative to traditional mutable database architectures. It argues for treating data as immutable facts and processing them as streams, drawing parallels to event sourcing. The core idea is to 'turn the database inside out' by leveraging a durable commit log (like Kafka) and stream processing tools to build simpler, more scalable, robust, and lower-latency applications.
  102. Announcing the Confluent Platform 1.0 | ConfluentThis post announces the general availability of Confluent Platform 1.0, which is a stream data platform powered by Apache Kafka. It highlights the core components: Apache Kafka (0.8.2.0 with critical patches), a schema management layer, improved Java and REST clients, and Camus for Kafka to Hadoop ETL. It positions the platform as enabling high-throughput, scalable, reliable, and low-latency stream data management.
  103. Announcing Confluent, a Company for Apache Kafka and Realtime Data | ConfluentThis foundational post announces the formation of Confluent, a company dedicated to Apache Kafka. It details the origin of Kafka at LinkedIn, driven by the need for scalable, real-time data integration and stream processing to overcome limitations of traditional batch ETL and siloed data systems. The post highlights Kafka's design principles (durability, persistence, scalability) and its evolution from an ETL tool to a central nervous system for companies, handling business activity, operational data, and system metrics as real-time streams. It emphasizes the open-source release of Kafka and the motivation for creating Confluent to provide the surrounding capabilities needed for pervasive real-time data.