Stream Processing Fundamentals

5/18/2022

This post defines stream processing and contrasts it with batch processing, explaining the core characteristics of stream processing: continuous processing, low latency, and support for event-time processing. It details common use cases for stream processing across various industries and explains how Apache Kafka facilitates stream processing through its stream-table duality and real-time data pipelines. It also introduces Kafka Streams and ksqlDB as methods for implementing stream processing solutions.

12/9/2021

This post defines serverless stream processing, stateless vs. stateful processing, FaaS, and serverless. It details how ksqlDB handles both stateless and stateful processing, and how AWS Lambda functions as a FaaS. A key contribution is the proposed pattern of combining ksqlDB with AWS Lambda for event-driven applications, where ksqlDB performs stateful stream processing and Lambda handles stateless tasks triggered by ksqlDB outputs. A concrete example is provided with a GitHub repository demonstrating a ksqlDB application integrated with AWS Lambda for two-way communication, processing stock trades and user data.

9/27/2021

This post details the integration of AWS Lambda with Confluent Cloud, enabling serverless data streaming. It specifically focuses on ingesting data from Kafka topics into Amazon DynamoDB tables by triggering an AWS Lambda function using the newly supported SASL/PLAIN authentication. The post provides a step-by-step guide for setting up Confluent Cloud, creating a cluster and topic, and launching AWS CloudFormation to provision the necessary AWS resources, including the Lambda function and DynamoDB table. It highlights the benefits of a serverless architecture for data-in-motion use cases.

2/19/2021

This post introduces significant enhancements to ksqlDB 0.15, primarily focusing on message key capabilities. Key contributions include: 1. Support for all serialization formats (JSON, AVRO, PROTOBUF, DELIMITED, KAFKA, JSON_SR) for message keys. 2. Expanded data type support for keys, including ARRAY, STRUCT, and nested combinations, with specific considerations for MAP types and single-field wrapping. 3. Introduction of composite keys, allowing multiple columns to form a single key. 4. Enhanced PARTITION BY and GROUP BY clauses to accept multiple expressions, leading to tables and streams with multiple key or primary key columns respectively. This directly impacts how data is structured, partitioned, and aggregated within ksqlDB.

1/29/2021

This post details the engineering efforts behind building a real-time table football score tracking application ('Töggelomat') using Apache Kafka, ksqlDB, and Quarkus. It covers event sourcing for player data, using ksqlDB to initialize player scores and process match results, calculating player score changes based on match outcomes, and applying these changes back to player scores. The post highlights the use of Quarkus for its reactive nature and low footprint in event-driven applications, and ksqlDB's ability to treat event streams like a database.

12/16/2020

Introduced multiple key lookups in pull queries using the IN predicate. Added ALTER syntax for incremental schema modification of streams and tables created with DDL. Implemented SQL variable substitution for dynamic query execution. Added support for DELIMITED and JSON key formats, with plans for Schema Registry integrated key formats and non-primitive key types in the future.

12/14/2020

This post details four ksqlDB techniques that simplify stream processing: modeling and accessing JSON structures (including heterogeneous and nested JSON), extracting nested fields from JSON strings stored as escaped strings using EXTRACTJSONFIELD, dynamically masking sensitive fields in real-time using the MASK function and creating new streams with masked data, and updating active stream processing queries with CREATE OR REPLACE syntax to modify masking logic for new events.

11/20/2020

This post details building data pipelines by transforming raw CSV data from car park occupancy into a structured format using ksqlDB, applying schemas, setting message keys for partitioning, adding source fields for lineage, deriving and storing timestamps, creating nested location structs, and calculating new fields. The processed data is then streamed to Elasticsearch for analytics and visualization with Kibana.

11/19/2020

This post explains the distributed architecture of ksqlDB, detailing how it scales workloads by distributing Kafka topic partitions across multiple ksqlDB servers. It elaborates on the use of Kafka consumer groups for partition assignment, load balancing, and fault tolerance. The post also differentiates between stateless and stateful operations, explaining their respective recovery mechanisms, and discusses how to optimize cluster sizing and thread parallelism for performance.

11/5/2020

This post introduces pull queries as a preview feature in Confluent Cloud ksqlDB. It explains how pull queries differ from persistent queries by following a request-response model to retrieve the latest result from a materialized view. The post provides detailed CLI configuration steps for authorizing clients, including obtaining application endpoints, IDs, and Kafka cluster IDs, creating API keys, and setting up ACLs. It also includes a tutorial for creating a materialized view and issuing a pull query. Limitations during the preview phase are outlined, including throughput rate limits, eventual consistency, potential performance impact on push queries, and the absence of an SLA. The post concludes by highlighting the flexibility offered by combining push and pull patterns for real-time applications and notes that pull queries will be free during preview.

10/22/2020

This post details how ksqlDB manages and limits memory usage, particularly focusing on off-heap allocations by RocksDB. It explains the configuration of RocksDB with a shared cache, shared write buffer manager, and index/bloom filter caching to bound memory. The post investigates a real-world scenario where memory usage grew unbounded due to glibc's malloc implementation retaining memory in arenas after query termination and restart, leading to memory leaks. The solution involved using alternative allocators like jemalloc or TCMalloc, or in Confluent Cloud, implementing a fix to ensure proper memory release.

10/13/2020

This post details how ksqlDB builds real-time materialized views using stateful stream processing, leveraging RocksDB for local storage and Kafka changelog topics for durability and recovery. It explains automatic repartitioning for GROUP BY operations and the role of changelog compaction. The post also introduces the LATEST_BY_OFFSET aggregation for retaining the last value per key.

10/2/2020

This post introduces the ksqlDB Java client, enabling programmatic interaction with ksqlDB servers. It demonstrates its use in an IoT application for dynamically managing sensor types, creating streams and materialized views using LATEST_BY_OFFSET, and performing administrative operations. The post also showcases asynchronous programming with CompletableFutures for non-blocking operations.

9/30/2020

This post introduces the ability to upgrade persistent ksqlDB queries in-place using `CREATE OR REPLACE` statements for both WHERE clause expressions and schema changes. It also adds automatic restarts for persistent queries that enter an ERROR state, improving resilience and reducing manual intervention.

9/29/2020

This post details how ksqlDB works by explaining its core concepts: declaring streams with schemas, inserting rows as records into Kafka topics, transforming streams with persistent queries that compile to Kafka Streams topologies, filtering rows, combining operations for efficiency, and rekeying streams for data locality using partitioning. It also touches on how ksqlDB manages processing progression and offsets, and how it handles out-of-order data.

8/12/2020

This post details a bug in ksqlDB's state store recovery mechanism where an optimization for avoiding changelog topics led to incorrect schema registration with Schema Registry during node relocation. The bug (KAFKA-10179) caused deserialization errors when ksqlDB attempted to read from the state store using the wrong schema ID and subject. A temporary fix involves registering the schema ID under the phantom changelog subject. The future roadmap includes improving ksqlDB's integration with Schema Registry to handle custom internal schemas more robustly.

8/4/2020

ksqlDB 0.11.0 introduces HTTP client caching for inter-node requests to improve pull query performance in multi-node clusters. It also fixes a bug where old windows were not properly expired from the underlying Kafka topic for windowed tables, and ensures that retention clauses for windowed aggregations now impact the underlying Kafka topic. New Java client methods are available for creating and managing streams, tables, and persistent queries (via executeStatement()), and for listing streams, tables, topics, and queries (listStreams(), listTables(), listTopics(), listQueries()). The WINDOWEND keyword is now supported in pull query filters.

7/31/2020

Introduces significant changes to ksqlDB's key handling in version 0.10, including the removal of implicit ROWKEY columns and the WITH(key='ID') syntax. Users can now explicitly name key columns for both tables (PRIMARY KEY) and streams (KEY), directly mapping to Kafka record keys without requiring data duplication in the value. This simplifies queries and improves performance by avoiding unnecessary repartitions.

7/27/2020

This post details how PushOwl uses ksqlDB to scale analytics and reporting use cases by migrating from a traditional PostgreSQL database approach to an event streaming platform. It covers the challenges of slow query execution and database scaling due to high volumes of push notification data. The solution involves publishing dispatch and ping events to Kafka, aggregating these events using ksqlDB for real-time reporting, and sinking the aggregated data back to PostgreSQL for dashboard access. The post also details the use of Kafka Connect JDBC sink and S3 sink connectors, and provides Python code examples for producing Avro-formatted messages to Kafka.

7/23/2020

This post introduces the concept and implementation of User-Defined Table Functions (UDTFs) in ksqlDB, specifically demonstrating their use for infrastructure monitoring. It details the process of creating a Java UDTF (`SplitPanel`) to transform raw panel data into individual breaker readings, enriching them with customer information. The post outlines the necessary ksqlDB schema definitions, the Java code for the UDTF, and the deployment process for custom functions.

7/20/2020

This post details the monitoring and alerting strategy for Confluent Cloud ksqlDB, focusing on ensuring prompt response to failures, proactive mitigation of potential issues, and confidence in the system's reliability. It covers health checks, specific alerting for system vs. user errors, proactive monitoring of resource utilization with trajectory analysis, and redundancy in the monitoring pipeline to prevent alert fatigue and ensure the monitoring system itself is robust.

6/26/2020

This post introduces the ksqlDB 0.10.0 release, highlighting the addition of a first-class Java client that supports pull and push queries, and row insertion. It also details enhancements to Kafka key support, allowing any column name for keys and removing the `WITH KEY` syntax. The release mandates explicit primary keys for tables and creates keyless streams when none are declared. New built-in functions for string manipulation, regex, array, and map operations are also introduced.

6/25/2020

This post introduces ksqlDB as an event streaming database that unifies stream and state processing with a single SQL query language. It highlights its integration with Kafka connectors and Schema Registry, and its availability as a managed service in Confluent Cloud. The post details how ksqlDB enables the creation of data pipelines (streaming ETL) and asynchronously materialized views, supporting various real-time application use cases.

6/3/2020

This post details the implementation of Kafka transactions to ensure consistent metastore recovery for ksqlDB in interactive mode. It addresses the previous design's flaws where the command topic acted as both a request queue and a log, leading to statement validation and execution errors. The new protocol treats the command topic strictly as a log by performing validation before enqueueing and isolating servers using Kafka's transactional.id. This ensures that only one server can validate and produce statements to the command topic at a time, preventing race conditions and guaranteeing deterministic metastore recreation.

5/18/2020

Introduces KIP-535 to allow Kafka Streams state stores to serve stale reads from standby partitions during rebalances or VM failures, improving availability. Details KIP-562 for querying specific partitions and optimizing lookups by avoiding iteration over all stores on an instance. Fixes KAFKA-9169 to prevent unnecessary restoration of standby tasks, reducing resource consumption and potential errors.

5/13/2020

Introduced support for multi-join statements in ksqlDB, enabling the combination of multiple JOIN operations into a single query for improved conciseness and efficiency. Enhanced LIKE expressions to support the '%' wildcard at any position and introduced the '_' wildcard for single character matching. Added the ESCAPE clause for literal interpretation of special characters in LIKE conditions. Added the COALESCE built-in function.

5/5/2020

This post details the engineering efforts to make pull queries remain highly available in the face of server failures by implementing faster failure detection via heartbeating and enabling lag-aware routing to standby replicas, allowing ksqlDB to operate as an AP system under certain conditions. It explains the trade-offs between CP and AP systems using the CAP and PACELC theorems, and describes the implementation of a ksqlDB server mesh for heartbeating and the use of KIP-535 for standby routing and lag-aware routing.

5/1/2020

This post introduces the concept of ksqlDB execution plans as a mechanism to manage query compatibility during upgrades and feature rollouts. It details how ksqlDB compiles queries into an internal, JSON-serializable execution plan (a DAG of steps) instead of directly building a Kafka Streams topology. This allows for independent evolution of the ksqlDB parsing and planning layer from the Kafka Streams topology generation layer, ensuring that running queries are not broken by upgrades. The post explains the motivation for this approach, the limitations of previous compatibility solutions, and the testing strategies (schema verification and QTTs) employed to maintain the integrity of the execution plan API.

4/28/2020

This post details how Broadcom modernized its Machine Learning & Anomaly Detection system (Mainframe Operational Intelligence - MOI) by integrating ksqlDB. It describes the architecture of MOI, which uses Kafka for inter-service communication and Schema Registry for schema management. The post outlines a prototype where ksqlDB is inserted between the Machine Learning Service and the Alarm Lifecycle Management Service to filter alert flags based on metric values and instance information. It covers the installation of ksqlDB server, creating streams with custom timestamps, selecting specific metric instances, and defining filter rules using INNER JOIN WITHIN for time-bound correlation. A key technical challenge addressed is the schema transformation required for ksqlDB to output data compatible with downstream services, including handling casing differences and Avro map serialization, which was resolved by retrieving and regenerating POJOs from Schema Registry.

4/14/2020

This post details how ksqlDB can be used by small and medium-sized businesses to build event streaming pipelines. It presents a use case for an online pharmacy to gain business intelligence by processing transactional data. The post explains how ksqlDB integrates with Kafka Connect and Debezium for CDC, and demonstrates the creation of streams and tables to analyze the impact of discounting on average order value. It highlights ksqlDB's ability to handle bespoke complexity and create new data.

4/6/2020

This post announces the general availability of Confluent Cloud KSQL, a fully managed stream processing service. It highlights the core capabilities of KSQL, such as transformations, stream-to-stream joins, aggregations, and data enrichment using SQL syntax. The post also introduces the intuitive web interface for managing KSQL applications, a 'Flow' view for visualizing processing topologies, and programmatic access via REST APIs. It emphasizes the seamless integration with Kafka and Schema Registry, and the elimination of operational overhead for users.

3/31/2020

This post details how Bolt uses ksqlDB for data replication to their data warehouse. It covers the requirements for a data replication pipeline (consistency, resilience, scalability, data manipulation, denormalization, integration) and explains why ksqlDB was chosen over other solutions. The post provides concrete ksqlDB SQL examples for creating streams from topics, selecting and filtering data, and merging streams. It also elaborates on ksqlDB's advantages: native Kafka integration, ease of deployment and management, linear scalability, Schema Registry integration, declarative query language, extensibility with UDFs, and ease of stream management. The post concludes by stating that ksqlDB has been successfully used to replicate over 500 topics.

3/26/2020

This post introduces ksqlDB as a bridge between real-time data and big data streaming, contrasting state-oriented and event-oriented architectures. It details how ksqlDB, as an event streaming database, allows for both immutable event logging and real-time querying of materialized views using SQL. It highlights the ability to perform both pull queries (to get the current state) and push queries (using EMIT CHANGES to continuously receive state updates) on event streams, demonstrating its capability to simplify architectures that require both event streaming and batch processing elements.

3/20/2020

This post introduces tunable retention and grace period for windowed aggregates in ksqlDB, allowing users to configure how long windowed results are stored and how late-arriving events are processed. It also adds the LATEST_BY_OFFSET aggregation function to track the latest value of a column based on offset order. Additionally, it provides an early look at the new client and server API (KLIP-15) and includes confluent-hub in ksqlDB Docker images for simplified connector installation.

2/21/2020

This post details how a proof of concept was built using Kafka Streams and Confluent Platform to ingest data from a REST API (flight systems) and process it into a near-real-time dashboard. It covers the architecture, including a Kafka producer for ingesting data, Kafka Streams for transformations and aggregations (e.g., flight lists, top airlines, fastest flights), and WebSockets for streaming processed data to a browser-based Progressive Web App (PWA) built with Vue.js and TypeScript. The post highlights challenges with the REST API's call limits and the decision to use Kafka Streams for its ability to handle backpressure and perform windowed aggregations. It also touches on data refinement, system resiliency, data update, and optimizing machine resources.

2/14/2020

Introduced highly available state for ksqlDB queries by enabling routing to standby servers during cluster rebalances, with configurable staleness tolerance (ksql.query.pull.max.allowed.offset.lag). Added count_distinct and cube functions. Expanded support for record keys to include INT, BIGINT, and DOUBLE. Added first-class constructors for native arrays, maps, and structs.

2/13/2020

This post introduces the integration of Latent Dirichlet Allocation (LDA) machine learning models into ksqlDB via User-Defined Functions (UDFs) for real-time anomaly detection in streaming data. It details a Lambda Architecture approach for training LDA models in batch and scoring incoming osquery logs in real-time. The post explains how ksqlDB queries are used to route scored events into 'GOOD', 'BAD', and 'UGLY' topics based on anomaly scores, enabling SIEM integration and feedback loops for model retraining.

2/6/2020

This post introduces the concept of materialized caches and demonstrates how to build them using ksqlDB. It explains how to model change data capture as a ksqlDB stream, define a query to aggregate events within a specific geographic area and time window, and then persist this query as a materialized view (table) for efficient, low-latency querying. It also covers setting up ksqlDB with Docker Compose, interactive CLI usage, and accessing the materialized cache via ksqlDB's REST API. The post highlights the benefits of this approach for offloading read pressure from traditional databases and enabling incremental updates to query results.

1/16/2020

This post dives into the architectural underpinnings of elasticity and fault tolerance in stream processing with Kafka Streams and ksqlDB. It explains how the stream-table duality, specifically the use of changelog topics, ensures fault tolerance for tables by providing a durable source of truth. It details how this mechanism also enables elastic scaling by allowing state to be restored on new instances during rebalancing. The post further elaborates on the benefits of topic compaction for reducing storage footprint and recovery time, and introduces the concept of standby replicas to minimize recovery time during failures and scale-in events.

1/15/2020

This post details the fundamental concepts of event processing in Apache Kafka, focusing on streams and tables as processing layer concepts distinct from Kafka's storage layer (topics, partitions, brokers). It explains how events in Kafka topics are transformed into streams (unbounded sequences with schemas) and tables (aggregated streams, often materialized views). The post also covers data contracts and schema management, emphasizing schema-on-read and the benefits of schema-on-write with Confluent Schema Registry. Finally, it delves into the partitioning of processing, explaining consumer groups and stream tasks as the units of parallelism for distributed applications, and the Kafka rebalance protocol for automatic workload redistribution.

12/23/2019

This post demonstrates the use of ksqlDB 0.6 with Twitter data, showcasing its enhanced capabilities compared to its predecessor, KSQL. Key features highlighted include pull query support for querying materialized views, native integration with Kafka Connect connectors, support for nested structures and array flattening (EXPLODE), improved Avro schema support, and a test runner. The post provides practical examples of configuring a Twitter connector via Kafka Connect, creating a ksqlDB stream over the resulting Kafka topic, filtering columns using push queries with `EMIT CHANGES`, and creating a new, thinner stream (`TWEETS_THIN`) by projecting specific fields from the original stream. It also mentions the use of GitHub for the full test rig.

12/3/2019

Deutsche Bahn details their journey building the RI-Plattform passenger information system using Apache Kafka and Confluent Platform. They describe migrating from Apache Storm to Kafka Streams, highlighting performance improvements (90% reduction in processing time for schedule imports) achieved through local state stores. The system architecture involves ~100 microservices on Kubernetes/AWS, processing 180 million events daily. They achieved 99.9% availability and are considering ksqlDB and Confluent Cloud. A challenge with Kafka Streams rebalancing causing brief downtime is noted, with interest in the Static Membership protocol.

11/20/2019

Introduced ksqlDB, a significant evolution of KSQL, by adding two major features: pull queries for point-in-time data lookups on materialized tables (complementing existing push queries for continuous stream processing) and direct management of Kafka Connectors (source and sink) within ksqlDB. This aims to simplify event streaming application development by consolidating data ingestion, transformation, and querying into a unified SQL interface, reducing the need for separate systems like Kafka Connect and traditional databases.

10/16/2019

This post details building streaming ETL solutions with Apache Kafka and Confluent Platform, focusing on ingesting data from ActiveMQ and REST endpoints using Kafka Connect and CLI tools. It elaborates on data wrangling using KSQL for tasks like splitting streams, applying schemas, serializing to Avro, deriving message keys, enriching data, joining with reference information, and resolving codes. It also showcases using KSQL for real-time aggregation and the integration of Kafka Connect with PostgreSQL and Amazon S3 for advanced analytics. A specific example of building an event-driven alerting system for train delays is provided, detailing the use of KSQL to join movement data with configuration topics and Kafka Connect for pushing alerts via REST API and Telegram.

9/25/2019

This post introduces the concept of event streams as the central nervous system of a business, contrasting them with traditional database architectures and highlighting Apache Kafka's role in enabling real-time data processing and reaction to events. It discusses how companies are increasingly defined by software and how event streams are a paradigm shift from passive data storage to active data flow and real-time reaction.

8/6/2019

Introduces a new Maven archetype (`ksql-udf-quickstart` or `ksqldb-udf-quickstart` for newer versions) to simplify the creation and bootstrapping of ksqlDB User-Defined Functions (UDFs) and User-Defined Aggregate Functions (UDAFs). The archetype generates a project structure with example UDF (REVERSE) and UDAF (SUMMARY_STATS) implementations, demonstrating how to define functions, handle multiple data types, and implement aggregation logic (initialize, aggregate, merge). It also provides instructions for setting up Maven repositories and building the project into an uber JAR for deployment to a ksqlDB server. The post also covers converting the generated Maven project to a Gradle project.

8/1/2019

This post details how Imperva used Kafka Streams to build shared state microservices for distributed systems. It covers the challenges of traditional state management with single databases, the requirements for new shared state microservices, and how Kafka Streams met these requirements. The post explains the processing topology using Kafka Streams, including source, processor with persistent key-value store, and sink. It also details building a RESTful CRUD API on top of Kafka Streams for state retrieval and updates, distributing processing load across a cluster of microservices for scalability and availability, and implementing task scheduling using the processor context's schedule method. The benefits and challenges of moving from a single microservice to a cluster are also discussed.

7/10/2019

This post details the process of building and deploying KSQL user-defined functions (UDFs) and Kafka Streams microservices using Gradle. It provides code examples for creating custom UDFs with Java/Groovy, testing them with the Spock framework, and packaging them into deployable JARs. The post also explains how to configure Gradle for this process, including applying plugins for Groovy and fat JAR creation, and demonstrates setting up a local development environment with Docker Compose.

6/11/2019

This post details how Spring Cloud Data Flow can be used to build and continuously deploy event streaming pipelines using Apache Kafka. It covers common streaming topology patterns like named destinations, parallel pipelines, partitioned streams, and function composition. The post also demonstrates how to handle multiple input/output destinations and provides an example of a Kafka Streams application for processing user click events.

6/7/2019

This post details TokenAnalyst's implementation of a 'block confirmer' using Kafka Streams to handle blockchain reorganizations. It describes maintaining a fork tree in a Kafka Streams state store to process incoming blocks and propagate them only after a confirmation threshold is met, ensuring effective once semantics by discarding already registered and confirmed blocks. It also highlights the use of Kafka Streams' fault tolerance for rolling deploys and zero-downtime environments.

5/30/2019

This post introduces Spring Cloud Data Flow as a toolkit for designing, developing, and continuously delivering data pipelines, focusing on event streaming applications. It explains how Spring Cloud Data Flow orchestrates the deployment of these pipelines, often leveraging Apache Kafka as the underlying messaging middleware. The post details the Spring Cloud Data Flow ecosystem, including its dashboard, shell, DSL, and APIs, and how it integrates with Spring Cloud Skipper for application lifecycle management. It also covers developing event streaming applications using Spring Cloud Stream, outlining source, processor, and sink application types, and demonstrates creating a simple pipeline using the Stream DSL with an http source, a transform processor, and a log sink, all communicating via Kafka topics.

5/29/2019

This post details the challenges of managing KSQL dependencies and pipelines in a source code repository. It introduces a custom Gradle plugin, `gradle-confluent`, which uses a file and directory structure to declare dependencies between KSQL scripts. The plugin ensures scripts are executed in the correct order, addressing issues with recreating KSQL streams and tables by managing dependencies between pipelines and within pipelines. It also highlights the complexity of dropping and recreating KSQL objects due to persistent query dependencies and the need for a more robust dependency management solution.

5/15/2019

This post details the integration of Oracle WMS Cloud with on-premises systems using Apache Kafka and Confluent Platform. It describes using Oracle GoldenGate for Big Data to deliver relational data change events to Kafka and MuleSoft to produce payloads from other APIs to Kafka. For data egress, the Kafka Connect JDBC Sink connector is used to deliver data back into legacy systems, and MuleSoft consumes topics for API calls. The post highlights the use of KSQL for data stream curation and Kafka Streams for final payload packaging, specifically addressing the need for nested messages for Oracle WMS Cloud and leveraging Kafka Streams' built-in state store.

4/30/2019

This post introduces the processor topology optimization framework in Kafka Streams starting from version 2.1.0. It details the evolution of topology generation, explaining the previous approach and highlighting inefficiencies in stateful operations. The post describes the refactoring that enables optimizations and introduces specific techniques available in Kafka 2.1. It also provides operational guidelines for enabling these optimizations and elaborates on how keyed stateful operators utilize data parallelism through repartition topics and sub-topology execution.

4/3/2019

This post introduces the CASE statement to KSQL, allowing for conditional logic within SQL queries. It demonstrates its use for data cleansing, deriving new columns, bucketing data based on ranges, masking sensitive information, inferring missing values, generating conditional aggregates, and routing traffic for A/B testing. Additionally, it details how to configure KSQL to log processing errors to a Kafka topic for improved observability.

3/20/2019

Introduces the `Suppress` operator for KTable in Kafka Streams, enabling controlled forwarding of updates. This operator allows for emitting only final results of windowed aggregations or rate-limiting updates, addressing use cases like alerting and integration with external systems that don't support continuous refinement. The post also clarifies the concept of window lifecycle and grace periods in Kafka Streams.

2/21/2019

This post demonstrates the application of ksqlDB for real-time security event processing by joining Sysmon ProcessCreate and NetworkConnect events to detect lateral movement. It provides a concrete example of using ksqlDB for security analytics, highlighting the benefits of event-driven processing and real-time data enrichment compared to traditional SIEM approaches. The post also details the integration with the HELK platform and explains the fundamental concepts of streams, tables, and joins within ksqlDB in the context of security event correlation.

1/9/2019

This post introduces a tutorial for developing real-time streaming applications using Kafka Streams, covering concepts like event-driven architectures, microservices, and the Kafka Streams API. It details a microservices ecosystem for order management, including services for validation, aggregation, and email notification, and highlights the use of ksqlDB for stream enrichment and fraud detection. The tutorial includes exercises on persisting events, event-driven applications, stream enrichment with joins (including CDC integration), filtering and branching, stateful operations with windowing, state stores, and enrichment with ksqlDB.

12/11/2018

This post details the deployment options for ksqlDB, building upon the Kafka Streams API for stream processing. It explains the execution model of Kafka Streams and how ksqlDB leverages it, offering two deployment options: headless (application mode) for production and interactive mode for exploration. Headless deployments compile SQL files into Kafka Streams applications and run them as independent instances, with load distributed via Kafka's consumer group protocol. Interactive mode allows users to interact with ksqlDB servers via a REST API, CLI, or Confluent Control Center for real-time exploration and query development.

10/4/2018

This post details troubleshooting KSQL by examining its underlying Kafka Streams topology, using `DESCRIBE EXTENDED` and `EXPLAIN` commands, and leveraging Confluent Control Center and JMX metrics for monitoring message throughput, latency, and errors. It also covers the importance of understanding Kafka Streams client IDs and server logs for deeper diagnostics.

9/27/2018

This post details troubleshooting common issues encountered with KSQL, focusing on why a KSQL query might not return data. It outlines five primary reasons: no data in the source topic, no new data arriving, KSQL consuming from a later offset, data not matching predicates, and deserialization errors. The post provides practical diagnostic steps using KSQL commands (DESCRIBE EXTENDED, SET 'auto.offset.reset') and external tools like kafkacat to verify topic contents and offset configurations. It also touches on the importance of correct data formats and predicate logic.

9/25/2018

This post details Zenreach's implementation of a real-time presence detection platform using Kafka Streams. It describes their use case of detecting walk-ins, enriching them with contact information, and maintaining walk-in counts. The post highlights the migration from Python scripts and Spark Streaming to Kafka Streams due to operational complexity and cost. It outlines the four services (Walk-In, DeviceToContact, Enriched Walk-In, Walk-In Count) and their roles in processing data. Key technical lessons learned include scaling Kafka Streams services by adjusting `num.stream.threads`, managing stateful services with persistent storage (EBS volumes) to ensure state availability during deployments, and improving fault tolerance by configuring `num.standby.replicas` to reduce recovery time after instance failures.

9/11/2018

Introduces the Dual Streaming Model, which unifies data streams and relational tables by presenting operator results as a stream of successive updates. This model addresses physical and logical order in data stream processing, decoupling out-of-order data handling from latency concerns and defining a design space between processing cost, latency, and result completeness. It also discusses implementation trade-offs and challenges.

9/7/2018

This post details data wrangling tasks using KSQL, including applying schemas, filtering, masking, restructuring, enriching, and unifying data streams. It demonstrates pulling data from REST sources using Kafka Connect, processing it with KSQL, and streaming it to Google Cloud Storage and BigQuery. The post also covers declaring schemas for JSON data and handling variations in data structures, such as arrays.

8/17/2018

Introduces the official API for building User-Defined Scalar Functions (UDFs) and User-Defined Aggregate Functions (UDAFs) in ksqlDB 5.0. Details the motivation for UDFs/UDAFs, the steps to create, deploy, and test them using a Java class and annotations, and provides an example of a deep-learning UDF for anomaly detection. Also outlines potential issues during development and testing and discusses future roadmap items.

8/10/2018

This post introduces Kafka Streams as a library for building event-driven applications in Java to process data in Apache Kafka topics. It highlights how applications using this library can perform sophisticated transformations on data streams, with fault tolerance and elastic distribution across application instances. The post contrasts Kafka's stream processing approach with Big Data frameworks like Storm, Flink, and Spark, emphasizing Kafka's goal of making stream processing accessible for event-driven applications rather than a heavy-weight 'big data' task. It also discusses the broader trend of event-driven architectures powered by Apache Kafka and the shift towards continuous, real-time processing of business events.

6/26/2018

This post introduces Confluent Cloud and KSQL as a means to simplify stream processing. It details how to design, configure, and manage streaming applications using Confluent Cloud, including launching a cluster, management with Confluent Control Center, stream processing with KSQL (including its UI and performance monitoring), data governance with Schema Registry, and getting data into and out of Confluent Cloud. A demo is also provided to showcase a hybrid deployment.

4/5/2018

This post introduces the use of KSQL for real-time processing of syslog data ingested into Apache Kafka. It demonstrates how to declare a KSQL stream over the 'syslog' topic and provides an example of filtering syslog messages using KSQL's SQL-like syntax. The post highlights the integration of Kafka Connect with a syslog plugin for data ingestion and the persistence capabilities of Kafka for data utilization.

2/1/2018

This post details the release of KSQL 0.4, introducing several new features and improvements. Key additions include enhanced `PRINT TOPIC` and `SHOW TOPICS` commands for data exploration, new aggregation functions `TOPK` and `TOPKDISTINCT` for analytics, and the ability to print KSQL metrics from the command line. It also announces the availability of binary tarballs and improvements in distributed failure testing for KSQL.

12/20/2017

Introduced Avro support for KSQL through integration with Confluent Schema Registry, enabling automatic schema inference and conversion between Avro, JSON, and delimited formats. Added the ability to join streams and tables regardless of their underlying data formats. Enhanced metrics and observability with DESCRIBE EXTENDED and EXPLAIN statements for streams, tables, and queries. Improved KSQL server startup time and resource utilization by detecting and avoiding redundant processing topologies.

11/21/2017

This post announces the November update to KSQL, a streaming SQL engine for Apache Kafka. The update focuses on addressing community-raised issues and requests for the Developer Preview, including 369 pull requests and 50 closed issues. It highlights KSQL's capabilities for real-time use cases such as monitoring, ETL, data exploration, and machine learning validation. The post also announces monthly releases for KSQL and the kick-off of a KSQL beta program, encouraging community involvement through feedback and code contributions.

11/9/2017

This post details building a microservices ecosystem using Kafka Streams and KSQL. It explains how to leverage Kafka Streams for stateful stream processing, including joining streams and tables, and managing consistency. It also introduces KSQL as an alternative for stateful stream processing in a separate process, allowing for SQL-like queries on event streams. The post provides a concrete example of building a simple email service using these technologies, highlighting the benefits of event-driven architectures for decoupling services and enabling parallelism.

11/8/2017

This post details using KSQL to process real-time device data from a gaming steering wheel. It covers writing a custom Kafka producer using SharpDX and Confluent's .NET Kafka Client to ingest raw axis inputs into Kafka. It then demonstrates creating a KSQL source stream from the Kafka topic and deriving new streams with custom transformations for steering (converting raw values to degrees), accelerator, and brake pedals (converting raw values to a 0-100 range). The post also shows how to visualize this processed data in Grafana.

10/24/2017

This post details Zalando's experience running Kafka Streams applications in production on AWS. It covers challenges and solutions related to AWS EBS I/O throttling, fine-tuning Kafka Streams configurations (e.g., max.poll.records, processing timeouts), and managing JVM heap and off-heap memory usage with RocksDB. It also introduces custom monitoring tools like Remora for consumer lag and discusses the importance of monitoring EBS burst balance, JVM heap, CPU, system memory, and disk usage.

10/19/2017

This post details how Apache Kafka's Streams API can be used to implement real-time ranking algorithms like HITS, processing web link data to identify influential domains. It covers the extraction of domain links, aggregation of these links using Kafka Streams' KTable, and the iterative calculation of Hub and Authority scores for real-time website ranking. The post highlights the flexibility of Kafka Streams for building scalable, real-time data processing applications.

10/10/2017

This post details how to ingest raw Twitter data into Apache Kafka using a Kafka Connect Twitter source connector. It then demonstrates how to use KSQL to define schemas for the incoming JSON data, query the raw stream, create derived streams with transformations (e.g., timestamp conversion, JSON field extraction, LCASE, pattern matching), and build aggregations (e.g., tweet counts per user per hour). It explains the concept of continuous queries and the distinction between streams and tables in KSQL, illustrating how aggregations result in tables. The post also shows how to create derived tables from aggregated data for easier querying.

10/9/2017

This post details how Pinterest uses the Kafka Streams API to build a predictive budgeting system for their ads infrastructure. The system aims to reduce ad overdelivery by calculating 'inflight spend' (cost of ads insertions not yet charged). Key technical contributions include: 1. Designing a system to handle tens of thousands of events per second with sub-10-second end-to-end delay and 100% uptime. 2. Choosing Kafka Streams for its millisecond delay guarantee and lightweight nature compared to Spark and Flink. 3. Implementing a plan involving an 'inflight spend' service that consumes ad insertion events, aggregates spend using Kafka Streams with a 10-second window store per adgroup, and fans out updates to thousands of consumer machines. 4. Optimizing performance by switching from hopping windows to tumbling windows, resulting in an 18x throughput improvement. 5. Employing delta encoding for adgroup IDs and lookup table encoding for spend data to achieve a 4x reduction in message size.

9/19/2017

This post details the join capabilities of Kafka Streams, including inner, left, and outer joins between KStreams and KTables, and the specific semantics of stream-stream joins with windowing. It explains the core concepts of KStream and KTable, the differences in join types, and provides detailed examples of inner and left KStream-KStream joins with their respective output.

8/28/2017

Introduces KSQL, a streaming SQL engine for Apache Kafka, enabling continuous queries on unbounded data streams. Details STREAM and TABLE abstractions, use cases like real-time monitoring, anomaly detection, and data integration. Explains KSQL's internal architecture using Kafka Streams API and its distributed nature.

5/3/2017

Critiques the Google Dataflow model's use of watermarks and triggers for handling time and out-of-order data in stream processing. Proposes Kafka Streams' 'Table' abstraction as a more general and simpler approach. Explains how Kafka Streams handles mutable data and windowed computations by treating them as continuously updated tables. Discusses operational tuning parameters (commit interval, cache size) for managing output volume and update lag. Highlights the benefits of interactive queries for accessing the latest results.

10/26/2016

Introduces the 'Interactive Queries' feature for Kafka Streams. This feature allows direct querying of the state maintained by a stream processing application, treating it as an embedded, highly available, and fault-tolerant database. It contrasts this with traditional materialized views and the inefficiencies of Lambda architectures, advocating for a converged processing and storage layer within Kafka. The post details the motivation, an example of real-time risk management, and the underlying mechanism using embedded databases like RocksDB, highlighting trade-offs for using this feature versus external databases.

8/15/2016

This post details how to reprocess input data from scratch in Kafka Streams applications. It introduces an application reset tool that handles global resets by clearing internal topics and state stores, and requires a local cleanup call within the application code. The post also dives into the internal mechanics of Kafka Streams, including intermediate topics, internal changelog topics, and committed offsets, to explain the complexities of manual resets and the benefits of the new tool.

7/21/2016

Details the security features of the Kafka Streams API, focusing on client authentication, authorization, and data encryption (in-transit). Explains how these features integrate with Apache Kafka's security mechanisms and provides a code example for configuring SSL encryption and client authentication within a Streams API application's configuration.