TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Stream Processing Fundamentals

84 posts · Jun 2016 – May 2026

What this trail covers

ksqlDB continues to evolve as a powerful stream processing engine, enabling declarative data manipulation and real-time data replication. This release introduces support for Avro data format through integration with Confluent Schema Registry, automatic schema inference, and the ability to convert between Avro, JSON, and delimited formats in real-time. It also enhances joins across different data formats and lays the groundwork for observability features with new DESCRIBE EXTENDED and EXPLAIN sta

All Confluent trails
  1. Add Your First Flink Streaming TransformationThis post introduces the concept of streaming transformations in Apache Flink, contrasting them with batch processing. It details how Flink can be incrementally adopted to process data in motion from Kafka topics, enabling real-time filtering, field projection, reshaping, and introductory aggregation. The post provides a step-by-step guide for adding a first Flink transformation, emphasizing stateless operations for simplicity and fault tolerance. It highlights the architectural shift from passive to active data streams and common first transformations like event filtering and field projection.
  2. Post-Quantum Cryptography in Confluent CloudThis post introduces 'Queues for Kafka' (GA on Confluent Cloud and Platform), built on KIP-932, which brings queue semantics to Kafka. It enables infrastructure consolidation by allowing both streaming and queuing workloads on a single platform. Key features include a new share consumer API and share groups that allow elastic scaling of consumers beyond partition count, per-message processing controls (acknowledge, release, reject, renew), and leveraging Kafka's durability. The mechanism involves broker-level acquisition locks on individual records with configurable timeouts. The post details the 'explicit' and 'implicit' acknowledgement modes and provides code examples for both. It also highlights the integration with Confluent Cloud Console and Confluent Platform Control Center for management and monitoring.
  3. Scaling Kafka Streams for High-Volume Data ProcessingThis post introduces strategies for scaling Kafka Streams applications for high-volume data processing. It explains the relationship between topic partitions and Kafka Streams tasks, detailing how partition-to-task assignment works for single and multiple sub-topologies. It outlines scaling out (horizontal) and scaling up (vertical) with specific use cases and configuration adjustments. Key performance tuning parameters like num.stream.threads, rocksdb.config.setter, cache.max.bytes.buffering, commit.interval.ms, and producer/consumer configs are detailed. The post also differentiates scaling for stateful vs. stateless applications, emphasizing co-partitioning for joins, and lists critical monitoring metrics like consumer lag and CPU utilization.
  4. Ensure Data Quality With Real-Time Validation and MonitoringThis post introduces real-time data validation and monitoring as a critical capability for ensuring data quality in streaming architectures. It contrasts batch validation with real-time approaches, highlighting the 'shift left' paradigm. The solution overview details how Apache Kafka and Confluent, specifically Schema Registry and stream processing engines like Flink or ksqlDB, can be used to enforce schema compatibility and apply business rule checks in real-time. It also emphasizes the importance of monitoring integrations for visibility. Use cases in financial services, retail, healthcare, and AI/ML are presented, followed by a step-by-step guide for implementing real-time validation, starting with data ingestion into Kafka topics.
  5. How to Build Real-Time Apache Kafka® Dashboards That Drive ActionThis post details how to build real-time dashboards by ingesting streaming data into Kafka topics, using a stream processor (ksqlDB, Flink, Kafka Streams) to create materialized views, and connecting these views to BI tools or custom applications for visualization and action. It emphasizes data modeling for real-time performance, including denormalization and partitioning strategies, and introduces delivery patterns like read-optimized views, search-optimized indexes, and app-embedded state using Kafka Streams interactive queries.
  7. Windowing in Kafka StreamsThis post introduces and details the four types of windowing in Kafka Streams and ksqlDB: hopping, tumbling, session, and sliding windows. It explains the concepts behind each window type, their respective use cases (e.g., moving averages, reporting, user activity tracking, event-based calculations), and how to configure them using Kafka Streams API parameters (`windowSize`, `advanceSize`, `inactivityGap`, `timeDifference`) and ksqlDB syntax (`WINDOW HOPPING`, `WINDOW TUMBLING`, `WINDOW SESSION`). It also discusses the handling of late events with grace periods and notes the deprecation of the default 24-hour grace period in Kafka Streams.
  8. Getting Started with Stream Processing: The Ultimate GuideThis post introduces ksqlDB as a simplified, SQL-based stream processing engine that integrates with Kafka. It highlights ksqlDB's benefits, including processing data in motion, simplified stream processing architecture by reducing infrastructure to Kafka and ksqlDB, and lightweight SQL syntax. It details technical use cases such as streaming data pipelines, materialized caches, and event-driven microservices. Key ksqlDB constructs like streams, tables, persistent queries, and push/pull queries are explained. The post positions ksqlDB as a component of Confluent's platform for harnessing data in motion.
  9. Stream Processing vs Batch ProcessingThis post defines stream processing and contrasts it with batch processing, explaining the core characteristics of stream processing: continuous processing, low latency, and support for event-time processing. It details common use cases for stream processing across various industries and explains how Apache Kafka facilitates stream processing through its stream-table duality and real-time data pipelines. It also introduces Kafka Streams and ksqlDB as methods for implementing stream processing solutions.
  10. Serverless Event Stream ProcessingThis post defines serverless stream processing, stateless vs. stateful processing, FaaS, and serverless. It details how ksqlDB handles both stateless and stateful processing, and how AWS Lambda functions as a FaaS. A key contribution is the proposed pattern of combining ksqlDB with AWS Lambda for event-driven applications, where ksqlDB performs stateful stream processing and Lambda handles stateless tasks triggered by ksqlDB outputs. A concrete example is provided with a GitHub repository demonstrating a ksqlDB application integrated with AWS Lambda for two-way communication, processing stock trades and user data.
  11. Serverless, Secure Data Streaming with Confluent Cloud and AWS LambdaThis post details the integration of AWS Lambda with Confluent Cloud, enabling serverless data streaming. It specifically focuses on ingesting data from Kafka topics into Amazon DynamoDB tables by triggering an AWS Lambda function using the newly supported SASL/PLAIN authentication. The post provides a step-by-step guide for setting up Confluent Cloud, creating a cluster and topic, and launching AWS CloudFormation to provision the necessary AWS resources, including the Lambda function and DynamoDB table. It highlights the benefits of a serverless architecture for data-in-motion use cases.
  12. Rethinking Distributed Stream Processing in Apache KafkaThis post introduces and elaborates on the concepts of consistency (exactly-once semantics) and completeness in distributed stream processing, specifically within the context of Apache Kafka and Kafka Streams. It explains how Kafka's persistent, replicated append-only logs enable ordered transactional log appends and replays for consistency, and how Kafka Streams uses speculative processing with revision-based mechanisms to handle out-of-order data for completeness. The post highlights the trade-offs between latency, throughput, and correctness guarantees inherent in these approaches.
  13. Performance Tuning RocksDB for Kafka Streams’ State StoresThis post details performance tuning for RocksDB state stores within Kafka Streams. It explains how Kafka Streams uses RocksDB for stateful operations, the layered architecture of state stores (metrics, cache, changelogging, RocksDB), and the fault tolerance provided by changelog topics. It introduces RocksDB's internal workings, including its key-value operations and compaction styles (level and universal). The post highlights the importance of RocksDB configuration via RocksDBConfigSetter and the utility of RocksDB-specific metrics (KIP-471, KIP-607) for operational diagnostics and tuning.
  14. Keys Unlocked - ksqlDB 0.15 Supports Even More Data TypesThis post introduces significant enhancements to ksqlDB 0.15, primarily focusing on message key capabilities. Key contributions include: 1. Support for all serialization formats (JSON, AVRO, PROTOBUF, DELIMITED, KAFKA, JSON_SR) for message keys. 2. Expanded data type support for keys, including ARRAY, STRUCT, and nested combinations, with specific considerations for MAP types and single-field wrapping. 3. Introduction of composite keys, allowing multiple columns to form a single key. 4. Enhanced PARTITION BY and GROUP BY clauses to accept multiple expressions, leading to tables and streams with multiple key or primary key columns respectively. This directly impacts how data is structured, partitioned, and aggregated within ksqlDB.
  15. Using Kafka, ksqlDB, and Quarkus for Real-Time Sports TrackingThis post details the engineering efforts behind building a real-time table football score tracking application ('Töggelomat') using Apache Kafka, ksqlDB, and Quarkus. It covers event sourcing for player data, using ksqlDB to initialize player scores and process match results, calculating player score changes based on match outcomes, and applying these changes back to player scores. The post highlights the use of Quarkus for its reactive nature and low footprint in event-driven applications, and ksqlDB's ability to treat event streams like a database.
  16. ksqlDB 0.14.0 - Newest Features and UpdatesIntroduced multiple key lookups in pull queries using the IN predicate. Added ALTER syntax for incremental schema modification of streams and tables created with DDL. Implemented SQL variable substitution for dynamic query execution. Added support for DELIMITED and JSON key formats, with plans for Schema Registry integrated key formats and non-primitive key types in the future.
  17. 4 ksqlDB Techniques That Make Stream Processing Easier Than EverThis post details four ksqlDB techniques that simplify stream processing: modeling and accessing JSON structures (including heterogeneous and nested JSON), extracting nested fields from JSON strings stored as escaped strings using EXTRACTJSONFIELD, dynamically masking sensitive fields in real-time using the MASK function and creating new streams with masked data, and updating active stream processing queries with CREATE OR REPLACE syntax to modify masking logic for new events.
  18. Analysing historical and live data with ksqlDB and Elastic CloudThis post details building data pipelines by transforming raw CSV data from car park occupancy into a structured format using ksqlDB, applying schemas, setting message keys for partitioning, adding source fields for lineage, deriving and storing timestamps, creating nested location structs, and calculating new fields. The processed data is then streamed to Elasticsearch for analytics and visualization with Kibana.
  19. How Real-Time Stream Processing Safely Scales with ksqlDBThis post details how ksqlDB achieves linear scalability and fault tolerance by leveraging Kafka's consumer group primitive. It explains how ksqlDB distributes workload across multiple servers by partitioning input streams and assigning partitions to servers. It also describes how consumer groups handle dynamic load balancing and fault recovery by automatically reassigning partitions when servers fail. The post differentiates between stateless and stateful operations and how fault tolerance is achieved for each, highlighting the role of offset commits for stateless recovery and Kafka Streams' state management for stateful operations.
  20. Announcing Pull Queries in Confluent Cloud ksqlDBThis post introduces pull queries as a preview feature in Confluent Cloud ksqlDB. It explains how pull queries differ from persistent queries by following a request-response model to retrieve the latest result from a materialized view. The post provides detailed CLI configuration steps for authorizing clients, including obtaining application endpoints, IDs, and Kafka cluster IDs, creating API keys, and setting up ACLs. It also includes a tutorial for creating a materialized view and issuing a pull query. Limitations during the preview phase are outlined, including throughput rate limits, eventual consistency, potential performance impact on push queries, and the absence of an SLA. The post concludes by highlighting the flexibility offered by combining push and pull patterns for real-time applications and notes that pull queries will be free during preview.
  21. How to Limit Memory Usage in ksqlDBThis post details how ksqlDB manages memory usage, focusing on bounding RocksDB's off-heap memory through shared caches and write buffer managers. It investigates a real-world scenario of unbounded memory growth, identifies the root cause in glibc's malloc behavior with terminated queries not releasing memory effectively, and proposes solutions. The post highlights the effectiveness of alternative allocators like jemalloc and TCMalloc, and explains the internal mechanics of glibc's arenas and heaps to understand memory retention. It also details how Confluent Cloud addresses this by configuring JVM heap and RocksDB limits, and by implementing a KsqlBoundedMemoryRocksDBConfigSetter to manage shared caches and write buffers.
  22. How Real-Time Materialized Views Work with ksqlDBThis post details how ksqlDB builds real-time materialized views using stateful stream processing, leveraging RocksDB for local storage and Kafka changelog topics for durability and recovery. It explains automatic repartitioning for GROUP BY operations and the role of changelog compaction. The post also introduces the LATEST_BY_OFFSET aggregation for retaining the last value per key.
  23. ksqlDB Meets Java: An IoT-Inspired Demo of the ksqlDB Java ClientThis post introduces the ksqlDB Java client, enabling programmatic interaction with ksqlDB servers. It demonstrates its use in an IoT application for dynamically managing sensor types, creating streams and materialized views using LATEST_BY_OFFSET, and performing administrative operations. The post also showcases asynchronous programming with CompletableFutures for non-blocking operations.
  24. Announcing ksqlDB 0.12.0 - Newest Features and UpdatesThis post introduces query upgrades for ksqlDB persistent queries using `CREATE OR REPLACE` for WHERE clause expressions and schema changes, allowing in-place evolution without data loss. It also adds automatic restarts for persistent queries that enter an ERROR state, improving resilience and reducing manual intervention. These features aim to reduce administration time and enhance the usability of ksqlDB.
  25. How Stream Processing Works with ksqlDBThis post provides a deep dive into the internal workings of ksqlDB, explaining how it processes streams of data from Kafka. It details the process of declaring streams with schemas, inserting rows (which are mapped to Kafka records with keys and values), and executing persistent queries. Persistent queries are compiled into Kafka Streams topologies that run as daemons, managing their own processing progression and durably maintaining input partition offsets. The post also covers data transformation, filtering, and rekeying operations, emphasizing how ksqlDB simplifies these complex stream processing tasks by abstracting away low-level details like schema management, serializers, and partitioning strategies.
  26. The Curious Incident of the State Store in Recovery in ksqlDBThis post details a bug in ksqlDB's state store recovery mechanism where an optimization for avoiding changelog topics led to incorrect schema registration with Schema Registry during node relocation. The bug (KAFKA-10179) caused deserialization errors when ksqlDB attempted to read from the state store using the wrong schema ID and subject. A temporary fix involves registering the schema ID under the phantom changelog subject. The future roadmap includes improving ksqlDB's integration with Schema Registry to handle custom internal schemas more robustly.
  27. ksqlDB 0.11.0 - New Features & ImprovementsThis post details ksqlDB 0.11.0, focusing on stability improvements and new features. Key contributions include: 1. HTTP client caching for inter-node requests to improve pull query performance in multi-node clusters. 2. Windowed table Apache Kafka topic retention, ensuring aggregate state and old windows are properly managed in the underlying Kafka topic. 3. Expanded Java client functionality with methods for creating/managing streams, tables, and persistent queries (via executeStatement()), and listing available streams, tables, topics, and queries (listStreams(), listTables(), listTopics(), listQueries()). 4. Enhanced pull query support with the addition of WINDOWEND keyword filtering, useful for session windows.
  28. SQL Changes and Key Columns in ksqlDB 0.10 ExplainedIntroduces significant changes to ksqlDB's key handling in version 0.10, including the removal of implicit ROWKEY columns and the WITH(key='ID') syntax. Users can now explicitly name key columns for both tables (PRIMARY KEY) and streams (KEY), directly mapping to Kafka record keys without requiring data duplication in the value. This simplifies queries and improves performance by avoiding unnecessary repartitions.
  29. How PushOwl Uses ksqlDB to Scale Analytics and Reporting Use CasesThis post details how PushOwl migrated from a PostgreSQL-based solution to Apache Kafka and ksqlDB to handle increasing data volumes and improve reporting performance. It describes two failed attempts with PostgreSQL, highlighting the limitations of traditional databases for high-volume event data. The final solution uses ksqlDB for stream aggregation, publishing events to Kafka, and then using Kafka Connect JDBC sink to store aggregated data back into PostgreSQL. Code examples for Avro schemas and producer configuration using the confluent-kafka Python package are provided.
  30. Using ksqlDB and UDTFs for Infrastructure MonitoringThis post introduces the creation and deployment of a User-Defined Table Function (UDTF) in ksqlDB for infrastructure monitoring. It details the Java code for a `SplitPanel` UDTF that takes a struct representing panel readings and outputs individual breaker readings along with system and panel identifiers. The post also outlines the ksqlDB schema definitions for creating streams and tables to ingest customer and panel data, and how to structure the panel data into a struct for the UDTF. The overall goal is to enrich raw infrastructure data with customer context for better analysis and prediction.
  31. Monitoring & Metrics for Confluent Cloud ksqlDB - Stream Processing ServiceThis post details the monitoring and alerting strategy for Confluent Cloud ksqlDB, focusing on ensuring prompt response to failures, proactive mitigation of potential issues, and confidence in the system's reliability. It covers health checks, specific alerting for system vs. user errors, proactive monitoring of resource utilization with trajectory analysis, and redundancy in the monitoring pipeline to prevent alert fatigue and ensure the monitoring system itself is robust.
  32. ksqlDB 0.10.0: New Java Client, Easy Kafka Data Processing, and MoreIntroduced a first-class Java client for ksqlDB, enabling programmatic interaction with push and pull queries, and row insertion. Enhanced Kafka key support by allowing any name for key columns (previously restricted to ROWKEY) and removing the redundant WITH KEY syntax. Introduced explicit primary key requirements for tables and keyless streams. Added numerous new built-in functions including CHR, ENCODE, REGEXP_EXTRACT_ALL, REGEXP_REPLACE, REGEXP_SPLIT, SPLIT_TO_MAP, EARLIEST_BY_OFFSET, INSTR, LPAD, RPAD, and various array and map manipulation functions.
  33. Unifying Streams and State: The Seamless Path to Real TimeThis post introduces ksqlDB as an event streaming database that unifies stream processing and state management, simplifying architectures by reducing the number of separate systems required and offering a single SQL query language for both streams and point-in-time state. It integrates with Kafka connectors and Schema Registry, and is available as a managed service in Confluent Cloud. The focus is on making data pipelines and asynchronously materialized views more accessible and cost-effective.
  34. Consistent Metastore Recovery for ksqlDB Using Kafka TransactionsThis post details the implementation of Kafka transactions to ensure consistent metastore recovery for ksqlDB in interactive mode. It addresses the flaws in the previous design where the command topic was treated as both a request queue and a log, leading to potential inconsistencies due to statement validation errors and race conditions. The new protocol treats the command topic strictly as a log by performing validation before enqueueing to the command topic and isolating servers using Kafka's transactional.id. This ensures that only one server can validate and produce to the command topic at a time, guaranteeing deterministic metastore recreation.
  35. Kafka Streams 2.5.0 - Even Higher Availability & Interactive QueriesIntroduces KIP-535 to allow Kafka Streams state stores to serve stale reads from standby partitions during rebalances or VM failures, improving availability. Details KIP-562 for querying specific partitions and optimizing lookups by avoiding iteration over all stores on an instance. Fixes KAFKA-9169 to prevent unnecessary restoration of standby tasks, reducing resource consumption and potential errors.
  36. Announcing ksqlDB 0.9.0 | New Features and UpdatesThis post introduces multi-join statement support in ksqlDB, enabling the combination of multiple joins into a single query for conciseness and efficiency. It also enhances LIKE expressions by allowing the '%' wildcard at any position and introducing the '_' wildcard for single character matching, along with an ESCAPE clause. A new COALESCE built-in function is also added.
  37. Building a Telegram Bot Powered by Kafka and ksqlDBThis post demonstrates how ksqlDB can be used to build stateful aggregates directly from Kafka streams, eliminating the need for external datastores. It details the process of creating a STREAM from raw Wireshark data, materializing state into a TABLE for aggregated statistics (probe count, first/last probe time, unique SSIDs), and querying this state using pull queries. The architecture leverages Kafka as the primary system of record and ksqlDB's ability to build materialized views accessible via a REST API. The example application is a Telegram bot that queries Wi-Fi probe activity statistics.
  38. Highly Available, Fault-Tolerant Pull Queries in ksqlDBThis post details the engineering efforts to make pull queries remain highly available in the face of server failures by implementing faster failure detection via heartbeating and enabling lag-aware routing to standby replicas, allowing ksqlDB to operate as an AP system under certain conditions. It explains the trade-offs between CP and AP systems using the CAP and PACELC theorems, and describes the implementation of a ksqlDB server mesh for heartbeating and the use of KIP-535 for standby routing and lag-aware routing.
  39. ksqlDB Execution Plans: Move Fast But Don’t Break ThingsIntroduces the concept of execution plans in ksqlDB as a solution for maintaining backward compatibility during upgrades. Execution plans are JSON serializable internal specifications that represent the ksqlDB query's structure, allowing the ksqlDB engine to evolve independently of the Kafka Streams topology generation. This enables new optimizations and features to be applied to new queries without breaking existing, running queries. The post details the motivation for execution plans, the previous compatibility challenges with compatibility-breaking configs, and the architecture of execution plans as a DAG of steps. It also outlines the testing strategies for ensuring the compatibility of this internal API.
  40. Broadcom Modernizes Machine Learning & Anomaly Detection with ksqlDBThis post details a prototype integration of ksqlDB into Broadcom's Mainframe Operational Intelligence (MOI) product to improve anomaly detection and reduce false positives in metric filtering. It demonstrates how ksqlDB can be used to create metric filters based on a combination of metric instance information, values, and types. The prototype involved inserting a ksqlDB server between the Machine Learning Service and the Alarm Lifecycle Management Service, with ksqlDB filtering alert flags. The post covers the installation of ksqlDB server, creating streams with custom timestamps, selecting specific metric instances, and defining filter rules using INNER JOIN WITHIN. It also addresses challenges with Avro schema mapping, specifically casing differences and map serialization, and outlines a solution involving retrieving schemas from Confluent Schema Registry and generating corresponding POJOs.
  41. Real-Time Business Intelligence & Analytics with ksqlDBThis post introduces ksqlDB as a tool for organizations of any size to build event streaming pipelines with minimal infrastructure and code. It details a use case with an online pharmacy to demonstrate how transactional data can be turned into business intelligence. The implementation focuses on using ksqlDB to answer questions about brand/product performance, customer satisfaction, sales trends, practitioner performance, and the impact of discounting. It highlights the event-driven nature, real-time results, minimal burden on existing systems, and extensibility of ksqlDB. The post provides code examples for creating derived streams and tables from user and order data.
  42. Confluent Cloud KSQL is Now AvailableThis post announces the general availability of Confluent Cloud KSQL, a fully managed stream processing service. It highlights the benefits of using SQL for stream processing, eliminating the need for complex application code. Key features include an intuitive web interface for creating streams/tables and submitting queries, a 'Flow' view for visualizing query topologies, and programmatic integration via a REST endpoint. The service seamlessly integrates with Kafka and Schema Registry, with future integration planned for Kafka Connect. No operational overhead is required from the user.
  43. Real-Time Data Replication with ksqlDBThis post details how Bolt uses ksqlDB for real-time data replication to their data warehouse. It outlines the requirements for a data replication pipeline (consistency, resilience, scalability, data manipulation, denormalization, ease of maintenance) and explains why ksqlDB was chosen over other frameworks. The post provides concrete ksqlDB SQL examples for creating streams, replicating data, filtering, merging, and using UDFs. It emphasizes ksqlDB's native integration with Kafka, ease of deployment, scalability, Schema Registry integration, declarative query language, and extensibility.
  44. ksqlDB: The Link Between Real-Time Data and Big Data StreamingThis post introduces ksqlDB as a bridge between real-time event streaming and traditional big data/state-oriented architectures. It contrasts state orientation (pull queries, databases) with event orientation (push events, EDA) and discusses the complexity of implementing both. The post highlights how ksqlDB, as an event streaming database, allows for both immutable event logging (stream tables) and real-time querying of current states (materialized views/tables) using SQL. It demonstrates both pull queries for current state and push queries (EMIT CHANGES) for continuous updates, enabling use cases like alerting and ML model application. The post positions ksqlDB as a simplification for architectures that previously required complex integration of both paradigms.
  45. Announcing ksqlDB 0.8.0 | New Features and UpdatesIntroduced tunable retention and grace period for windowed aggregates in aggregation statements (e.g., WINDOW TUMBLING (SIZE 1 MINUTE, RETENTION 7 DAYS, GRACE PERIOD 10 MINUTES)). Added the LATEST_BY_OFFSET aggregation function to track the latest value of a column based on offset order. Included an early look at the new client and server API (KLIP-15). Bundled confluent-hub into ksqlDB server and CLI Docker images for easier connector installation. Added REGEXP_EXTRACT and ARRAY_LENGTH built-in functions.
  46. Kafka Streams: Turning Data at REST to Data in MotionThis post details how a proof of concept was built using Kafka Streams and Confluent Platform to ingest data from a REST API (flight systems) and process it into a near-real-time dashboard. It covers the architecture, including a Kafka producer for ingesting data, Kafka Streams for transformations and aggregations (e.g., flight lists, top airlines, fastest flights), and WebSockets for streaming processed data to a browser-based Progressive Web App (PWA) built with Vue.js and TypeScript. The post highlights challenges with the REST API's call limits and the decision to use Kafka Streams for its ability to handle backpressure and perform windowed aggregations. It also touches on data refinement, system resiliency, data update, and optimizing machine resources.
  47. Announcing ksqlDB 0.7.0 | New Features and UpdatesIntroduced highly available state for ksqlDB queries by enabling routing to standby servers during cluster rebalances, with configurable staleness tolerance (ksql.query.pull.max.allowed.offset.lag). Added count_distinct and cube functions. Expanded support for record keys to include INT, BIGINT, and DOUBLE. Added first-class constructors for native arrays, maps, and structs.
  48. SIEM with Anomaly Detection Using Machine Learning and ksqlDBThis post introduces the integration of Latent Dirichlet Allocation (LDA) machine learning models into ksqlDB via User-Defined Functions (UDFs) for real-time anomaly detection in streaming data. It details a Lambda Architecture approach for training LDA models in batch and scoring incoming osquery logs in real-time. The post explains how ksqlDB queries are used to route scored events into 'GOOD', 'BAD', and 'UGLY' topics based on anomaly scores, enabling SIEM integration and feedback loops for model retraining.
  49. Building a Materialized Cache with ksqlDBThis post introduces the concept of a materialized cache and demonstrates how to build one using ksqlDB. It covers defining streams from CDC events, using windowed aggregations with geospatial filtering, creating persistent tables from these aggregations for pull queries, and accessing the materialized cache via ksqlDB's REST API. It highlights the benefits of offloading read-heavy queries from traditional databases to a stream processing engine for improved performance and scalability.
  50. Streams and Tables in Apache Kafka: Elasticity, Fault Tolerance & Advanced ConceptsThis post dives into the architectural underpinnings of elasticity and fault tolerance in stream processing with Kafka Streams and ksqlDB. It explains how the stream-table duality, specifically the use of changelog topics, ensures fault tolerance for tables by providing a durable source of truth. It details how this mechanism also enables elastic scaling by allowing state to be restored on new instances during rebalancing. The post further elaborates on the benefits of topic compaction for reducing storage footprint and recovery time, and introduces the concept of standby replicas to minimize recovery time during failures and scale-in events.
  51. Streams and Tables in Apache Kafka: Event Processing FundamentalsThis post explains the fundamental concepts of streams and tables in Kafka's processing layer, contrasting them with the storage layer. It details how topics are transformed into streams (unbounded, schema-aware data) and tables (aggregated streams, often bounded). The post also covers the importance of data contracts and schema management (schema-on-read vs. schema-on-write), and how processing is partitioned per topic partition, managed by consumer groups and stream tasks for parallel processing.
  52. Exploring ksqlDB with Real-Time Twitter DataThis post demonstrates the use of ksqlDB 0.6 to ingest real-time Twitter data via a Kafka Connect source connector, create a stream from the topic, filter columns, create a new, thinner stream, and build aggregate materialized views with pull queries. It highlights improvements in ksqlDB since its predecessor, KSQL, including pull query support, native Kafka Connect integration, and support for nested structures and array flattening.
  53. Event Streaming at Deutsche Bahn with Apache Kafka & Confluent PlatformDeutsche Bahn details their journey building the RI-Plattform passenger information system using Apache Kafka and Confluent Platform. They describe migrating from Apache Storm to Kafka Streams, highlighting performance improvements (90% reduction in processing time for schedule imports) achieved through local state stores. The system architecture involves ~100 microservices on Kubernetes/AWS, processing 180 million events daily. They achieved 99.9% availability and are considering ksqlDB and Confluent Cloud. A challenge with Kafka Streams rebalancing causing brief downtime is noted, with interest in the Static Membership protocol.
  54. Introducing ksqlDBIntroduces ksqlDB, a significant evolution of KSQL, by adding two major features: pull queries and connector management. Pull queries enable point-in-time lookups on materialized tables, complementing existing push queries (continuous streams of changes). Connector management allows direct control of Kafka Connectors within ksqlDB for data ingestion and export. This aims to simplify event streaming application architectures by reducing the number of distributed systems required.
  55. Build Streaming ETL Solutions with Apache Kafka & Rail DataThis post details building streaming ETL solutions with Apache Kafka and Confluent Platform, focusing on ingesting data from ActiveMQ and REST endpoints using Kafka Connect and CLI tools. It elaborates on data wrangling using KSQL for tasks like splitting streams, applying schemas, serializing to Avro, deriving message keys, enriching data, joining with reference information, and resolving codes. It also showcases using KSQL for real-time aggregation and the integration of Kafka Connect with PostgreSQL and Amazon S3 for advanced analytics. A specific example of building an event-driven alerting system for train delays is provided, detailing the use of KSQL to join movement data with configuration topics and Kafka Connect for pushing alerts via REST API and Telegram.
  56. ksqlDB UDFs and UDAFs Made Easy | ConfluentThis post introduces a new Maven archetype for ksqlDB UDFs and UDAFs, simplifying project bootstrapping and dependency management. It demonstrates how to use the archetype to generate a project with example UDFs (Reverse) and UDAFs (SummaryStats), and explains the core methods for UDAFs (initialize, aggregate, merge). It also covers building the project with Maven and converting it to a Gradle project.
  57. Building Shared State Microservices for Distributed Systems Using Kafka Streams | ConfluentThis post details how Imperva used Kafka Streams to build shared state microservices for distributed systems. It covers the challenges of traditional state management with single databases, the requirements for new shared state microservices, and how Kafka Streams met these requirements. The post explains the processing topology using Kafka Streams, including source, processor with persistent key-value store, and sink. It also details building a RESTful CRUD API on top of Kafka Streams for state retrieval and updates, distributing processing load across a cluster of microservices for scalability and availability, and implementing task scheduling using the processor context's schedule method. The benefits and challenges of moving from a single microservice to a cluster are also discussed.
  58. Deploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams | ConfluentThis post details the rationale for using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. It covers the implementation of KSQL UDFs using Java/Groovy with annotations like @UdfDescription and @Udf, and their testing with the Spock framework and data tables. The post also explains how to configure Gradle to build UDF artifacts, including the use of the com.github.johnrengelman.shadow plugin to create uber JARs that include all transitive dependencies for deployment to KSQL.
  59. Spring for Apache Kafka Deep Dive – Part 4: Continuous Delivery of Event Streaming Pipelines | ConfluentThis post details how Spring Cloud Data Flow can be used to build and continuously deploy event streaming pipelines using Apache Kafka. It covers common streaming topology patterns like named destinations, parallel pipelines, partitioned streams, and function composition. The post also demonstrates how to handle multiple input/output destinations and provides an example of a Kafka Streams application for processing user click events.
  60. Reliable, Fast Access to On-Chain Data Insights | ConfluentThis post details TokenAnalyst's implementation of a 'block confirmer' using Kafka Streams to handle blockchain reorganizations. It describes maintaining a fork tree in a Kafka Streams state store to process incoming blocks and propagate them only after a confirmation threshold is met, ensuring effective once semantics by discarding already registered and confirmed blocks. It also highlights the use of Kafka Streams' fault tolerance for rolling deploys and zero-downtime environments.
  61. Spring for Apache Kafka Deep Dive – Part 3: Apache Kafka and Spring Cloud Data Flow | ConfluentThis post introduces Spring Cloud Data Flow as a toolkit for designing, developing, and continuously delivering data pipelines, focusing on event streaming applications. It explains how Spring Cloud Data Flow orchestrates the deployment of these pipelines, often leveraging Apache Kafka as the underlying messaging middleware. The post details the Spring Cloud Data Flow ecosystem, including its dashboard, shell, DSL, and APIs, and how it integrates with Spring Cloud Skipper for application lifecycle management. It also covers developing event streaming applications using Spring Cloud Stream, outlining source, processor, and sink application types, and demonstrates creating a simple pipeline using the Stream DSL with an http source, a transform processor, and a log sink, all communicating via Kafka topics.
  62. Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations | ConfluentThis post details the challenges of developing, building, and deploying KSQL applications, introducing a custom Gradle plugin (gradle-confluent) to manage KSQL script dependencies and deployments. It demonstrates using Gradle to execute and test KSQL code, and to build and deploy KSQL applications continuously. The approach uses a simple alphanumeric file and directory structure for managing dependencies within and between KSQL pipelines, addressing the need to create objects before dependent ones and to manage pipelines as cohesive units. It also tackles the bottleneck of recreating KSQL streams and tables by providing a mechanism to handle existing objects, including terminating dependent queries before dropping and recreating them.
  63. Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation | ConfluentThis post details the integration of Oracle WMS Cloud with on-premises systems using Apache Kafka and Confluent Platform. It describes using Oracle GoldenGate for Big Data to deliver relational data change events to Kafka and MuleSoft to produce payloads from other APIs to Kafka. For data egress, the Kafka Connect JDBC Sink connector is used to deliver data back into legacy systems, and MuleSoft consumes topics for API calls. The post highlights the use of KSQL for data stream curation and Kafka Streams for final payload packaging, specifically addressing the need for nested messages for Oracle WMS Cloud and leveraging Kafka Streams' built-in state store.
  64. Optimizing Kafka Streams Applications | ConfluentThis post introduces the processor topology optimization framework in Kafka Streams starting from version 2.1.0. It details the evolution of topology generation, explaining the previous approach and highlighting inefficiencies in stateful operations. The post describes the refactoring that enables optimizations and introduces specific techniques available in Kafka 2.1. It also provides operational guidelines for enabling these optimizations and elaborates on how keyed stateful operators utilize data parallelism through repartition topics and sub-topology execution.
  65. Testing Event-Driven Systems | ConfluentThis post introduces the 'Test Machine,' a library developed by Funding Circle for robust end-to-end testing of event-driven systems built with Kafka. It addresses the challenges of flaky and slow integration tests by providing a structured approach to testing complex architectures involving multiple services and technologies (e.g., Ruby on Rails, Samza, Kafka Connect, Kafka Streams). The Test Machine defines tests as a sequence of 'write' and 'watch' commands, enabling portable tests that can run against various targets like local Kafka clusters, Confluent REST Proxy, or mock topology processors. It emphasizes 'tests as data' using EDN for input notation and a journal for consuming and validating output. The post also discusses strategies for improving test performance through parallelization and efficient resource management.
  66. KSQL: What’s New in 5.2 | ConfluentThis post details the introduction of the CASE statement in ksqlDB 5.2, enabling more flexible data manipulation. It demonstrates use cases for data cleansing, deriving new columns, bucketing data based on ranges, selectively masking data, generating values for missing attributes, conditional aggregates, and traffic routing for A/B testing. It also introduces the ability to log processing errors to a Kafka topic, with configuration options for broker list, topic name, and automatic stream creation.
  67. Kafka Streams' Take on Watermarks and Triggers | ConfluentIntroduces the `Suppress` operator for KTable in Kafka Streams, enabling controlled forwarding of updates. This operator allows for emitting only final results of windowed aggregations or rate-limiting updates, addressing use cases like alerting and integration with external systems that don't support continuous refinement. The post also clarifies the concept of window lifecycle and grace periods in Kafka Streams.
  68. Sysmon Security Event Processing in Real Time with KSQL and HELK | ConfluentThis post demonstrates the application of ksqlDB for real-time security event processing by joining Sysmon ProcessCreate and NetworkConnect events to detect lateral movement. It provides a concrete example of using ksqlDB for security analytics, highlighting the benefits of event-driven processing and real-time data enrichment compared to traditional SIEM approaches. The post also details the integration with the HELK platform and explains the fundamental concepts of streams, tables, and joins within ksqlDB in the context of security event correlation.
  69. Stream Processing Tutorial Part 2: Testing Your Streaming ApplicationThis post details testing methodologies for Kafka Streams applications, including unit testing with TopologyTestDriver and MockProcessorContext, integration testing with EmbeddedKafkaCluster, and schema compatibility testing with MockSchemaRegistryClient and the Schema Registry Maven plugin. It also introduces Confluent Cloud tools for more realistic deployment testing.
  70. Stream Processing Tutorial Part 1: Building Event Streaming ApplicationsThis post introduces a tutorial for developing real-time streaming applications using Kafka Streams, covering concepts like event-driven architectures, microservices, and the Kafka Streams API. It details a microservices ecosystem for order management, including services for validation, aggregation, and email notification, and highlights the use of ksqlDB for stream enrichment and fraud detection. The tutorial includes exercises on persisting events, event-driven applications, stream enrichment with joins (including CDC integration), filtering and branching, stateful operations with windowing, state stores, and enrichment with ksqlDB.
  71. Deep Dive into ksqlDB Deployment Options | ConfluentThis post provides a deep dive into ksqlDB deployment options, explaining its foundation on the Kafka Streams API. It details two primary deployment modes: headless (application mode) for production, emphasizing isolation and minimizing operational mistakes, and interactive mode for exploration and development via REST API, CLI, or Confluent Control Center. The post elaborates on how ksqlDB queries compile into Kafka Streams applications and leverage Kafka's consumer group protocol for scalability and fault tolerance, including rebalancing mechanisms. It also references a Kafka Summit talk on query processing.
  72. Troubleshooting KSQL – Part 2: What's Happening Under the Covers? | ConfluentThis post, the second in a series on KSQL troubleshooting, delves into the internal workings of KSQL queries. It explains how to monitor message throughput and failed messages using `DESCRIBE EXTENDED`, how to view running queries with `SHOW QUERIES`, and how to understand the execution plan and processing topology using `EXPLAIN`. It also highlights the use of Confluent Control Center for monitoring streams and JMX metrics for deeper inspection of KSQL, Kafka Streams, and Kafka producer/consumer behavior. The post provides practical examples and links to GitHub for code samples.
  73. Troubleshooting KSQL – Part 1: Why Isn't My KSQL Query Returning Data? | ConfluentThis post provides a foundational guide to troubleshooting ksqlDB queries that are not returning data. It breaks down common issues into five categories: no data in the source topic, no new data arriving, ksqlDB consuming from a later offset, data not matching predicates, and deserialization errors. For each issue, it offers diagnostic steps using ksqlDB commands (DESCRIBE EXTENDED, LIST PROPERTIES, SET) and external tools like kafkacat. It emphasizes the importance of verifying source topics, understanding offset management (setting 'auto.offset.reset' to 'earliest'), and systematically debugging predicates and data formats.
  74. Real-Time Presence Detection at Scale with Apache Kafka on AWS | ConfluentThis post details Zenreach's implementation of a real-time presence detection platform using Kafka Streams. It describes their use case of detecting walk-ins, enriching them with contact information, and maintaining walk-in counts. The post highlights the migration from Python scripts and Spark Streaming to Kafka Streams due to operational complexity and cost. It outlines the four services (Walk-In, DeviceToContact, Enriched Walk-In, Walk-In Count) and their roles in processing data. Key technical lessons learned include scaling Kafka Streams services by adjusting `num.stream.threads`, managing stateful services with persistent storage (EBS volumes) to ensure state availability during deployments, and improving fault tolerance by configuring `num.standby.replicas` to reduce recovery time after instance failures.
  75. Data Wrangling with Apache Kafka and KSQL | ConfluentThis post details data wrangling tasks using KSQL, including applying schemas, filtering, masking, restructuring, enriching, and unifying data streams. It demonstrates pulling data from REST sources using Kafka Connect, processing it with KSQL, and streaming it to Google Cloud Storage and BigQuery. The post also covers declaring schemas for JSON data and handling variations in data structures, such as arrays.
  76. How to Build a UDF and/or UDAF in ksqlDB 5.0 | ConfluentThis post introduces the official API for creating User-Defined Functions (UDFs) and User-Defined Aggregate Functions (UDAFs) in ksqlDB 5.0. It details the motivation for UDFs/UDAFs, the process of writing a Java class with specific annotations (@UdfDescription, @Udf), and the deployment steps involving building an uber JAR and placing it in the ksqlDB extension directory. An advanced example of a deep-learning UDF for anomaly detection is provided, showcasing how to embed external dependencies. Potential development and testing issues are also discussed.
  77. Kafka Streams in Action | ConfluentThis post is a foreword to a book about Kafka Streams, highlighting the paradigm shift towards event-driven architectures powered by Apache Kafka. It explains the drivers behind this shift, including increased digitization, transformation in data usage, and advancements in stream processing technology. The foreword contrasts Kafka's approach to stream processing (modeling it after REST applications, with streams as the primary entity) with traditional Big Data frameworks, emphasizing its simplicity and suitability for building event-driven applications. It positions Kafka Streams and KSQL as key enablers of this transformation, aiming to make stream processing accessible enough for natural integration into event-driven applications.
  78. We ❤️ syslogs: Real-time syslog Processing with Apache Kafka and KSQL—Part 1: FilteringThis post introduces the use of KSQL for real-time processing of syslog data ingested into Apache Kafka. It demonstrates how to declare a KSQL stream over the 'syslog' topic and provides an example of filtering syslog messages using KSQL's SQL-like syntax. The post highlights the integration of Kafka Connect with a syslog plugin for data ingestion and the persistence capabilities of Kafka for data utilization.
  79. KSQL in Action: Enriching CSV Events with Data from RDBMS into AWSThis post demonstrates how to enrich CSV-based order events in real-time by joining them with customer data from a MySQL RDBMS using ksqlDB. It details the setup of customer data in MySQL, including schema creation and population, and highlights the importance of a primary key for compacted topics to avoid CORRUPT_MESSAGE errors. The process involves using Kafka Connect to ingest data from MySQL and then using ksqlDB to perform the join and stream the enriched data to a datastore for analysis.
  80. KSQL January release: Streaming SQL for Apache KafkaThis release (KSQL 0.4) introduces several new features and improvements for ksqlDB. Key additions include an enhanced PRINT TOPIC command for data discovery, an improved SHOW TOPICS command to display consumer group information, and new aggregation functions TOPK and TOPKDISTINCT for advanced analytics. Operational improvements include the ability to print KSQL metrics from the command line and the availability of binary tarballs. Furthermore, the release highlights continued investment in distributed system tests for enhanced production readiness.
  81. KSQL December Release: Streaming SQL for Apache Kafka | ConfluentIntroduced support for Avro data format in ksqlDB, integrating with Confluent Schema Registry for automatic schema inference and management. Enabled real-time conversion between Avro, JSON, and delimited data formats. Enhanced join capabilities to work across different data formats. Added initial metrics and observability features with DESCRIBE EXTENDED and EXPLAIN statements. Improved ksqlDB server startup performance and resource utilization.
  82. November Update of KSQL Developer Preview AvailableThis post announces the November update to KSQL, a streaming SQL engine for Apache Kafka. It highlights the inclusion of KSQL as a component of the Confluent Platform. The update addresses community-raised issues and requests for the Developer Preview, with 369 pull requests and 50 closed issues. It also mentions diverse use cases companies are solving with KSQL, such as real-time ETL, data exploration, and driving automated ad-tech auctioning. The post emphasizes KSQL's open-source nature and encourages community involvement through feedback and contributions. Future plans include monthly releases and a beta program.
  83. Building a Microservices Ecosystem with Kafka Streams and KSQLThis post details building a microservices ecosystem using Kafka Streams and KSQL. It explains how to leverage Kafka Streams for stateful stream processing, including joining streams and tables, and managing consistency. It also introduces KSQL as an alternative for stateful stream processing in a separate process, allowing for SQL-like queries on event streams. The post provides a concrete example of building a simple email service using these technologies, highlighting the benefits of event-driven architectures for decoupling services and enabling parallelism.
  84. Taking KSQL for a Spin Using Real-time Device DataThis post details the process of setting up a real-time data pipeline using a gaming device as a data source, Apache Kafka for ingestion, and ksqlDB for real-time stream processing and analysis. It covers writing a custom Kafka producer using C# and the Confluent .NET Kafka Client to capture device inputs (steering, braking, acceleration) and send them as JSON messages to Kafka. The post then demonstrates how to create ksqlDB streams from the raw Kafka topic, filter and transform the data (e.g., converting raw axis values to degrees or percentages), and create derived streams for specific inputs like steering, accelerator, and brake. The final output is visualized in Grafana, showcasing the real-time dashboard. This post highlights the low-latency, high-throughput capabilities of Kafka and ksqlDB for IoT-like scenarios.
  85. Running Kafka Streams Applications in AWS | ConfluentThis post details Zalando's experience running Kafka Streams applications in production on AWS. It covers challenges and solutions related to AWS EBS I/O throttling, fine-tuning Kafka Streams configurations (e.g., max.poll.records, processing timeouts), and managing JVM heap and off-heap memory usage with RocksDB. It also introduces custom monitoring tools like Remora for consumer lag and discusses the importance of monitoring EBS burst balance, JVM heap, CPU, system memory, and disk usage.
  86. Ranking Websites in Real-time with Apache Kafka’s Streams APIThis post details how Apache Kafka's Streams API can be used to implement real-time ranking algorithms like HITS, processing web link data to identify influential domains. It covers the extraction of domain links, aggregation of these links using Kafka Streams' KTable, and the iterative calculation of Hub and Authority scores for real-time website ranking. The post highlights the flexibility of Kafka Streams for building scalable, real-time data processing applications.
  87. Getting Started Analyzing Twitter Data in Apache Kafka through KSQLThis post introduces ksqlDB for stream processing on Kafka, demonstrating how to ingest Twitter data using Kafka Connect, define schemas, and perform SQL-like queries. It covers continuous queries, derived streams, and aggregations, highlighting the distinction between streams and tables. The post also shows how to use KSQL to explore and transform streaming data, build aggregates, and create derived tables from aggregated data.
  88. Using Kafka Streams API for predictive budgetingThis post details how Pinterest uses the Kafka Streams API to build a predictive budgeting system for their ads infrastructure. The system aims to reduce ad overdelivery by calculating 'inflight spend' (cost of ads insertions not yet charged). Key technical contributions include: 1. Designing a system to handle tens of thousands of events per second with sub-10-second end-to-end delay and 100% uptime. 2. Choosing Kafka Streams for its millisecond delay guarantee and lightweight nature compared to Spark and Flink. 3. Implementing a plan involving an 'inflight spend' service that consumes ad insertion events, aggregates spend using Kafka Streams with a 10-second window store per adgroup, and fans out updates to thousands of consumer machines. 4. Optimizing performance by switching from hopping windows to tumbling windows, resulting in an 18x throughput improvement. 5. Employing delta encoding for adgroup IDs and lookup table encoding for spend data to achieve a 4x reduction in message size.
  89. Crossing the Streams – Joins in Apache Kafka | ConfluentThis post details the join semantics in Kafka Streams, specifically focusing on inner, left, and outer joins between KStreams and KTables. It explains the windowing mechanisms for stream-stream joins and the differences in behavior compared to SQL joins. The post also introduces GlobalKTables as a way to perform star joins and avoid data repartitioning. An example scenario involving ad views and clicks is used to illustrate the output of various join types.
  90. Introducing KSQL: Streaming SQL for Apache Kafka | ConfluentThis post introduces KSQL, a streaming SQL engine for Apache Kafka, lowering the entry bar for stream processing. It provides a simple, interactive SQL interface for continuous transformations on unbounded data streams, supporting aggregations, joins, windowing, and sessionization. KSQL uses Kafka's Streams API internally and introduces two core abstractions: STREAM (unbounded sequence of facts) and TABLE (evolving collection of facts). Use cases include real-time monitoring, analytics, security, anomaly detection, online data integration, and application development. The post details a demo for clickstream analytics and anomaly detection, and explains the internal architecture of KSQL servers and the REST API.
  91. Watermarks, Tables, Event Time, and the Dataflow Model | ConfluentThis post critiques the Google Dataflow model's use of watermarks and triggers for handling out-of-order data and proposes Kafka Streams' table-centric approach as a more general and less complex alternative. It introduces the concept of tables as streams of updates, applicable to both mutable data (like customer accounts) and windowed computations. The post details how Kafka Streams uses commit interval and record cache size as operational tuning parameters to manage output volume and update lag, contrasting this with the complexity of Dataflow's triggers. It also touches upon interactive queries for accessing the latest results on demand.
  92. Unifying Stream Processing and Interactive Queries in Apache KafkaIntroduces the 'Interactive Queries' feature for Kafka Streams. This feature allows direct querying of the state maintained by a stream processing application, treating it as an embedded, highly available, and fault-tolerant database. It contrasts this with traditional materialized views and the inefficiencies of Lambda architectures, advocating for a converged processing and storage layer within Kafka. The post details the motivation, an example of real-time risk management, and the underlying mechanism using embedded databases like RocksDB, highlighting trade-offs for using this feature versus external databases.
  93. Flink vs Kafka Streams: A Complete ComparisonThis post compares Apache Flink and Kafka Streams, highlighting their architectural differences in deployment, lifecycle management, coordination, data sources/sinks, and semantic guarantees. Flink is presented as a cluster framework for self-contained streaming computations, typically managed by data infrastructure teams. Kafka Streams is described as an embeddable library for standard Java applications, managed by application developers and integrated with existing tooling. The post emphasizes that Flink and Kafka Streams solve orthogonal problems, with Flink excelling in analytics-focused domains and Kafka Streams in building core applications and microservices.
  94. Data Reprocessing with the Streams API in Kafka: Resetting a Streams Application | ConfluentThis post introduces a new application reset tool for Kafka Streams, simplifying the process of reprocessing input data from scratch. It also provides a deep dive into the internal mechanisms of Kafka Streams, including internal topics, operator state, and offset commits, to explain how manual resets can be performed. The post highlights the importance of stopping all application instances before using the reset tool and warns about potential data inconsistencies if not used carefully.
  95. Secure Stream Processing with the Streams API in Kafka | ConfluentDetails the security features of the Kafka Streams API, focusing on client authentication, authorization, and data encryption (in-transit). Explains how these features integrate with Apache Kafka's security mechanisms and provides a code example for configuring SSL encryption and client authentication within a Streams API application's configuration.
  96. Distributed, Real-time Joins and Aggregations on User Activity Events using Kafka Streams | ConfluentThis post introduces a common stream processing use case: enriching an incoming stream of user click events with side data (user geo-region) and then computing aggregations (total clicks per region). It contrasts a naive approach of querying an external database, highlighting scalability, latency, and availability issues. The post then presents Kafka Streams as a solution, leveraging the stream-table duality and KTables backed by local state stores. It demonstrates a KStream-KTable join operation to achieve data locality and efficient lookups, effectively acting as a lightweight embedded database for real-time data enrichment and aggregation.