Data Serialization Formats · Confluent Feature Trail

Jul 2026Tableflow: Turn Kafka Topics into Iceberg TablesIntroduces Tableflow, a managed feature in Confluent Cloud that materializes Kafka topics into Apache Iceberg or Delta Lake tables. It details the end-to-end pipeline: Kafka topic ingestion, Kora storage layer processing, Parquet conversion, table metadata generation, catalog publishing, and schema governance via Schema Registry. The post also provides a step-by-step tutorial for enabling Tableflow with Amazon S3 storage, including AWS IAM role configuration and policy setup, and discusses schema evolution handling based on Confluent Schema Registry compatibility rules.

May 2026Confluent's architectural approach to digital sovereignty for real-time streamingThis post introduces Confluent's architectural approach to digital sovereignty for real-time streaming, emphasizing architectural guarantees ('We cannot') over policy assurances ('We will not'). It details the BYOC pattern with stateless agents in customer VPCs and messages landing in customer object storage, combined with BYOK for encryption. The post redefines the schema as the 'new sovereignty boundary,' integrating data contracts, sensitivity/jurisdiction tags, encryption directives (CSFLE/CSPE), and quality/policy rules. It also highlights protocol-level portability with open standards like Kafka and Flink, contrasting contractual exit plans with executable workflows. Finally, it discusses the operational costs of sovereignty and provides a checklist for architectural decisions.

May 2026Feed your data lake with AI-ready tables for 30-50% lower costThis post introduces Confluent's Tableflow, a feature that directly converts Kafka topics into open table formats (Apache Iceberg, Delta Lake) for analytics engines. It details how Tableflow reuses Kafka segments from the Kora engine, converts them to Parquet, and handles schema evolution, type conversions, and CDC semantics. The post quantifies 30-50% cost savings by eliminating egress charges, reducing Kafka topic storage by avoiding triple replication, removing independent sink connectors and Kafka Connect task fees, and bundling lakehouse compute and maintenance into a per-topic-hour and per-GB processed fee. It also highlights architectural benefits such as vendor-neutrality, operational simplicity, near-real-time data readiness, and shift-left processing with Flink.

Mar 2026Why CISOs Are Building Open Security Lakes with IcebergThis post introduces the application of Confluent's data streaming and Tableflow capabilities to build 'Open Security Lakes' using Apache Iceberg. It details how Freight clusters and WarpStream can be used for cost-effective ingestion of high-volume security logs directly to object storage. It also highlights the integration with Apache Flink and SOC Prime for real-time threat detection using Sigma rules, and the use of Tableflow to create Iceberg tables from Kafka topics for forensic analysis and AI model training, effectively eliminating ETL pipelines for security data.

Mar 2026Why Batch ELT Breaks at Enterprise ScaleThis post argues that traditional batch ELT pipelines break at enterprise scale due to cascading failures from schema drift, resource inefficiency, complexity, and lack of visibility. It proposes a streaming-first data integration architecture using Kafka, connectors, Tableflow, Schema Registry, open table formats (Iceberg/Delta Lake), and various processing engines. The core contribution is highlighting Kafka's role as a durable, scalable, reusable data movement layer that enables an integration fabric, contrasting it with the limitations of point-to-point batch ELT.

Mar 2026Introducing Schema IDs in Kafka Headers for Smarter Data GovernanceIntroduces storing Schema Registry schema IDs in Kafka headers, enabling backward-compatible schematization of existing topics without payload changes. This allows for incremental adoption of Schema Registry, reduces data errors by enforcing schemas centrally, and powers AI/analytics by providing consistent, validated schemas. The post details producer and consumer behavior changes, the new GUID format for schema IDs in headers, and ecosystem support across Confluent clients and platforms.

Nov 2025How Superwall Scales with WarpStream and ClickHouse CloudThis post details Superwall's migration from a self-managed Kafka and ClickHouse OSS stack to WarpStream (Kafka-compatible streaming) and ClickHouse Cloud, connected by ClickPipes. It highlights the challenges of managing disk space and costs with EC2-based ClickHouse, the benefits of ClickHouse Cloud's separation of storage and compute, and the operational simplicity of WarpStream's BYOC model. The integration of ClickPipes is emphasized for its managed, reliable data ingestion from WarpStream to ClickHouse Cloud, including error handling for invalid records. The post quantifies the scale (100 MB/sec, 300+ TB data) and discusses the trade-offs and evolution of their data stack.

Oct 2025Confluent Tableflow + Databricks Unity Catalog | Real-Time Data to AI-Ready TablesThis post announces the general availability of Confluent Tableflow's support for Delta Lake tables integrated with Databricks Unity Catalog. It details the under-the-hood implementation, highlighting the use of Kora's tiered storage, direct reading from object storage bypassing brokers, conversion to Parquet, and leveraging the Delta Kernel for ACID transactions, efficient commits, and future Delta Lake feature adoption. New features include schema evolution for Delta tables (optional fields, type widening), upserts, multi-format control, and multiple catalog integrations per cluster.

Oct 2025Introducing Confluent Private CloudThis post announces the General Availability (GA) of Delta Lake and Unity Catalog integration for Tableflow, enabling direct materialization of Kafka topics into Delta Lake tables stored in customer-managed object storage (BYOS) and automatic registration in Databricks Unity Catalog. It also introduces upsert materialization for Change Data Capture (CDC) data, allowing for real-time updates and deletes to be reflected in analytics tables with low materialization lag. Additionally, it details Dead Letter Queue (DLQ) support for resilient error handling and Bring Your Own Key (BYOK) encryption for enhanced data security at rest. The post also highlights expanded multi-cloud support with Tableflow on Azure.

Oct 2025Better-Governed Data Lake Architectures with Confluent TableflowThis post introduces Confluent Tableflow as a solution to data governance challenges in data lakes, specifically focusing on data quality, error handling, and data discoverability. It advocates for a 'shift-left analytics' approach, bringing governance closer to the data source. Tableflow leverages Schema Registry for structured data representation and schema evolution, ensuring data trust. It also enhances catalog management by directly synchronizing metadata to multiple analytics catalogs (AWS Glue, Polaris, Databricks Unity), reducing the reliance on batch jobs and crawlers for catalog updates. The post highlights how this approach makes data more trustworthy, usable, and traceable for downstream analytics and AI agents.

Sep 2025Ensure Data Quality With Real-Time Validation and MonitoringIntroduces real-time data validation and monitoring as a continuous process, contrasting it with batch validation. Details how to achieve this using Kafka and Confluent by: 1. Schema enforcement with Schema Registry to ensure data structure. 2. Real-time business rule checks with Apache Flink or ksqlDB for logic like out-of-range values. 3. Monitoring integrations (e.g., Grafana, Datadog) for visibility into data ecosystem health. Provides use cases in financial services, retail/ecommerce, healthcare, and AI/ML, and outlines a step-by-step guide for implementation.

Jul 2025Zero-ETL vs Tableflow | Integrating Apache Kafka® and Zero-ETL via Confluent TableflowThis post clarifies the distinction between Confluent's Tableflow and the broader concept of zero-ETL. It explains how Tableflow materializes Kafka topics into open table formats (Iceberg/Delta Lake), offering reusability and loose coupling between producers and consumers, unlike the tight coupling often seen in zero-ETL implementations. The post contrasts Kafka's many-to-many connectivity with zero-ETL's one-to-many model, emphasizing Kafka's role as a shared, centralized gateway for data and Tableflow's ability to bridge operational and analytical estates by unifying logs and tables.

Jul 2025Unlocking Real-Time Analytics With Tableflow & AWS Glue CatalogThis post details the integration of Confluent's Tableflow with the AWS Glue Data Catalog. It explains how Tableflow materializes Kafka topics into Apache Iceberg tables stored in Amazon S3, and how these tables are then made discoverable in the AWS Glue Data Catalog. The post walks through the setup process, including provider integration for S3 and Glue, and demonstrates how AWS analytics services like Amazon Athena and Amazon Redshift Spectrum can query these Iceberg tables. It highlights the benefits of eliminating complex ETL pipelines and enabling real-time analytics from streaming data.

Jun 2025Build an AI Personalization Engine with Confluent & DatabricksThis post details the integration of Confluent Cloud with Databricks for building an AI personalization engine. It highlights the use of Confluent's Tableflow to convert Kafka topics into Delta Lake tables, enabling real-time data access for Databricks AI models. The architecture involves ingesting data from Oracle via CDC, processing with Flink, and then syncing to Databricks for AI-driven marketing campaigns. The quickstart guide outlines the steps for setting up the environment, ingesting mock data, processing with Flink, streaming to Delta Lake with Tableflow, and generating insights with Databricks Genie.

Jun 2025Delta Lake & Unity Catalog Now Integrate With TableflowThis post announces the integration of Confluent Tableflow with Databricks Unity Catalog, enabling the seamless materialization of Kafka topics into Delta Lake tables. Key technical contributions include: 1. Support for Delta Lake as a target table format in Tableflow. 2. Integration with Databricks Unity Catalog for centralized data discovery, access control, end-to-end lineage tracking, and data sharing. 3. Detailed architectural explanation of how Tableflow syncs Delta Lake tables to Unity Catalog, including registration of S3 storage as an External Location and configuration of Unity Catalog integration at the Kafka cluster level. 4. Discussion of downstream capabilities unlocked by this integration, such as leveraging Mosaic AI and BI tools.

Jun 2025Inside Tableflow GA: Real-Time Kafka to IcebergThis post announces the General Availability (GA) of Confluent Tableflow, a system that materializes Kafka topics into Apache Iceberg tables. It details the architecture, which leverages Kora's storage layer to convert Kafka segments into Parquet files and uses a metadata publishing service that integrates with Schema Registry to generate Iceberg metadata, handle schema mapping, evolution, and type conversions. Key capabilities highlighted include data conversion (Avro, JSON, Protobuf to Iceberg/Parquet), automatic schema evolution detection, catalog syncing with Snowflake Open Catalog, AWS Glue, Apache Polaris, and Unity Catalog, and automatic table maintenance like file compaction and snapshot expiration. The integration with Snowflake Open Catalog is demonstrated, showing how to configure Snowflake to connect to S3 storage where Tableflow writes Iceberg data, making it discoverable and queryable in Snowflake for analytics and AI/ML workloads via services like Cortex AI.

Apr 2025Achieve Data Efficiency By Shifting Governance and Processing LeftThis post introduces the concept of 'shift-left' integration for data efficiency, emphasizing the use of schemas for data consistency and quality, and stream processing (Apache Flink) for efficient data transformation. It highlights open table formats like Apache Iceberg for data reusability and consistency. Three strategies are proposed: implementing no-copy/zero-ETL solutions, shifting data governance left, and processing data at the source. The post argues that data efficiency is critical for organizational growth in the face of exponential data expansion, drawing parallels to the refactoring of monoliths into microservices for data.

Apr 2025Improving Organizational Efficiency and Trust with Data ProductsThis post introduces the concepts of data products and data contracts as a means to 'shift left' data quality and governance responsibilities to data producers (application engineers). It details how data contracts, published to Schema Registry, define the metadata and agreements for data products. These data products can then be exposed as Kafka topics or tabular formats via Tableflow. The post emphasizes the role of the data platform team in providing tooling to support this process, such as CD tasks for publishing data contracts.

Mar 2025Confluent Cloud Q1 ’25 introduces Tableflow, Freight clusters, and Flink AI enhancementsAnnounces the general availability of Tableflow, which represents Kafka topics and schemas as Apache Iceberg or Delta Lake tables. Highlights simplified data maintenance, reuse of tables, and integration with various catalogs (AWS Glue, Databricks Unity, Snowflake Open Catalog) and analytics engines (Athena, EMR, Redshift, SageMaker, Databricks, Snowflake, Dremio, Imply, Onehouse, Starburst, Spark, Flink, Trino). Introduces early access for Delta Lake support.

Mar 2025Tableflow is now generally availableTableflow is now generally available on AWS, with Azure and GCP support coming soon. This release introduces generally available Iceberg tables and Open Preview for Delta Lake tables. The post details the challenges of streaming Kafka data into data lakes (fragile pipelines, bad data, duplication, type conversions, slow/expensive processing, small file issues, schema evolution) and how Tableflow solves them by representing Kafka topics as Iceberg or Delta Lake tables. It explains the behind-the-scenes operations involving Kora Storage Layer for converting Kafka segments to Parquet files, using Schema Registry for metadata, handling schema evolution, catalog syncing (AWS Glue, Snowflake Open Catalog, Apache Polaris, Unity Catalog), and table maintenance (compaction, snapshot expiration). The GA release also highlights seamless interoperability with partner catalogs and analytics engines like AWS, Databricks, and Snowflake.

Feb 2025How Confluent and Databricks Are Unlocking Real-Time AIThis post details the expanded partnership between Confluent and Databricks, focusing on a Delta Lake-first integration. It describes how Confluent's Tableflow converts Kafka event streams into Delta Lake tables, which are then integrated with Databricks' Unity Catalog. This enables bidirectional data flow: operational data from Kafka can be structured into Delta Lake for AI/analytics (including training models with Databricks' Mosaic AI on streaming data), and AI-generated insights from Databricks can flow back into Kafka via updated Delta tables. The integration emphasizes unified governance through Confluent Stream Governance and Databricks Unity Catalog, ensuring data trust, lineage, and compliance for AI applications. Key technical aspects include the seamless structuring of event streams, feature engineering, ML model training on streaming data, and the automated flow of insights back into operational systems for real-time decision-making.

Jan 2025Allium and Confluent: How to Build a Foundational Data Platform for BlockchainThis post details how Allium uses Confluent Cloud to ingest and process blockchain data. They extract data from blockchain nodes, send it to Confluent Cloud (either directly or after initial transformation on GCP Dataflow), and then use Confluent's managed connectors to send it to Snowflake or an observability service. The article highlights the use of Confluent's platform for scalability and real-time data freshness. It also discusses future plans to leverage Confluent Cloud for Apache Flink and Tableflow to shift data processing and governance 'left', closer to the data source, reducing reliance on downstream data warehouses for initial cleaning and transformation.

Dec 2024Materialize Your Topics with Tableflow and Amazon SageMaker LakehouseThis post details the integration of Confluent Tableflow with Amazon SageMaker Lakehouse. It explains how Tableflow materializes Kafka topics into Apache Iceberg tables stored in S3, using AWS Glue Data Catalog as the Iceberg catalog. The post outlines the process of configuring Tableflow for Iceberg integration and provider integration within Confluent Cloud, and demonstrates how AWS analytics services (Athena, Redshift, EMR) and Apache Spark/Flink can consume these materialized tables. It also touches upon using Confluent Cloud for Apache Flink for real-time stream processing before materialization.

Dec 2024Securely Streaming Data from Confluent into Amazon Redshift with mTLSThis post details the integration of Confluent Cloud with Amazon Redshift for streaming data using mTLS authentication. It covers setting up a Confluent Cloud dedicated cluster, generating a private CA with AWS Private Certificate Authority, configuring mTLS on Confluent Cloud, setting up an AWS client certificate, creating an IAM policy and role for Redshift to access certificates, and finally configuring Amazon Redshift to consume from Confluent Cloud via mTLS. The post highlights the use of materialized views in Redshift for optimized data representation and real-time data ingestion from Kafka topics.

Oct 2024Headless Data Architecture, Part 2: How to Build ItThis post introduces the concept of a headless data architecture, advocating for a shift-left approach where data cleanup, structuring, and schematization are pushed upstream into source systems or data products. It emphasizes the creation of data products composed of both Kafka streams and Iceberg tables, enabling low-latency operational use cases and high-latency analytical workloads. The post details the problems with traditional multi-hop and medallion architectures (latency, cost, brittleness, data sprawl) and outlines a modular, incremental approach to building a headless architecture by creating source-to-stream workflows and materializing streams into Iceberg tables. It highlights the benefits of managing stream and table evolution together and integrating data validations into source application deployment pipelines.

Oct 2024Headless Data Architecture, Explained for DevelopersThis post introduces and explains the concept of a 'headless data architecture,' which decouples data storage, management, and access from processing services. It positions Apache Kafka as the foundational streaming layer and Apache Iceberg as the key technology for managing tables independently of query engines. The post details the components of Iceberg (table storage/optimization, catalog, transactions, time travel) and explains how it enables flexible integration with various processing 'heads' (e.g., Trino, Flink, Spark). It outlines the benefits of this architecture, such as reduced data duplication, cost savings, and engine choice flexibility, and differentiates it from traditional data lake architectures. The post also touches upon how Kafka Connectors and services like Tableflow can be used to materialize Kafka streams into Iceberg tables, forming the core of this headless data layer.

Oct 2024Preventing and Fixing Bad Data in Event Streams, Part 2This post introduces strategies for handling bad data in event streams, focusing on event design patterns (State vs. Delta events) and leveraging Kafka's log compaction for data correction and deletion. It details how to fix data at the source (databases, topics) and handle external data sources like FTP directories. State events are highlighted for their ability to be corrected at the source and their compatibility with log compaction for eventual data correctness and deletion via tombstones. The post also discusses the cost-effectiveness of storing state data in modern cloud environments.

Oct 2024Preventing and Fixing Bad Data in Event Streams, Part 1This post introduces strategies for preventing and fixing bad data in event streams, contrasting batch processing approaches with the challenges and solutions specific to Kafka. It highlights the reactive nature of data engineering, the reliance on reprocessing in batch systems, and the need for different strategies in streaming environments, particularly concerning data quality and immutability. The post discusses how bad data can arise from source system changes, the difficulties in rectifying immutable logs, and the limitations of batch processing techniques like 'cut out the bad, put in the good' and incremental processing for streaming data. It sets the stage for discussing specific streaming solutions in a subsequent post.

Sep 2024How Developers Can Use Generative AI to Improve Data QualityIntroduces the use of generative AI to automate the creation of metadata and schemas for data governance. This involves presenting generative AI models with data examples or production code to induce schemas and generate machine-readable and human-readable metadata. The post emphasizes the need for human review to catch AI errors and highlights the benefits of integrating this process with a data streaming platform for real-time metadata application and data product creation.

Aug 2024BigQuery Continuous Queries & Confluent Yield Real-Time AnalyticsThis post details the integration of BigQuery continuous queries with Confluent Cloud, specifically using the Google Cloud Pub/Sub Source Connector. It outlines the architecture for streaming BigQuery continuous query results in real-time to Confluent Cloud. The setup involves configuring BigQuery continuous queries to export to Pub/Sub and then using the Confluent Cloud Pub/Sub Source Connector to ingest these messages into Kafka topics. The post provides a step-by-step guide for setting up this integration, including service account configuration, BigQuery slot reservations, Pub/Sub topic creation, and Confluent Cloud connector setup. It highlights how this enables real-time analytics, personalization, and anomaly detection by combining BigQuery's batch processing power with Confluent's real-time streaming capabilities.

Mar 2024Introducing Tableflow: Unifying Streaming and AnalyticsIntroduces Tableflow, a new capability that simplifies the process of converting Kafka topics and schemas into Apache Iceberg or Delta Lake tables. Tableflow utilizes Confluent's Kora Storage Layer to write Kafka segments to Parquet files and a metadata materializer that integrates with Schema Registry to generate Iceberg metadata and Delta transaction logs. It handles schema mapping, evolution, and type conversions, and automatically compacts small files. Initial integration uses the Iceberg REST catalog, with future support for customer-owned object storage and additional catalog services. The post also demonstrates how Apache Flink Actions can be used with Tableflow for real-time data masking and transformation before landing data in the target table.

Mar 2024Real-Time Data Streaming for Smart WarehousesThis post details the implementation of real-time data streaming for smart warehouses, integrating IoT, robotics, and machine learning. It highlights challenges in traditional warehousing and how Confluent's platform, including Kafka, Flink, and connectors, enables real-time data flow for inventory management, automation, and predictive maintenance. The architecture involves on-prem and cloud clusters, Cluster Linking, and stream processing with Flink SQL for alerts and data products.

Jan 2024Streamline Data Ingestion with Google BigQuery Sink V2 Connector on ConfluentIntroduces the new fully managed BigQuery Sink V2 Connector, an upgrade to the legacy connector. Key enhancements include leveraging the Google Storage Write API for improved efficiency and cost-effectiveness (up to 50% reduction in streaming data ingestion), OAuth 2.0 support for enhanced security, an internal thread pool for parallel streaming and greater scalability, support for multiple data formats (Avro, JSON Schema, Protobuf, JSON), time-based table partitioning, and invalid record routing to a DLQ. Provides migration recommendations from the legacy connector.

Nov 2023Core Banking ModernizationThis post details the modernization of core banking systems using a streaming data platform. It highlights the challenges of legacy mainframe systems (COBOL, skills shortage, cost) and proposes an incremental modernization approach. Key technical components discussed include the MQ on Z connector for mainframe unlock without MSU costs, Oracle CDC connector for unlocking Oracle data without GoldenGate licensing costs, Apache Flink for real-time data enrichment, and MongoDB Atlas as a destination. Architectural details and considerations for each component are provided, emphasizing cost savings and improved real-time capabilities.

May 2023Streaming Data Quality, Integrity, and Custom Connectors with Kora and ConfluentThis post introduces Data Quality Rules within Confluent's Stream Governance suite. These rules allow for domain validation using CEL predicates and event-condition-action rules to enforce data integrity and compatibility. Complex schema migration rules are also introduced to simplify schema evolution. The post also announces the general availability of Custom Connectors, enabling users to upload and run their own Kafka Connect plugins on Confluent Cloud without managing the underlying infrastructure. It details the capabilities of custom connectors, including support for SMTs and monitoring via logs and metrics. Stream Sharing is introduced as a new feature for securely sharing real-time data streams across organizations. The Kora Engine is highlighted for its cloud-native architecture and performance enhancements.

Mar 2023Data Contracts Are More Than Just APIsThis post discusses the concept of data contracts, drawing parallels to API specifications and schema evolution practices in software engineering. It highlights that data contracts go beyond just APIs, representing an agreement between data producers (software engineers) and data consumers (data teams) to prevent upstream changes from breaking downstream dependencies. The post emphasizes the need for collaboration between software and data teams to establish these contracts, moving away from treating database schemas as non-consensual APIs. It advocates for loose coupling through abstraction, similar to how APIs and event streams are used in software development, to ensure data trustworthiness and enable data teams to deliver insights effectively.

Jan 2023Data Governance for Real-Time Data StreamsThis post introduces Confluent Cloud's Stream Governance features, specifically Schema Registry, Stream Catalog, and Stream Lineage, as solutions to challenges in managing data streaming applications at scale. Schema Registry is highlighted for managing schema evolution and ensuring backward/forward compatibility through validation rules. Stream Catalog is presented as a tool for discoverability, allowing users to find and tag data streams with metadata, and Stream Lineage is introduced for real-time visualization of data flow and evolution.

Jan 2023How Change Data Capture (CDC) WorksThis post details how Change Data Capture (CDC) works, converting database changes into events for analytics, operational use cases, and hydrating databases. It discusses the limitations of traditional databases and introduces CDC as a pattern to gain event-driven architecture advantages without ripping out existing systems. The post contrasts polling-based connectors with transaction log-based CDC, recommending the latter for accuracy and performance, and highlights Debezium as a popular framework. It also addresses the importance of denormalizing data for event streams and provides conceptual examples.

Jun 2022How to Keep Bad Data Out of Apache Kafka with Stream Quality | Confluent Streaming Data Quality - Keep Bad Data Out of Apache KafkaThis post details how Confluent's Stream Quality, comprising Schema Registry and Schema Validation, helps prevent bad data in Apache Kafka. It explains the importance of data contracts and schema evolution, recommending Avro, Protocol Buffers, and JSON as serialization formats. It outlines the process of registering schemas, producing/consuming records with schema IDs, and evolving schemas with backward compatibility. The post also introduces schema linking for sharing schemas across environments and uses a StreamWearables e-commerce example to illustrate the consequences of poor data quality without proper schema management.

Apr 2022Accelerate Cloud Database Modernizations and Migrations with Confluent | Confluent Migrate Data to Any Cloud Database with ConfluentThis post details how Confluent's data streaming platform can accelerate cloud database modernization and migrations. It highlights the challenges of traditional database migrations, such as downtime and batch processing, and positions Confluent as a solution for real-time data movement and ETL pipelines into any cloud database. Key features discussed include linking on-prem and cloud systems, using ksqlDB for stream processing, leveraging Change Data Capture (CDC) connectors (for Oracle, SQL Server, MySQL, PostgreSQL, Salesforce), and the benefits of fully managed services. Cluster Linking is mentioned for bridging on-premises Kafka to Confluent Cloud. The post also references Forrester's findings on TCO savings with Confluent Cloud and showcases specific cloud provider integrations (AWS Aurora, Azure Cosmos DB, Google Cloud SQL) with solution briefs and marketplace links. The core technical contribution is the architectural pattern of using Confluent for real-time data streaming to facilitate seamless and cost-efficient database modernization and migration, emphasizing CDC for capturing changes and ksqlDB for in-flight processing.

Dec 2021Unlock DB2 and Mainframe Data with tcVISION and ConfluentThis post details the integration of tcVISION with Confluent Cloud for real-time replication of mainframe data (Db2, VSAM, IMS/DB, etc.) to Kafka. It covers setting up Confluent Cloud, creating a Kafka topic, and configuring tcVISION to capture change data capture (CDC) events from mainframe sources and publish them as JSON to Confluent Cloud. The post highlights tcVISION's ability to handle data transformations and mainframe resource optimization.

Nov 2021Easily Copy and Migrate Data with Schema LinkingIntroduces Schema Linking, a new feature for managing and copying schemas between Schema Registry clusters. Schema Linking adds two new concepts: schema contexts (independent scopes within Schema Registry) and schema exporters (components that continuously export schemas between clusters or contexts). Schema Linking can be used independently of Cluster Linking for schema aggregation, backup, staging, and migration, or to complement Cluster Linking for syncing both schemas and topic data. It replaces and extends Confluent Replicator functionality, enabling bidirectional schema replication between Confluent Cloud clusters and between Confluent Platform clusters.

Feb 2021Keys Unlocked - ksqlDB 0.15 Supports Even More Data TypesThis post introduces significant enhancements to ksqlDB's handling of message keys, expanding support for various serialization formats (JSON, AVRO, PROTOBUF, DELIMITED, KAFKA, JSON_SR) and structured data types (ARRAY, STRUCT, nested combinations). It also enables composite keys with multiple columns and introduces multi-expression PARTITION BY and GROUP BY clauses. These changes allow for more expressive and flexible data modeling within ksqlDB, directly impacting how data is structured and processed in Kafka topics.

Feb 2021Announcing ksqlDB 0.15Introduced expanded support for message key columns in ksqlDB, including structured data types (ARRAY, STRUCT) and multiple key/primary key columns for PARTITION BY and GROUP BY clauses. Simplified persistent query termination by allowing DROP commands to automatically terminate associated queries.

Feb 2021Build Streaming Data Pipelines with Confluent, Databricks, and AzureThis post demonstrates a practical implementation of streaming data pipelines using Confluent Cloud, Azure Databricks, and Azure Data Lake Storage. It details the steps for configuring Confluent Cloud (Kafka cluster, Schema Registry, Datagen Source connector) and Azure Databricks to consume AVRO data from Kafka, parse it, and store it in Delta Lake format. Key technical aspects include setting up Kafka API keys, Schema Registry API keys, creating a Kafka topic, configuring the Datagen Source connector for clickstream data, and writing Spark Structured Streaming code in Python to read from Confluent Cloud, parse AVRO messages using the Schema Registry client, and write to ADLS Gen2.

Dec 2020ksqlDB 0.14.0 - Newest Features and UpdatesIntroduced multiple key lookups in ksqlDB pull queries using the IN predicate. Added incremental schema alteration for streams and tables created with DDL statements via the ALTER syntax, allowing addition of columns without specifying the full schema. Implemented SQL variable substitution for dynamic configuration of SQL statements in CLI sessions. Added support for DELIMITED and JSON key formats, in addition to the default KAFKA format, with plans for Schema Registry integrated key formats in a future release.

Nov 2020Analysing historical and live data with ksqlDB and Elastic CloudThis post details techniques for transforming raw CSV data from a Kafka topic into a structured format suitable for analytics. It demonstrates using ksqlDB to apply a schema, set a partitioning key (car park name), add a source field for lineage, derive and store a timestamp using STRINGTOTIMESTAMP, create a nested struct for location data (latitude/longitude), and create a calculated field. The post highlights the benefits of using Confluent Schema Registry for schema management and Protobuf for serialization, and shows how to process historical data by setting the offset to 'earliest'. It also illustrates the integration with Elastic Cloud (Elasticsearch and Kibana) for real-time dashboarding.

Nov 2020Self-Describing Events Using Spring Kafka and Confluent Schema RegistryThis post introduces a pattern for handling self-describing events using Spring Kafka and Confluent Schema Registry with Protobuf. It details how to configure KafkaProtobufDeserializer to deserialize binary payloads into DynamicMessage objects at runtime, leveraging schemas from Schema Registry. The post demonstrates a generic JSON converter using JsonFormat.Printer and explores generic data enhancement techniques, such as converting timestamps to Unix epoch time or preserving proto format, by traversing the DynamicMessage object as a tree structure. This enables a single, reusable processor to handle any compliant event schema without code redeployments.

Nov 2020Spring for Apache Kafka and Protobuf Part 1: Event Data ModelingThis post introduces three approaches to event data modeling for enterprise event-driven platforms: 'bare letter', 'deep envelope', and 'shallow envelope'. It details the advantages and disadvantages of each approach, focusing on aspects like pre-release compatibility checks, maintainability of models and code, and organizational impact. The post uses Protobuf and Spring for Apache Kafka as examples to illustrate these concepts, highlighting the importance of standardized event structures for effective data processing and communication.

Oct 2020ksqlDB Meets Java: An IoT-Inspired Demo of the ksqlDB Java ClientIntroduces the ksqlDB Java client, enabling programmatic interaction with ksqlDB servers from within applications. Demonstrates its use for adding and removing sensor types in an IoT scenario by programmatically creating ksqlDB streams and materialized views. Highlights support for push and pull queries, stream/table creation/deletion, and admin operations. Showcases asynchronous programming with CompletableFutures for non-blocking operations.

Aug 2020The Curious Incident of the State Store in Recovery in ksqlDBThis post details a specific bug in ksqlDB's state store recovery mechanism related to Avro serialization and Schema Registry integration. When a ksqlDB node was relocated, a cluster was unable to restart due to a deserialization exception. The root cause was identified as an optimization in Kafka Streams that, under certain conditions, incorrectly used the changelog topic name for serialization, leading to the registration of a "phantom" subject in Schema Registry. During recovery, data from the changelog topic was directly inserted into the state store using a schema ID that was not registered under the expected subject. The post describes the immediate fix (registering the schema ID under the changelog subject) and a more robust patch that attempts deserialization under both the original and phantom subjects to improve future recovery scenarios. It also highlights the need for ksqlDB to manage its internal schemas more effectively to prevent future issues.

Jul 2020SQL Changes and Key Columns in ksqlDB 0.10 ExplainedIntroduced named key columns in ksqlDB 0.10, replacing the implicit ROWKEY and the WITH(key='ID') syntax. This allows users to directly name their primary and key columns, simplifying data modeling and eliminating the need to duplicate Kafka record key data into the record value. This change aims to reduce confusion and improve the semantic clarity of data modeling in ksqlDB.

Jul 2020Multiple Event Types in the Same Kafka Topic - RevisitedThis post revisits the topic of multiple event types in the same Kafka topic, presenting schema references as an alternative to Avro unions or newer subject name strategies (RecordNameStrategy, TopicRecordNameStrategy). It details how schema references work with Avro, JSON Schema, and Protobuf, including necessary serializer configurations (auto.register.schemas=false, use.latest.version=true) for Avro and JSON Schema when using unions. It also highlights the automatic registration capabilities for Protobuf when schemas are wrapped in a message, and the manual registration requirements for Avro and JSON Schema.

Apr 2020Confluent Platform 5.5 - Enhanced Event Streaming for All DevelopersConfluent Platform 5.5 adds support for Protobuf and JSON schemas to Confluent Schema Registry, enabling schema validation and compatibility rules for these formats across the platform. It also introduces customizable plug-in schemas for other formats and enhances Schema Validation to support Protobuf and JSON at the broker level.

Mar 2020Build a Cloud ETL Pipeline - Multi-Cloud Integration with Confluent CloudThis post demonstrates building a multi-cloud ETL pipeline using Confluent Cloud, specifically for log ingestion. It details the extraction of logs from Amazon Kinesis into a Confluent Cloud Kafka topic using a KinesisSource connector. The transformation step involves using ksqlDB to count log events and sum durations of successful events, with the output stored in Avro format. Finally, it outlines loading the transformed data into cloud storage (GCS, Azure Blob Storage, S3) using sink connectors. The post highlights the use of Confluent Cloud CLI for automation and management of Kafka clusters, connectors, Schema Registry, and ksqlDB.

Dec 2019Transferring Avro Data Across Schema Registries with Kafka ConnectIntroduces a custom Kafka Connect Single Message Transform (SMT) to transfer Avro schemas across independent Schema Registry clusters without overwriting existing schemas. The SMT registers schemas from a source Schema Registry to a destination Schema Registry and updates the schema ID within replicated Kafka records. This addresses the challenge of schema ID mismatches when replicating topics between environments with separate Schema Registries. The post also discusses potential ordering issues during schema registration and provides a demo architecture and order of operations.

Oct 2019Get Started with Rust and Apache KafkaThis post details the integration of Rust with Apache Kafka, focusing on the use of Confluent Schema Registry and Avro serialization. It describes the challenges of translating Avro data between Rust and Java clients, the use of Rust libraries like `kafka` and `rdkafka`, and the implementation of a Rust command handler for an event sourcing bank simulation. The post highlights the use of `TopicRecordNameStrategy` for managing multiple schemas within a single Kafka topic and the development of custom code to handle Avro serialization/deserialization in Rust.

Sep 2019Schema Validation with Confluent Platform 5.4This post introduces Schema Validation for Confluent Platform 5.4, enabling brokers to validate message schemas against Confluent Schema Registry. It details the configuration parameters (`confluent.schema.registry.url`, `confluent.key.schema.validation`, `confluent.value.schema.validation`) and the integration with the producer protocol to reject invalid messages. The post also explains the importance of centralized data governance and the role of schema validation in achieving it.

Feb 2019Improving Stream Data Quality with Protobuf Schema Validation | ConfluentDeliveroo details their implementation of Protobuf schema validation for their 'Franz' inter-service messaging framework built on Kafka. They chose Protobuf over Thrift and Avro for its cross-language support and binary serialization. The post outlines Protobuf's built-in forwards and backwards compatibility features and details specific rules enforced via unit tests on schema evolution: field numbers must not be amended, field types must not be amended, removed fields must be added to a reserved statement, and field names must not be amended. They also describe using Protobuf custom options to enforce topic ownership contracts ('Producer X owns Topic Y with Message Format Z').

Nov 2018Kafka Connect Deep Dive – Converters and Serialization Explained | ConfluentThis post details the internal workings of Kafka Connect converters and their role in serialization and deserialization, covering JSON, Avro, and Protobuf formats, common errors, and troubleshooting tips. It explains how Kafka messages are just bytes and the importance of choosing a serialization format based on schema support, ecosystem compatibility, message size, and language support. It also covers configuring converters at the worker and connector level, and the nuances of using JSON with and without schemas.

Aug 2018Decoupling Systems with Apache Kafka, Schema Registry and Avro | ConfluentThis post details the practical implementation of Confluent Schema Registry and Apache Avro with .NET, demonstrating how to decouple systems integrated via Kafka. It covers using both GenericRecord for dynamic data handling and avrogen to generate specific C# classes for Avro schemas. The post explains schema evolution rules, the role of Schema Registry in managing schema IDs and subjects, and the configuration of compatibility modes (BACKWARD, FULL). It provides code examples for producing messages using both GenericRecord and generated classes, and illustrates how to evolve a schema by adding a new field with a default value.

Sep 2015Yes, Virginia, You Really Do Need a Schema Registry | ConfluentThis post introduces the concept and benefits of a Schema Registry for managing data schemas in distributed systems, particularly in the context of Hadoop and stream processing. It highlights challenges with schema management in systems like Hadoop and emphasizes the importance of a central registry for data discovery, resilient data pipelines, safe schema evolution, storage efficiency, and data policy enforcement. It also introduces Confluent's Schema Registry as a first-class component of the Confluent Platform, detailing its REST API, multi-version schema support with compatibility validation, serializers, and formatters.

May 2015How I Learned to Stop Worrying and Love the Schema | ConfluentThis post argues for the necessity of schemas in data management, drawing parallels to language standardization. It illustrates the problems arising from schema-less data evolution with the 'temp-o-meter' example, highlighting issues with format changes, data type inconsistencies, and tight coupling between producers and consumers. The author advocates for schemas as a single source of truth and an automatically enforceable contract between data producers and consumers, promoting the DRY principle for data definitions.

Apr 2015Bottled Water: Real-time integration of PostgreSQL and Kafka | ConfluentThis post introduces Bottled Water, an open-source tool for PostgreSQL that leverages logical decoding to transform database changes into a stream of structured Kafka events. It addresses the challenges of real-time data integration by providing a consistent snapshot and a real-time stream of changes, enabling downstream systems to stay up-to-date with minimal latency. The tool encodes data in Avro format and sends it to Kafka, where each table becomes a topic and each row becomes a message. It utilizes Kafka's log compaction feature to manage data, retaining the latest message for a given key indefinitely and garbage-collecting older messages for deleted or updated rows. This allows Kafka to serve as both a snapshot and a real-time message store.

Feb 2015Why Avro for Kafka Data? | ConfluentThis post argues for the adoption of Apache Avro as the preferred data serialization format for Kafka, highlighting its advantages over formats like JSON. Key benefits discussed include Avro's compact binary representation, high performance, broad language support, rich schema definition language, and superior compatibility for evolving data schemas over time. It emphasizes how schemas protect against malformed data, provide clarity and semantics, and enable smoother data evolution, especially when integrating with systems like Hadoop. The post details the structure of Avro schemas and their role in ensuring data quality and usability at scale.

What this trail covers