TechPrufer
Sign inGet started

Blogs›Confluent›Data Serialization Formats

Data Serialization Formats

Data Serialization Formats

4
posts
2015–2026

This release details how Confluent Cloud addresses digital sovereignty for real-time streaming through architectural guarantees rather than policy assurances. Key concepts include Bring Your Own Cloud (BYOC) and Bring Your Own Key (BYOK) for data plane control and encryption, treating schemas as sovereignty boundaries with integrated data contract enforcement, sensitivity/jurisdiction tagging, encryption directives (CSFLE/CSPE), and quality/policy rules. It also emphasizes protocol-level portability with open formats like Apache Kafka and Apache Flink, contrasting contractual exit plans with executable workflows. The post outlines the operational costs of architectural sovereignty, such as managing the data plane, upgrades, patching, capacity planning, key lifecycle management, and cross-cluster governance, and provides a checklist for evaluating vendor architectures and stream-level governance.

2026

Confluent's architectural approach to digital sovereignty for real-time streaming

5/15/2026

This post introduces Confluent's architectural approach to digital sovereignty for real-time streaming, emphasizing architectural guarantees ('We cannot') over policy assurances ('We will not'). It details the BYOC pattern with stateless agents in customer VPCs and messages landing in customer object storage, combined with BYOK for encryption. The post redefines the schema as the 'new sovereignty boundary,' integrating data contracts, sensitivity/jurisdiction tags, encryption directives (CSFLE/CSPE), and quality/policy rules. It also highlights protocol-level portability with open standards like Kafka and Flink, contrasting contractual exit plans with executable workflows. Finally, it discusses the operational costs of sovereignty and provides a checklist for architectural decisions.

2023

How Change Data Capture (CDC) Works

1/10/2023

This post details how Change Data Capture (CDC) works, converting database changes into events for analytics, operational use cases, and hydrating databases. It discusses the limitations of traditional databases and introduces CDC as a pattern to gain event-driven architecture advantages without ripping out existing systems. The post contrasts polling-based connectors with transaction log-based CDC, recommending the latter for accuracy and performance, and highlights Debezium as a popular framework. It also addresses the importance of denormalizing data for event streams and provides conceptual examples.

2015

Bottled Water: Real-time integration of PostgreSQL and Kafka | Confluent

4/23/2015

Introduces Bottled Water, an open-source tool for PostgreSQL to Kafka integration. Leverages PostgreSQL 9.4's logical decoding to capture row-level change events (inserts, updates, deletes) and coordinates them with a consistent snapshot. Encodes data in Avro format and streams it to Kafka, with each table becoming a topic and each row a message keyed by its primary key. Utilizes Kafka's log compaction feature to retain the latest message for each key indefinitely, effectively storing both the database snapshot and real-time changes within Kafka.

Why Avro for Kafka Data? | Confluent

2/25/2015

This post introduces Apache Avro as a recommended data format for Kafka, detailing its advantages over formats like JSON and XML. It explains Avro's data model, binary representation, and schema definition language. Key benefits discussed include efficiency, speed, language bindings, schema evolution, and its role in ensuring data quality and compatibility in large-scale streaming platforms. The post also argues for the necessity of schemas in event-driven architectures to maintain data integrity and facilitate data science.