Schema Registry Best Practices

3/21/2024

Introduces best practices for Confluent Schema Registry, covering schema IDs, subjects, versions, and data contracts. Recommends pre-registering schemas, enabling schema normalization, and using topic-based subject naming strategies with union types and schema references. Discusses subject aliases and compatibility levels, advocating for backward compatibility.

10/18/2023

This post introduces and demonstrates the implementation of data contracts within Confluent Schema Registry. It covers defining an initial Avro schema, enhancing it with business metadata (owner, SLOs), adding data quality rules using Google Common Expression Language (CEL) for validation (e.g., totalPriceCents > 0), and specifying custom rule executors and actions. An example of a custom `EmailAction` is provided to check timeliness SLOs and send alerts. The post details how to register these enhancements as new schema versions in Schema Registry and provides command-line examples for producer/consumer interaction and registration.

3/30/2023

This post extends the understanding of data contracts beyond just APIs, emphasizing their role in fostering collaboration between software and data teams. It highlights how established API practices like versioning, schema evolution, and compatibility levels can be applied to data streaming systems using Confluent Schema Registry. The post advocates for treating data consumers (data teams) as first-class citizens, necessitating negotiated schemas and avoiding direct database access. It reinforces the value of Schema Registry in enabling loose coupling and abstraction for data architectures, ultimately driving better business outcomes through improved collaboration.

3/8/2023

This post introduces the concept of data products and how to manage their lifecycle using descriptor files and a self-serve platform, with a track and trace example using Confluent Cloud components like Kafka topics, Schema Registry, and ksqlDB. It elaborates on the principles of data mesh and event-driven architectures, emphasizing data as a product and automation through self-serve platforms.

1/24/2023

This post introduces Confluent Cloud's Stream Governance capabilities, focusing on Schema Registry for managing schema evolution and compatibility rules, Stream Catalog for data discoverability and metadata tagging, and Stream Lineage for visualizing data flow. It highlights how these tools address challenges like data silos and stale documentation in data streaming architectures.

6/29/2022

This post dives deeper into Schema Registry's functionality, detailing its components (REST service, serializers/deserializers), integration points with Kafka clients, Connect, ksqlDB, and Confluent GUI. It provides a step-by-step guide on using Schema Registry, including choosing serialization formats, registering schemas, producing/consuming records, and evolving schemas with compatibility types. It also introduces schema linking for sharing schemas across environments and uses the StreamWearables example to demonstrate the practical implications of schema management.

11/10/2021

Introduces Schema Linking, a new feature for Confluent Cloud and Confluent Platform 7.0, which allows for the replication of schemas between Schema Registry clusters. It introduces two new concepts: schema contexts, which provide independent scopes within Schema Registry, and schema exporters, which continuously export schemas from one Schema Registry cluster to another or across contexts within the same cluster. Schema Linking can be used independently or to complement Cluster Linking for data synchronization.

12/3/2020

This post details the implementation of security features for Confluent Schema Registry, including HTTPS endpoints, mTLS, Schema Validation, and RBAC. It provides configuration examples for Schema Registry and Kafka clients, and explains the authentication and authorization flow with Metadata Service (MDS) and LDAP. It also outlines the steps for enabling schema validation on a per-topic basis and the role of RBAC for fine-grained access control.

11/12/2020

This post introduces a generic processor for handling diverse Protobuf event schemas using Spring Kafka and Confluent Schema Registry. It details how to use `DynamicMessage` to parse any Protobuf record at runtime, converting it to JSON for reporting systems. The post also presents a mechanism for enhancing timestamp representations within the JSON output (e.g., Unix epoch, preserved proto format) by traversing the `DynamicMessage` object as a tree structure. This enables a flexible and adaptable event processing pipeline that can handle new event schemas without code redeployments.

11/9/2020

This post introduces three event modeling approaches for event-driven architectures: 'bare letter', 'deep envelope', and 'shallow envelope'. It details the advantages and disadvantages of each, focusing on aspects like schema overlap, consumer effort, data quality, maintainability, and compatibility checks. The 'bare letter' approach offers independence but risks schema overlap and increased consumer effort. 'Deep envelope' standardizes metadata for generic processing but adds complexity. 'Shallow envelope' provides flexibility with schema-on-read but defers risk to runtime and can lead to code duplication. The post also touches upon the use of Protobuf and Spring for Apache Kafka in conjunction with Confluent Schema Registry for implementing these models.

4/24/2020

This post introduces support for Protobuf and JSON schemas in Confluent Schema Registry, expanding its capabilities beyond Avro. It also adds support for customizable plug-in schemas for other formats. The post details the preview release of admin functions for Confluent REST Proxy, enabling programmatic administrative operations. Furthermore, it introduces exactly-once semantics for librdkafka-based clients (C/C++, Python, Go, .NET) through idempotent producers and transactions. Enhancements to ksqlDB 0.7 are also highlighted, including COUNT_DISTINCT, support for non-string keys, and improved pull query availability, along with the ksqlDB Flow View in Confluent Control Center.

12/9/2019

Introduces a custom Kafka Connect Single Message Transform (SMT) that enables the transfer of Avro data across independent Schema Registry clusters. The SMT looks up Avro schema text from a source Schema Registry, registers it in a destination Schema Registry, and updates the schema ID within the Kafka record. It also discusses potential ordering issues with schema registration and provides a workaround by setting compatibility to FULL or NONE. A demo architecture and order of operations are detailed, along with prerequisites for a Docker-based runbook.

9/27/2019

Introduces Schema Validation on write for Confluent Platform 5.4. This feature allows Kafka brokers to validate that data produced to a topic adheres to a registered schema in Confluent Schema Registry. It requires configuring `confluent.schema.registry.url` at the broker level and can be enabled per topic using `confluent.key.schema.validation` and `confluent.value.schema.validation`. The producer protocol is updated to provide specific error feedback for invalid messages within a batch.

4/17/2019

This post introduces the integration of Confluent Control Center with Confluent Schema Registry, enabling UI-based management of schemas. Key features include viewing schemas for any topic, editing schemas, checking schema compatibility against previous versions, and setting schema compatibility policies. It demonstrates how Control Center rejects incompatible schema changes and allows for schema version comparison. The post also outlines different schema compatibility options.

2/22/2019

This post details Deliveroo's 'Franz' framework, which enforces Protobuf schema validation and evolution for inter-service communication over Kafka. It describes the need for structured message formats, the evaluation of serialization formats (Thrift, Protobuf, Avro), and the decision to use Protobuf. The post elaborates on Protobuf's forwards and backwards compatibility features and outlines specific schema evolution rules enforced through unit tests: immutability of field numbers, no type amendments, reserving removed fields, and not amending field names. It also introduces the use of Protobuf custom options to define topic ownership and message formats, inspired by Confluent Schema Registry.

9/24/2015

This post introduces the critical need for a Schema Registry in data platforms, particularly in the context of Hadoop and stream processing. It details the challenges of managing schemas in distributed systems, the benefits of a centralized Schema Registry (organizational, resilience, evolution, efficiency, discovery, cost, policy enforcement), and how Confluent's Schema Registry addresses these needs with its REST API, versioning, compatibility validation, serializers, and formatters. It emphasizes the importance of Schema Registry for production-ready data pipelines.

5/19/2015

This post argues for the necessity of schemas in software development, drawing parallels to language standardization. It illustrates the problems arising from schema-free data with a 'temp-o-meter' example, highlighting how evolving data formats (CSV to JSON, and variations within JSON) lead to downstream consumer breakage and tight coupling. The post emphasizes that schemas provide a single, enforceable contract for data definitions, promoting the DRY principle and simplifying data compatibility testing. It sets the stage for a future post on how schemas facilitate data format evolution.