TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Schema Registry Best Practices

17 posts · May 2015 – Mar 2026

What this trail covers

Confluent Schema Registry now supports storing schema IDs in Kafka headers, enabling backward-compatible adoption of schema governance without breaking legacy consumers or changing payload formats. This facilitates incremental migration from schemaless data to a governed data plane, improving data quality, powering contextualized AI/analytics, and simplifying lakehouse integration. The new format uses 16-byte GUIDs for schema identification, offering broader compatibility across environments. This post details how data contracts, CEL rules, DLQs, and client-side field-level encryption (CSFLE) can be used to protect PII in Kafka.

All Confluent trails
  1. Introducing Schema IDs in Kafka Headers for Smarter Data GovernanceIntroduced the ability to store Schema Registry schema IDs in Kafka headers, offering a backward-compatible mechanism for adopting schema governance. This allows for incremental migration from schemaless data to a governed data plane by keeping existing payload formats intact. The new format uses 16-byte GUIDs for schema identification, which are resolvable via Schema Registry APIs. This feature is supported across various client libraries and integrates with Kafka Connect, Flink, and the Confluent UI/CLI.
  2. How to Protect PII in Kafka With Schema Registry and Data ContractsThis post details how to protect PII in Kafka using Confluent Schema Registry by implementing data contracts. It demonstrates using CEL expressions for data quality validation (e.g., email format, SSN format) and simple masking (e.g., last four digits of SSN). It also shows how to configure a Dead Letter Queue (DLQ) to capture invalid messages and introduces Client-Side Field-Level Encryption (CSFLE) for encrypting sensitive fields like email, leveraging envelope encryption with Key Encryption Keys (KEKs) and Data Encryption Keys (DEKs) managed by a DEK Registry.
  3. Schema Registry Best PracticesThis post details best practices for using Confluent Schema Registry, including understanding schema IDs, subjects, and versions. It elaborates on data contracts, emphasizing structure, integrity constraints, metadata, rules/policies, and change/evolution. The post recommends pre-registering schemas via CI/CD or REST APIs and enabling schema normalization for interoperability. It advocates for the topic-based subject name strategy with union types and schema composition using schema references. Subject aliases are introduced for flexible subject naming. The post thoroughly explains compatibility levels (backward, forward, full) and their implications, recommending backward compatibility as the default and discussing how to handle compatibility checks and potential issues with open content models.
  4. Using Data Contracts to Ensure Data Quality and ReliabilityThis post details the implementation of data contracts within Confluent Schema Registry. It provides concrete examples of defining Avro schemas, registering metadata (owner, SLOs), and adding data quality rules using CEL expressions. It also outlines the Java interfaces for custom rule executors and actions, demonstrating how to implement a custom email alert action for timeliness SLO violations. The post walks through registering these enhancements as new schema versions.
  5. Data Contracts Are More Than Just APIsThis post extends the understanding of data contracts beyond just APIs, emphasizing their role in fostering collaboration between software and data teams. It highlights how established API practices like versioning, schema evolution, and compatibility levels can be applied to data streaming systems using Confluent Schema Registry. The post advocates for treating data consumers (data teams) as first-class citizens, necessitating negotiated schemas and avoiding direct database access. It reinforces the value of Schema Registry in enabling loose coupling and abstraction for data architectures, ultimately driving better business outcomes through improved collaboration.
  7. Benefits of Event-Driven Architecture (EDA) and Data MeshThis post introduces the concept of data products and how to manage their lifecycle using descriptor files and a self-serve platform, with a track and trace example using Confluent Cloud components like Kafka topics, Schema Registry, and ksqlDB. It elaborates on the principles of data mesh and event-driven architectures, emphasizing data as a product and automation through self-serve platforms.
  8. Data Governance for Real-Time Data StreamsThis post introduces Confluent Cloud's Stream Governance capabilities, focusing on Schema Registry for managing schema evolution and compatibility rules, Stream Catalog for data discoverability and metadata tagging, and Stream Lineage for visualizing data flow. It highlights how these tools address challenges like data silos and stale documentation in data streaming architectures.
  9. How to Keep Bad Data Out of Apache Kafka with Stream Quality | Confluent Streaming Data Quality - Keep Bad Data Out of Apache KafkaThis post dives deeper into Schema Registry's functionality, detailing its components (REST service, serializers/deserializers), integration points with Kafka clients, Connect, ksqlDB, and Confluent GUI. It provides a step-by-step guide on using Schema Registry, including choosing serialization formats, registering schemas, producing/consuming records, and evolving schemas with compatibility types. It also introduces schema linking for sharing schemas across environments and uses the StreamWearables example to demonstrate the practical implications of schema management.
  10. Easily Copy and Migrate Data with Schema LinkingIntroduces Schema Linking, a new feature for Confluent Cloud and Confluent Platform 7.0, which allows for the replication of schemas between Schema Registry clusters. It introduces two new concepts: schema contexts, which provide independent scopes within Schema Registry, and schema exporters, which continuously export schemas from one Schema Registry cluster to another or across contexts within the same cluster. Schema Linking can be used independently or to complement Cluster Linking for data synchronization.
  11. Ensure Data Quality and Evolvability with a Secured Schema RegistryThis post details the implementation of security features for Confluent Schema Registry in production environments. It covers configuring HTTPS endpoints and mTLS for encrypted communication, enabling server-side schema validation on Kafka brokers, and leveraging Metadata Service (MDS) with Role-Based Access Control (RBAC) for authentication and authorization. It also provides example Kafka client configurations for secure communication with both Kafka brokers and Schema Registry, and demonstrates RBAC role binding for subject access.
  12. Self-Describing Events Using Spring Kafka and Confluent Schema RegistryThis post introduces a generic processor for handling diverse Protobuf event schemas using Spring Kafka and Confluent Schema Registry. It details how to use `DynamicMessage` to parse any Protobuf record at runtime, converting it to JSON for reporting systems. The post also presents a mechanism for enhancing timestamp representations within the JSON output (e.g., Unix epoch, preserved proto format) by traversing the `DynamicMessage` object as a tree structure. This enables a flexible and adaptable event processing pipeline that can handle new event schemas without code redeployments.
  13. Spring for Apache Kafka and Protobuf Part 1: Event Data ModelingThis post introduces three event modeling approaches for event-driven architectures: 'bare letter', 'deep envelope', and 'shallow envelope'. It details the advantages and disadvantages of each, focusing on aspects like schema overlap, consumer effort, data quality, maintainability, and compatibility checks. The 'bare letter' approach offers independence but risks schema overlap and increased consumer effort. 'Deep envelope' standardizes metadata for generic processing but adds complexity. 'Shallow envelope' provides flexibility with schema-on-read but defers risk to runtime and can lead to code duplication. The post also touches upon the use of Protobuf and Spring for Apache Kafka in conjunction with Confluent Schema Registry for implementing these models.
  14. Confluent Platform 5.5 - Enhanced Event Streaming for All DevelopersThis post introduces support for Protobuf and JSON schemas in Confluent Schema Registry, expanding its capabilities beyond Avro. It also adds support for customizable plug-in schemas for other formats. The post details the preview release of admin functions for Confluent REST Proxy, enabling programmatic administrative operations. Furthermore, it introduces exactly-once semantics for librdkafka-based clients (C/C++, Python, Go, .NET) through idempotent producers and transactions. Enhancements to ksqlDB 0.7 are also highlighted, including COUNT_DISTINCT, support for non-string keys, and improved pull query availability, along with the ksqlDB Flow View in Confluent Control Center.
  15. Transferring Avro Data Across Schema Registries with Kafka ConnectIntroduces a custom Kafka Connect Single Message Transform (SMT) that enables the transfer of Avro data across independent Schema Registry clusters. The SMT looks up Avro schema text from a source Schema Registry, registers it in a destination Schema Registry, and updates the schema ID within the Kafka record. It also discusses potential ordering issues with schema registration and provides a workaround by setting compatibility to FULL or NONE. A demo architecture and order of operations are detailed, along with prerequisites for a Docker-based runbook.
  16. Schema Validation with Confluent Platform 5.4Introduces Schema Validation on write for Confluent Platform 5.4. This feature allows Kafka brokers to validate that data produced to a topic adheres to a registered schema in Confluent Schema Registry. It requires configuring `confluent.schema.registry.url` at the broker level and can be enabled per topic using `confluent.key.schema.validation` and `confluent.value.schema.validation`. The producer protocol is updated to provide specific error feedback for invalid messages within a batch.
  17. Dawn of Kafka DevOps: Managing and Evolving Schemas with Control Center | ConfluentThis post introduces the integration of Confluent Control Center with Confluent Schema Registry for managing and evolving schemas. It highlights how Control Center allows users to view schemas, edit them, check compatibility against previous versions, and set compatibility policies. The post uses an example of a Java producer with an Avro schema to demonstrate schema registration and deserialization in Control Center. It also explains the importance of schema evolution and compatibility checks, illustrating how Control Center rejects incompatible schema changes and allows for schema version comparison.
  18. Improving Stream Data Quality with Protobuf Schema Validation | ConfluentThis post details Deliveroo's 'Franz' framework, which enforces Protobuf schema validation and evolution for inter-service communication over Kafka. It describes the need for structured message formats, the evaluation of serialization formats (Thrift, Protobuf, Avro), and the decision to use Protobuf. The post elaborates on Protobuf's forwards and backwards compatibility features and outlines specific schema evolution rules enforced through unit tests: immutability of field numbers, no type amendments, reserving removed fields, and not amending field names. It also introduces the use of Protobuf custom options to define topic ownership and message formats, inspired by Confluent Schema Registry.
  19. Securing the Confluent Schema Registry for Apache Kafka | ConfluentThis post details how to secure the Confluent Schema Registry for Apache Kafka by configuring SASL/SSL for authentication and authorization, and enabling HTTPS for the REST API. It covers setting up JAAS files, configuring Kafka ACLs for the `_schemas` topic, and enabling SSL encryption for the Schema Registry's REST API. The analogy of a bank is used to explain the security concepts: employee identification (SASL), job-specific permissions (ACLs), and private client discussions (HTTPS/SSL).
  20. Yes, Virginia, You Really Do Need a Schema Registry | ConfluentThis post introduces the critical need for a Schema Registry in data platforms, particularly in the context of Hadoop and stream processing. It details the challenges of managing schemas in distributed systems, the benefits of a centralized Schema Registry (organizational, resilience, evolution, efficiency, discovery, cost, policy enforcement), and how Confluent's Schema Registry addresses these needs with its REST API, versioning, compatibility validation, serializers, and formatters. It emphasizes the importance of Schema Registry for production-ready data pipelines.
  21. How I Learned to Stop Worrying and Love the Schema | ConfluentThis post argues for the necessity of schemas in software development, drawing parallels to language standardization. It illustrates the problems arising from schema-free data with a 'temp-o-meter' example, highlighting how evolving data formats (CSV to JSON, and variations within JSON) lead to downstream consumer breakage and tight coupling. The post emphasizes that schemas provide a single, enforceable contract for data definitions, promoting the DRY principle and simplifying data compatibility testing. It sets the stage for a future post on how schemas facilitate data format evolution.