TechPrufer
Sign inGet started

Blogs›Confluent›Stream Lineage

Stream Lineage

Stream Lineage

3
posts
2021–2022

Stream Lineage provides an interactive, graphical UI to visualize data flows in Apache Kafka and event-driven architectures. It automatically maps producers, consumers, Kafka topics, Connectors, Kafka Streams applications, and ksqlDB streams/tables. Users can trace data origins, transformations, and destinations, search for specific components, and adjust time windows up to 24 hours. Use cases include developing data pipelines, monitoring and troubleshooting data flows, and supporting compliance. This post introduces the Stream Governance suite, which includes Stream Lineage, Stream Catalog, and Stream Quality features. Stream Catalog enables classification and discovery of event streams, while Stream Quality focuses on data integrity and validation through features like schema linking and broker-side schema ID validation.

2022

How to Find, Share, and Organize Your Data Streams with Stream Catalog | Confluent

7/14/2022

This post introduces Confluent's Stream Catalog, a new capability for discovering, understanding, organizing, and reusing data streams. It details the underlying technology (Apache Atlas, JanusGraph) and the available APIs (REST and GraphQL) for interacting with the catalog. The post explains how schemas and their components are automatically cataloged and can be tagged with business metadata and PII tags. It also highlights the integration with Stream Lineage and Stream Quality as part of Confluent's broader Stream Governance suite.

How to Visualize Your Apache Kafka Data the Easy Way with Stream Lineage | Confluent

7/6/2022

This post introduces Confluent's Stream Lineage tool, which automatically visualizes data flows in Kafka and event-driven architectures. It details how the tool maps producers, consumers, topics, connectors, Kafka Streams, and ksqlDB, and allows users to trace data origins, transformations, and destinations. The post outlines three key use cases: development of data pipelines, monitoring and troubleshooting of data flows, and compliance tracking for data processing. It also explains how to adjust time windows and use search functionality.

2021

Governing Data in Motion with Confluent Stream Governance

9/14/2021

This post introduces the Confluent Stream Governance suite, which includes Stream Catalog, Stream Quality, and Stream Lineage. It details the introduction of Stream Catalog for classifying and discovering event streams, and Stream Quality features like schema linking (for syncing schemas across clusters) and broker-side schema ID validation (for ensuring data integrity). The post uses an air traffic control scenario to illustrate these capabilities, emphasizing security, schema definition, and data classification.