Stream Lineage

4 posts · Sep 2021 – Oct 2025

What this trail covers

Stream Lineage provides an interactive map of data flows in event-driven architectures, enabling users to understand data origins, transformations, and destinations. It visualizes producers, consumers, connectors, Kafka Streams applications, and ksqlDB streams/tables. The tool supports time-windowed views (up to 24 hours) and search functionality for specific clients, topics, connectors, ksqlDB queries, and consumer group IDs. Use cases include development of data pipelines, troubleshooting of d

All Confluent trails

Oct 2025Real-Time Compliance & Audit Logging With Apache Kafka®This post details building a real-time compliance and audit logging pipeline using Apache Kafka, Kafka Connect for ingestion, stream processing (Kafka Streams/Flink) for normalization and enrichment, and tiered storage (object storage) for long-term retention. It emphasizes data accuracy, immutability, guaranteed retention, security (encryption, authentication, RBAC), real-time processing, and data lineage. It outlines a reference architecture with specific components for ingestion, processing, storage, and access layers, and provides a 7-step implementation guide.
Dec 2023Data Portal for Confluent Stream GovernanceThis post announces the general availability of Data Portal on Confluent Cloud. It details the functionality of Data Portal, which leverages Stream Catalog and Stream Lineage to provide a self-service interface for discovering Kafka topics. Users can search and filter topics by metadata, view topic details including schema and lineage, and request access through an approval workflow. The post also highlights the integration with Apache Flink for querying data directly from the UI.
Jul 2022How to Find, Share, and Organize Your Data Streams with Stream Catalog | ConfluentThis post introduces the Confluent Stream Catalog, detailing its core components (Apache Atlas, JanusGraph) and its functionality for searching, discovering, and organizing Kafka topics and schemas. It highlights the availability of a REST API for programmatic interaction, including searching for entities and tagging them with metadata like PII. A new GraphQL API is also introduced, leveraging the graph nature of the metadata for more complex queries and exploring relationships between entities. The post emphasizes the catalog's role in data governance and its integration with other Stream Governance pillars like Stream Lineage and Stream Quality.
Jul 2022How to Visualize Your Apache Kafka Data the Easy Way with Stream Lineage | ConfluentThis post introduces and details Confluent's Stream Lineage tool, explaining its functionality for visualizing data flows in Kafka and event-driven architectures. It covers understanding data origins, transformations, and destinations, and lists the types of components visualized (Connectors, Kafka Clients, Kafka Streams, ksqlDB). The post highlights automatic generation without extra instrumentation, time-windowed views, search capabilities, and provides three detailed use cases: development of data pipelines (with a step-by-step example), monitoring and troubleshooting of data pipelines (with a scenario for a paused connector), and compliance tracking of PII data flows (with an export feature for auditors).
Sep 2021Governing Data in Motion with Confluent Stream GovernanceThis post introduces Confluent's Stream Governance suite, focusing on three pillars: Stream Quality, Stream Catalog, and Stream Lineage. It details how to secure sensitive data using RBAC, ACLs, Audit Logs, and encryption. It also covers defining and validating schemas using Schema Registry, including schema linking and broker-side validation. The Stream Catalog is enhanced with tagging capabilities for data classification and discovery. The post uses an air traffic control scenario to illustrate these concepts.
Advertisement