TechPrufer
Get started
Netflix
Netflix logoNetflix · Feature trail

Real-time Service Topology

4 posts · Oct 2020 – Jul 2026

What this trail covers

Netflix is evolving its real-time service topology system to provide engineers with a unified, up-to-the-minute view of service dependencies. This system ingests data from multiple sources (eBPF network flows, IPC metrics, distributed tracing) and processes it through reactive pipelines with backpressure handling to deliver near real-time topology updates. The architecture features a multi-layer approach with physically separate topology layers (Network, IPC, Tracing) optimized for different data stores and query patterns. This post details the engineering challenges and solutions for building this system at scale, focusing on the three-stage distributed aggregation pipeline for network flow logs, backpressure handling in reactive streams, and the trade-offs involved in achieving real-time processing.

All Netflix trails
  1. Building Service Topology at Scale: Architecture, Challenges, and Lessons LearnedThis post details the engineering implementation of Netflix's real-time service topology system, focusing on the 'how'. Key contributions include: a streaming-first architecture to overcome batch processing limitations for real-time observability; the critical role of backpressure in reactive streams for handling millions of flow records per second without data loss or system crashes; a three-stage distributed aggregation pipeline for network flow logs that resolves network intermediaries to provide application-level dependencies; and the architectural decision of physically separating network, IPC, and tracing layers for independent optimization and unified querying.
  2. MediumThis post details how Netflix leverages eBPF flow logs to enhance its real-time service topology system, specifically focusing on the network layer. It describes the challenges of collecting and processing high-volume network flow data at scale, the architecture of the eBPF-based data collection pipeline, and how this data is integrated into the existing topology system to provide granular network insights. The post highlights the use of eBPF for efficient kernel-level data capture and the subsequent processing and aggregation of this data to build a comprehensive view of service interdependencies.
  3. MediumThis post details the challenges and solutions in achieving observability in asynchronous workflows, specifically focusing on the complexities of distributed tracing and event correlation in a microservices environment. It highlights the need for robust mechanisms to reconstruct the flow of requests across multiple services and the importance of correlating asynchronous events to gain a holistic understanding of system behavior. The post discusses the use of correlation IDs and event sequencing to piece together the journey of a request through various services, emphasizing the difficulties introduced by asynchronous communication patterns and the need for specialized tooling to overcome these hurdles.
  4. Building Netflix’s Distributed Tracing InfrastructureThis post details the design and implementation of Netflix's distributed tracing infrastructure, which powers troubleshooting tools like Edgar. It covers trace instrumentation using Open-Zipkin's B3 HTTP header propagation, a hybrid head-based sampling approach for trace data collection, and stream processing using Mantis for ingesting and processing traces. The infrastructure stores traces in Cassandra and aims to provide engineers with service topology, retry/error tags, and latency measurements for all service calls to reconstruct and diagnose streaming session failures.