Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

This post details the engineering implementation of Netflix's real-time service topology system, focusing on the 'how'. Key contributions include: a streaming-first architecture to overcome batch processing limitations for real-time observability; the critical role of backpressure in reactive streams for handling millions of flow records per second without data loss or system crashes; a three-stage distributed aggregation pipeline for network flow logs that resolves network intermediaries to provide application-level dependencies; and the architectural decision of physically separating network, IPC, and tracing layers for independent optimization and unified querying.