Real-Time Distributed Graph Querying

Netflix is evolving its Real-Time Distributed Graph (RDG) by developing a sophisticated query serving layer. This layer is designed to efficiently handle diverse query patterns, from shallow-wide fan-outs to deep-narrow traversals, all while maintaining sub-100ms latency. Key architectural choices include a breadth-first traversal strategy, an async-first execution pipeline, selective caching of stable data, opt-in enrichments from external services, and a default to eventual consistency. This post details the ingestion and processing of data into the RDG, focusing on the challenges of handling high-volume, real-time data streams and ensuring data consistency and availability. It describes the use of Kafka for event ingestion, Flink for stream processing, and a custom data model for representing graph entities. The system employs techniques like idempotent writes and backpressure handling to manage the data flow effectively.