Building highly reliable data pipelines at Datadog | Datadog

What this post added

This post details Datadog's best practices for building highly reliable batch data pipelines. It covers architectural decisions such as using isolated, short-lived clusters for Spark jobs, often leveraging AWS spot instances. The strategy involves breaking down long-running jobs vertically (persisting intermediate data) and horizontally (partitioning input data based on Kafka partitions). Robust monitoring is achieved through Datadog Agent deployment on all nodes, cluster tagging for isolation and filtering, and collecting metrics on cluster health and data lag.