Blogs›Datadog›Data Completeness Measurement
Datadog built a system to measure data completeness across all ingestion pipelines in real-time for every customer. The system tracks payloads end-to-end by splitting pipelines into segments and counting creates and acknowledgments for unique payload identifiers within time buckets. This allows for localization of completeness degradation and aggregation into an end-to-end view, handling complex pipeline structures and edge cases like delayed data and retries. The Stream Router system, responsible for data aggregation and post-processing, is a key component. The company also employs best practices for building highly reliable batch data pipelines, including using isolated, short-lived clusters (often on spot instances) for Spark jobs, breaking down long-running jobs vertically and horizontally, and implementing robust monitoring with cluster-level tagging. This approach ensures fault tolerance and efficient recovery from failures.
7/1/2026
Introduced a novel approach to measure data completeness at scale by defining pipeline segments and tracking payloads via create and acknowledgment events. Developed an idempotent time-bucket model for counting, and a method to compute end-to-end pipeline completeness from segment ratios, including handling sequential and parallel pipeline structures.
6/23/2026
This post details the migration of Datadog's Stream Router system from a Key-Value (KV) storage model (FoundationDB and RocksDB) to a relational model using PostgreSQL for writes and DuckDB for reads. The migration was driven by scaling limitations of the KV model, specifically FoundationDB's transaction size limits, which hindered the system's ability to manage complex relationships between routes, sharding strategies, and rules. The new schema introduces explicit foreign key relationships, allowing the database to enforce constraints and simplifying application logic. The migration process was significantly accelerated by AI-assisted refactoring tools (Claude and Cursor) within a test-driven development workflow, focusing on safe evolution of a critical production system.
6/23/2025
This post details the design and implementation of Datadog's Log Forwarding feature, focusing on building a reliable system for delivering logs to thousands of unpredictable external endpoints. The system addresses challenges of scale, low latency, high throughput, and multi-tenancy by abstracting the complexities of external endpoint unreliability. It leverages cloud object storage as a distribution center, grouping data by destination to avoid blocking progress due to a single unavailable endpoint. This approach provides the isolation benefits of per-destination Kafka topics without the associated operational overhead and improves recovery of live data after endpoint downtime.
2/19/2025
Datadog's Streaming Platform abstracts Kafka complexity for real-time reliability at scale. It introduces 'Streams' that span multiple Kafka clusters, enabling seamless traffic redirection and failovers without reconfigurations. Consumer semantics are shifted to at-least-once delivery with relaxed ordering for parallelism. A custom coordinator, the Assigner, monitors cluster health and workload distribution in real-time. To address head-of-line blocking, 'Stream lanes' provide QoS differentiation, and an advanced commit log tracks multiple offsets for concurrent processing of live traffic and backlogs.
12/21/2023
This post details the evolution of Datadog's self-serve analytics capabilities, focusing on the data intake, transformation, discovery, and reporting tools that empower 5,000 employees. It expands on the existing 'Data Completeness Measurement' by describing the broader ecosystem of tools and processes that create a 'single source of truth' for internal data. Key technical components include Airbyte and Spark for data intake, dbt for transformation with enforced conventions, an internal Data Discovery tool, and Metabase/Jupyter notebooks for analysis and reporting. The post also highlights the 'Bring Your Own Data' (BYOD) initiative and the importance of data quality monitoring and documentation.
5/12/2022
This post details the expansion of the Clarity tool into the Clarity License Manager (CLM) to optimize software licensing efficiency and manage SaaS spend. CLM monitors account activity across SaaS applications, uses a Slackbot for employee interaction, and automates deactivation/reactivation. It leverages direct API integrations and Google Workspace SAML audit logs. The architecture is microservice-based, using Python on AWS Lambda with a central Postgres database and application-specific adapters. The goal is to deactivate unused accounts, reduce security risks, cut SaaS spend, and enable data-driven procurement.
2/22/2022
This post details how Datadog's product design team uses Datadog's own Real User Monitoring (RUM), logs, and notebooks to gain visibility into the user experience of the Datadog platform. It provides three examples: selecting a monospace font by tracking actual loaded fonts via the CSS Font Loading API, updating the DraggablePane component by logging user interactions with minimize/maximize buttons and implementing double-click functionality, and expanding the input syntax for custom time frames in the DateRangePicker by analyzing user-entered custom inputs in Datadog Logs. The post highlights the use of RUM to verify the rollout of the new monospace font and the use of Datadog Logs to inform design decisions for UI components.
4/2/2019
This post details Datadog's best practices for building highly reliable batch data pipelines. It covers architectural decisions such as using isolated, short-lived clusters for Spark jobs, often leveraging AWS spot instances. The strategy involves breaking down long-running jobs vertically (persisting intermediate data) and horizontally (partitioning input data based on Kafka partitions). Robust monitoring is achieved through Datadog Agent deployment on all nodes, cluster tagging for isolation and filtering, and collecting metrics on cluster health and data lag.
11/1/2018
This post details the integration of Datadog Log Management with Amazon SES to provide enhanced observability for critical emails like password resets. The solution involves configuring SES to publish events (send, reject, bounce, complaint, delivery, open, click) to an SNS topic, which then triggers a Lambda function. This Lambda transfers the email event data to Datadog Log Management. The post includes Terraform snippets for setting up the AWS resources (SNS topic, SES configuration set, SES event destination, Lambda function) and explains how to configure Datadog to view and facet these logs for easier searching and troubleshooting.