Data Completeness Measurement

9 posts · Oct 2018 – Jun 2026

What this trail covers

Datadog built a system to measure data completeness across all ingestion pipelines in real-time for every customer. The system tracks payloads end-to-end by splitting pipelines into segments and counting creates and acknowledgments for unique payload identifiers within time buckets. This allows for localization of degradation and aggregation into an end-to-end view, handling complex pipeline structures and edge cases like delayed data and retries. The Stream Router system, responsible for data i

All Datadog trails

Jul 2026How we measure data completeness at scale | DatadogThis post details the evolution of Datadog's data completeness measurement system. The core innovation is the segment-based tracking of payloads, where each pipeline is divided into segments, and completeness is measured by comparing 'create' and 'acknowledgment' events for unique payload identifiers within time buckets. This approach handles retries and complex pipeline structures by ensuring idempotency through time-bucketed counts and allows for end-to-end pipeline completeness to be computed by combining segment ratios, particularly addressing challenges with parallel pipelines by using mathematical operations to combine segment-level measurements.
Jun 2026How we migrated a live routing system using AI-assisted refactoring | DatadogThis post details the migration of Datadog's Stream Router from a KV-based storage model (FoundationDB/RocksDB) to a relational schema backed by PostgreSQL (write path) and DuckDB (read path). The migration was driven by the limitations of the KV model in handling complex relationships between routes, sharding strategies, and rules, which led to transaction size limits and performance bottlenecks. The new schema directly models these relationships with foreign keys. The migration process was significantly accelerated using AI-assisted refactoring tools (Claude, Cursor) in a test-driven workflow, where AI generated code based on old implementations, new schemas, and failing tests. The post highlights the benefits of this approach for safely evolving critical production systems, emphasizing the human expertise still required for validation and complex logic.
Jun 2025How we built reliable log delivery to thousands of unpredictable endpoints | DatadogDatadog built a system for reliable log delivery to thousands of unpredictable external endpoints. This system leverages cloud object storage as a distribution center to store data for multiple destinations in single large files, grouped by destination. This provides the isolation advantages of a Kafka topic per destination endpoint without the operational overhead. It also improves recovery of live data after a destination endpoint has been temporarily unavailable, by allowing undeliverable logs to wait without blocking packages ready for delivery.
Feb 2025Achieving relentless Kafka reliability at scale with the Streaming Platform | DatadogDatadog's Streaming Platform abstracts Kafka complexity for real-time reliability at scale. It introduces 'Streams' that span multiple Kafka clusters, enabling seamless traffic redirection and failovers without reconfigurations. Consumer semantics are shifted to at-least-once delivery with relaxed ordering for parallelism. A custom coordinator, the Assigner, monitors cluster health and workload distribution in real-time. To address head-of-line blocking, 'Stream lanes' provide QoS differentiation, and an advanced commit log tracks multiple offsets for concurrent processing of live traffic and backlogs.
Nov 2024How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems | DatadogThis post details the application of formal modeling (TLA+), lightweight simulations, and chaos testing to the design of Datadog's new message queuing service, Courier. It describes Courier's requirements for multi-tenancy, at-least-once delivery, graceful degradation, and horizontal scalability. The design leverages multiple FoundationDB clusters for multi-tenancy and a broker layer for API exposure. The use of TLA+ and simulations is highlighted as a method to validate system-level properties and failure modes during the design phase, complementing traditional testing methods.
Advertisement
Dec 2023Scaling Self-Serve Analytics: The Tools Empowering 5,000 Employees | DatadogThis post details the evolution of Datadog's self-serve analytics platform, focusing on scaling data intake, transformation, quality monitoring, discovery, and reporting to empower 5,000 employees. It highlights the architecture for data intake using Airbyte, Spark, and a "Bring Your Own Data" (BYOD) capability. The data transformation layer utilizes dbt with strong conventions and observability. Data discovery tools provide comprehensive metadata. For reporting, Metabase is used with a focus on an intuitive interface, customizability, and automation via APIs. The post also emphasizes the importance of knowledge sharing through education and user training to foster a data-driven culture.
May 2022How Datadog’s IT team automated account inactivity and SaaS spend management | DatadogThis post details the expansion of the Clarity tool into the Clarity License Manager (CLM) to optimize software licensing efficiency and manage SaaS spend. CLM monitors account activity across SaaS applications, uses a Slackbot for employee interaction, and automates deactivation/reactivation. It leverages direct API integrations and Google Workspace SAML audit logs. The architecture is microservice-based, using Python on AWS Lambda with a central Postgres database and application-specific adapters. The goal is to deactivate unused accounts, reduce security risks, cut SaaS spend, and enable data-driven procurement.
Feb 2022How Datadog uses Datadog to gain visibility into the Datadog user experience | DatadogThis post details how Datadog's product design team uses Datadog's own Real User Monitoring (RUM), logs, and notebooks to gain visibility into the user experience of the Datadog platform. It provides three examples: selecting a monospace font by tracking actual loaded fonts via the CSS Font Loading API, updating the DraggablePane component by logging user interactions with minimize/maximize buttons and implementing double-click functionality, and expanding the input syntax for custom time frames in the DateRangePicker by analyzing user-entered custom inputs in Datadog Logs. The post highlights the use of RUM to verify the rollout of the new monospace font and the use of Datadog Logs to inform design decisions for UI components.
Apr 2019Building highly reliable data pipelines at Datadog | DatadogThis post details Datadog's best practices for building highly reliable data pipelines, focusing on architectural design, fault tolerance, and monitoring. Key strategies include using isolated, short-lived clusters for Spark jobs, leveraging spot instances to enforce fault tolerance, and breaking down long jobs vertically and horizontally. The post also emphasizes the importance of comprehensive monitoring through tagged clusters and the Datadog Agent to detect failures early and ensure timely data delivery.
Nov 2018Improving trust with Datadog Log Management | DatadogThis post details the integration of Datadog Log Management with Amazon SES to provide enhanced observability for critical emails like password resets. The solution involves configuring SES to publish events (send, reject, bounce, complaint, delivery, open, click) to an SNS topic, which then triggers a Lambda function. This Lambda transfers the email event data to Datadog Log Management. The post includes Terraform snippets for setting up the AWS resources (SNS topic, SES configuration set, SES event destination, Lambda function) and explains how to configure Datadog to view and facet these logs for easier searching and troubleshooting.