How we measure data completeness at scale | Datadog

This post details the evolution of Datadog's data completeness measurement system. The core innovation is the segment-based tracking of payloads, where each pipeline is divided into segments, and completeness is measured by comparing 'create' and 'acknowledgment' events for unique payload identifiers within time buckets. This approach handles retries and complex pipeline structures by ensuring idempotency through time-bucketed counts and allows for end-to-end pipeline completeness to be computed by combining segment ratios, particularly addressing challenges with parallel pipelines by using mathematical operations to combine segment-level measurements.