How we built reliable log delivery to thousands of unpredictable endpoints | Datadog

What this post added

This post details the design and implementation of Datadog's Log Forwarding feature, focusing on building a reliable system for delivering logs to thousands of unpredictable external endpoints. The system addresses challenges of scale, low latency, high throughput, and multi-tenancy by abstracting the complexities of external endpoint unreliability. It leverages cloud object storage as a distribution center, grouping data by destination to avoid blocking progress due to a single unavailable endpoint. This approach provides the isolation benefits of per-destination Kafka topics without the associated operational overhead and improves recovery of live data after endpoint downtime.