How we built reliable log delivery to thousands of unpredictable endpoints | Datadog

Datadog built a system for reliable log delivery to thousands of unpredictable external endpoints. This system leverages cloud object storage as a distribution center to store data for multiple destinations in single large files, grouped by destination. This provides the isolation advantages of a Kafka topic per destination endpoint without the operational overhead. It also improves recovery of live data after a destination endpoint has been temporarily unavailable, by allowing undeliverable logs to wait without blocking packages ready for delivery.