Husky: Exactly-once ingestion and multi-tenancy at scale | Datadog

What this post added

This post details the evolution of Datadog's Husky event store to achieve exactly-once ingestion and multi-tenancy at scale. It introduces the Shard Router service for deterministic event routing to shards based on event ID and timestamp, leveraging time-bounded Shard Placements to handle dynamic tenant shard assignments. A Sharding Allocator service is used to establish consensus among Shard Router nodes. The system aims to minimize duplicate events and optimize storage costs by limiting the number of tenants processed by individual storage nodes.