Husky: Exactly-once ingestion and multi-tenancy at scale | Datadog

This post details the evolution of Datadog's Husky event store's ingestion pipeline, focusing on achieving exactly-once ingestion and multi-tenancy at scale. It introduces the Shard Router service, which uses deterministic routing based on event ID and timestamp to map events to specific shards. To handle dynamic changes in shard assignments and ensure consensus across distributed Shard Router nodes, the system employs time-bounded Shard Placements and a central Sharding Allocator. This approach minimizes storage costs and improves performance by limiting the number of tenants processed by each worker and reducing the overhead associated with logical namespaces.