Scaling Exact COUNT(DISTINCT) for High-Cardinality Non-Rollup Metrics in Distributed Data Pipelines

Introduced a chunked aggregation buffer strategy for computing exact distinct counts of high-cardinality non-rollup metrics in JVM-based data pipelines. This strategy partitions the aggregation buffer using the top 16 bits of hashed identifiers, creating independent chunks that serialize and deserialize separately. This circumvents the JVM's 2 GB array indexing limit, allowing for safe computation of cardinalities up to approximately 11.7 trillion. The solution also enables concurrent execution of backfill jobs by eliminating shared state and ensures backward compatibility with legacy monolithic bitmap formats through a self-describing magic number router. The final output is a cardinality-only 8-byte long, avoiding bitmap serialization at the reduce stage.