Astra Dynamic Chunks

This thread tracks the evolution of Slack's Astra log search engine, focusing on optimizing storage and cost efficiency. Initial posts detailed the problem of fixed-size log data chunks leading to wasted disk space and increased infrastructure costs. This post details the redesign of Astra's chunk management system to implement dynamic chunk sizing. The changes involved modifying the Cluster Manager and Cache nodes to utilize bin packing algorithms for more efficient allocation of cache node capacity, reducing cache node requirements by up to 50% and overall cache node costs by 20%.