Optimizing data warehouse storage

This post introduces AutoOptimize, a system designed to efficiently and transparently optimize data and metadata storage layout in data warehouses, focusing on use cases like file merging, sorting, compaction, and metadata optimization. It employs design principles of 'just-in-time' optimization, 'essential' optimization, and 'minimum replacement' to reduce resource usage and latency. The architecture is split into a decision-making Service and action-performing Actors, leveraging Apache Iceberg for table format and introducing 'Partition Entropy' for early pruning.