Broccoli: Syncing faster by syncing less

This post details the development and implementation of Broccoli, a modified Brotli encoder for optimizing file sync. It describes the evaluation of various compression algorithms (7zip, zstd, zlib, Brotli), the selection of Brotli due to its file size advantages, security (Rust implementation), familiarity, and HTTP support. The post explains the 'Broccoli' modification to enable faster, multi-core compression and in-process integration, and how it's layered into the block sync protocol for both uploads and downloads. Results show median latency reductions of over 30% and data transfer reductions of the same amount, with significant bandwidth savings for users.