Improving compression with a preset DEFLATE dictionary

This post details the exploration and experimental application of preset DEFLATE dictionaries to improve compression ratios across all HTML pages served by Cloudflare. It explains the DEFLATE algorithm's LZ77 stage and how a preset dictionary can act as an initial back-reference for common strings, improving compression efficiency. An experiment is described where over 20,000 HTML pages were scanned to create 16KB and 32KB dictionaries, which were then benchmarked against a set of 19,000 HTML files. The results showed that using a 16KB dictionary improved compression by approximately 25% compared to DEFLATE alone.