Optimized Compression Libraries

5 posts · Dec 2012 – Sep 2024

What this trail covers

Cloudflare's compression capabilities have evolved to include Zstandard (zstd) as a supported algorithm for browser-to-Cloudflare traffic. Zstandard offers faster compression and decompression speeds than Brotli while maintaining comparable compression ratios to GZIP, leading to reduced latency and improved website performance. This builds upon existing efforts in optimizing compression algorithms and protocols for better web performance. The post also discusses the rollout of Encrypted Client Hello (ECH) and ongoing work on HTTP/3 prioritization and BBR congestion control.

All Cloudflare trails

Sep 2024New standards for a faster and more private InternetThis post details the integration and benefits of Zstandard (zstd) compression for Cloudflare's network traffic. It quantifies performance gains, comparing zstd to Brotli and GZIP in terms of compression speed and ratio. The post explains the technical underpinnings of zstd, including its larger sliding window, repcode modeling, parallel stream processing, branchless design, and Finite State Entropy (FSE) encoding. It also touches upon the rollout of Encrypted Client Hello (ECH) and ongoing work on HTTP/3 prioritization and BBR congestion control.
Nov 2020My internship: Brotli compression using a reduced dictionaryThis post details an internship project that improved Brotli compression by using a reduced, specialized dictionary for web content. The project implemented a radix trie, hash table, and bloom filter for fast dictionary lookup, and employed heuristics such as searching the dictionary at positions with poor LZ77 matches, considering only the longest match and its transforms, and including only useful transforms. Separate dictionaries were generated for HTML, CSS, and JavaScript based on representative web content, achieving compression improvements of 10-20% for smaller files with negligible performance impact.
Jul 2015Fighting Cancer: The Unexpected Benefit Of Open Sourcing Our CodeThis post details Cloudflare's optimized fork of the zlib compression library, highlighting specific technical improvements such as using uint64_t, an iSCSI CRC32 hash function, a longer match search window, SIMD instructions for window rolling, and the PLCMULQDQ instruction for CRC32 checksums. It also presents comprehensive benchmarks comparing the Cloudflare fork against the original zlib and Intel's optimized version across various datasets, demonstrating significant performance advantages.
Jan 2014Welcome 1&1 Internet, CloudFlare’s Newest Optimized Partner!This post announces the partnership with 1&1 Internet, making Cloudflare's Railgun technology available to their 12 million customers. It explains the technical challenge of offering Railgun to a broad audience and how the 'Optimized Partner' program, using hosting providers as aggregation points, solves this by allowing one or more Railgun servers to accelerate performance for multiple websites. This makes a technology that previously cost thousands of dollars per month available at no cost to 1&1 customers.
Jul 2013Railgun Gives our Ecommerce Sites the EdgeThis post details Luxury Link's experience using Cloudflare's Railgun technology. They observed a 40% reduction in HTML load time from their Los Angeles-hosted home page when accessed from the US East Coast, and a 2x speedup from London. Crucially, Railgun dramatically reduced outbound bandwidth consumption from their origin servers, flattening traffic peaks from a 10:1 inbound to outbound ratio to a 1:1 ratio during peak times, and from 5:1 to 3:1 during off-peak times. This was attributed to Railgun's delta compression algorithm, which effectively sends only the differences between uncached HTML documents.
Advertisement
Feb 2013CloudFlare's Railgun: Easier Than EverThis post announces version 3.3.3 of Cloudflare's Railgun technology, which has been battle-tested on high-traffic sites. It provides RPMs for popular Linux and BSD variants, and an AMI for AWS. It also announces that a majority of the world's leading hosting providers now support Railgun, allowing users to enable it with a single click. The post highlights Railgun as a revolutionary new protocol that makes dynamic web performance significantly faster and less bandwidth-intensive.
Dec 2012Efficiently compressing dynamically generated web contentThis post introduces the concept of using external dictionaries for compression, going beyond generic methods like gzip. It details two proposals, SDCH and ESI, and discusses their potential and challenges. The article highlights how these techniques can significantly reduce the size of dynamically generated HTML content by referencing common 'blocks' of data, thereby minimizing the impact of latency. It also mentions Cloudflare's own Railgun technology as a solution for this problem.