HTML Parsing and Modification

2 posts · Sep 2016 – Feb 2023

What this trail covers

Cloudflare's HTML parsing and modification capabilities have evolved from a complex, difficult-to-maintain Ragel-based parser to a new, faster, and more maintainable streaming parser named cf-html (LOL HTML). This evolution aims to improve the efficiency and reliability of features that modify HTML on the fly, such as email obfuscation, server-side excludes, and automatic HTTPS rewrites. The transition to cf-html is ongoing, with a focus on migrating all functionalities to the new parser. The HTML parser bug that caused memory leaks has been fixed by addressing pointer errors in the Ragel implementation and ensuring proper buffer handling, particularly at the end of buffers and in error conditions.

All Cloudflare trails

Feb 2023ROFL with a LOL: rewriting an NGINX module in RustThis post details the development of ROFL, a new NGINX module written in Rust, to replace the legacy C-based cf-html module. It highlights the motivation for the rewrite, including memory safety concerns stemming from the Cloudbleed incident and the desire for a more performant and maintainable system. The post explains the process of writing an NGINX module in Rust, including generating FFI bindings using Bindgen and integrating with NGINX's dynamic module loading mechanism. It also mentions the use of lol-html, a Rust-based HTML parser, as the core component of ROFL.
Nov 2019Rust streaming HTML parser/rewriterThis post introduces LOL HTML, a new Rust-based streaming HTML parser and rewriter designed for Cloudflare Workers. It features a dual-parser architecture (lexer and tag scanner) to optimize performance by skipping non-scoped content. A Rust macro-based DSL is used to implement the parser's state machine. Performance optimizations include token outlines and lazy character decoding. A custom CSS selector matching engine is also developed as a virtual machine.
Nov 2019A History of HTML Parsing at Cloudflare: Part 1This post details the initial evolution of HTML parsing at Cloudflare, starting with ad-hoc parsers for email obfuscation using Ragel and progressing to a more spec-compliant HTML parser (lazyhtml) by 2016. It explains the concept of streaming HTML rewriters, their requirements (output latency, parser throughput, memory limitations), and the challenges of handling HTML at Cloudflare's scale. The post also introduces the open-sourced Rust library LOL HTML.
Nov 2019Introducing the HTMLRewriter API to Cloudflare WorkersIntroduces the generally available HTMLRewriter API for Cloudflare Workers, a streaming parser that allows developers to manipulate HTML content at the edge. The API provides a jQuery-like experience for DOM manipulation using selectors and handlers, enabling dynamic applications and reducing origin load. The post includes an example of rewriting URLs in anchor and image tags and discusses the benefits of edge-side HTML rewriting over client-side modifications, such as improved performance, consistency, and reduced client-side bloat.
Sep 2019Not so static... Introducing the HTMLRewriter API Beta to Cloudflare WorkersIntroduced the HTMLRewriter API beta for Cloudflare Workers, enabling streaming HTML parsing and DOM manipulation via a JavaScript API. This allows for dynamic transformations on static HTML served via Workers Sites, facilitating edge-side rendering for JAMstack applications and improving TTFB by avoiding full response ingestion. The API offers a jQuery-like experience for selecting and modifying HTML elements.
Advertisement
Feb 2017Incident report on memory leak caused by Cloudflare parser bugThis post details a critical security incident where a bug in Cloudflare's Ragel-based HTML parser led to memory leaks, potentially exposing sensitive data. The bug, present for years but only manifesting due to subtle changes in buffering introduced by the new cf-html parser, was fixed by modifying the Ragel parser's error handling to include `fhold` and by disabling affected features (Email Obfuscation, Server-side Excludes, Automatic HTTPS Rewrites). The incident highlights the importance of rigorous testing, robust error handling in parser implementations, and the benefits of a global, cross-functional response team for rapid mitigation.
Sep 2016Fixing the mixed content problem with Automatic HTTPS RewritesThis post introduces the Automatic HTTPS Rewrites feature, which addresses the 'mixed content' problem by automatically upgrading HTTP resource URLs to HTTPS. It explains the dangers of mixed content (active and passive) and how browsers handle them. The post details the challenges of manually fixing mixed content and presents Automatic HTTPS Rewrites as a one-click solution that modifies links in real-time as pages pass through Cloudflare's service, ensuring that all subresources are served over HTTPS.