Process Management & Graceful Upgrades

6/25/2026

This post introduces saga rollbacks for Cloudflare Workflows, enabling developers to define compensation logic directly within workflow steps. This feature addresses the challenge of handling failures in multi-step processes by allowing for semantic reversal of operations, enhancing the durability and reliability of workflows by ensuring that partial work from completed steps can be undone in a controlled manner.

6/22/2026

This post details the discovery and resolution of a subtle race condition in the hyper HTTP library that affected the Cloudflare Images binding. The bug caused intermittent truncation of image data for larger images, returning a 200 OK status without errors. The investigation involved extensive debugging, including the use of `strace` to analyze kernel syscalls, ultimately leading to a four-line code fix in hyper. This highlights the ongoing effort to ensure the reliability of core network services and their dependencies, even when dealing with complex interactions between services like Workers, internal intermediaries, and external libraries.

6/1/2026

This post details how Cloudflare significantly reduced core unit boot times from hours to minutes by addressing issues with UEFI firmware updates and network boot interface probing. It highlights the optimization of the boot sequence by declaring the correct network boot interface upfront, working with vendors to enable programmatic boot order control, and standardizing network interface strings. This directly contributes to more efficient and less disruptive process management and upgrades of critical infrastructure.

5/7/2026

This post details Cloudflare's rapid and effective response to the 'Copy Fail' Linux kernel vulnerability. It highlights their preparedness through a custom Linux kernel release process, automated patching, and advanced behavioral detection systems that identified the exploit pattern within minutes. The post also describes the development and deployment of a runtime mitigation (bpf-lsm) and the subsequent rollout of patched kernels, all while ensuring no customer impact or service disruption.

5/1/2026

This post details the completion of the 'Code Orange: Fail Small' initiative, which significantly enhances Cloudflare's infrastructure resilience. Key technical contributions include the development of Snapstone for health-mediated configuration deployments, the implementation of 'fail stale', 'fail open', and 'fail close' strategies for reducing failure impact, and the segmentation of services by customer cohorts for blast radius mitigation. It also covers the revision of 'break glass' procedures with backup authorization pathways for 18 key services and the creation of a mandatory internal Codex, enforced by AI code reviews, to codify improvements and prevent regressions.

4/22/2026

This post introduces significant advancements in the reliability of Rust Workers by detailing the implementation of WebAssembly Exception Handling for panic and abort recovery. It explains how `panic=unwind` support, enabled by the WebAssembly Exception Handling proposal, allows Rust Workers to recover from panics without losing instance state, ensuring destructors run and the WebAssembly instance remains reusable. Furthermore, it addresses abort recovery mechanisms, including the use of exception tags to distinguish between recoverable and non-recoverable errors, and introduces an experimental `--reset-state-function` for wasm-bindgen libraries to enable automatic abort recovery. This work contributes upstream to wasm-bindgen and advances Rust's Wasm support for exception handling.

3/26/2026

This post details a specific optimization for Kubernetes pod restarts by identifying and resolving a bottleneck related to `fsGroup` permission changes on large persistent volumes. The core contribution is the identification of the `fsGroupChangePolicy` setting and its application with the `OnRootMismatch` value to significantly reduce pod initialization time, thereby improving the efficiency of restarts for stateful applications like Atlantis.

2/13/2026

This post details the open-sourcing of 'ecdysis', a Rust library that implements graceful process restarts. It explains the challenges of zero-downtime upgrades for high-throughput network services, the limitations of naive approaches, and how ecdysis addresses these by using a fork-then-exec model with explicit socket inheritance. The post also covers security considerations and provides a code example for a TCP echo server using ecdysis with Tokio and systemd integration.

12/19/2025

This post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes via the Quicksilver system, mirroring existing Health Mediated Deployment (HMD) for software releases. 2. Reviewing and improving interface contracts between critical services to ensure graceful failure handling, including using validated defaults and allowing traffic to pass with passable classifications when modules fail. 3. Revising 'break glass' procedures and removing circular dependencies to enable faster access to tools during emergencies while maintaining security.

12/19/2025

This post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes, mirroring software release processes, using an enhanced Quicksilver system. 2. Reviewing and improving failure mode handling between critical services by assuming failure and implementing sane defaults. 3. Revising 'break glass' procedures and removing circular dependencies to expedite emergency response.

11/13/2025

This post details the development of an automated root cause analysis system for Salt configuration management failures. It addresses the architectural problem of job results living only on Salt Masters by building a solution that caches job results on minions. This enables faster investigation of failures by correlating them with git commits, external service failures, and ad hoc releases, ultimately reducing release delays and SRE toil.

9/26/2025

This post details the development of FL2, a new implementation of Cloudflare's core request processing logic (FL), written in Rust using the Oxy framework. It replaces the legacy Nginx/LuaJIT-based FL1 system. Key technical contributions include the use of Rust for improved safety and performance, the Oxy framework for proxy development, a strict module framework for rigid modularization of product logic, and enhanced graceful restart capabilities leveraging systemd socket activation and a new Rust-based coordination mechanism called 'shellflip'. The post also outlines the design principles of FL2's module system, emphasizing explicit interactions and a defined order of phases.

5/7/2025

Introduces `udpgrm`, a lightweight daemon for graceful UDP server restarts. Details the challenges of stateful UDP flow migration during restarts and presents a solution using Linux's `SO_REUSEPORT` API and eBPF. Explains the concepts of socket generations, working generation, and flow dissectors. Describes the `udpgrm` daemon's functionality for administrators and programmers, including its CLI for listing reuseport groups, sockets, and metrics.

5/5/2025

This post introduces Health Mediated Deployments (HMD), a data-driven system for automating software releases across Cloudflare's global network. HMD leverages Prometheus metrics queried via Thanos and Service Level Objectives (SLOs) to monitor service health in real-time, automatically reverting problematic code changes. It also details optimizations for scaling this system, including the use of recording rules and distributed query processing, and its use of R2 for historical data storage and backtesting to ensure rapid response to potential issues.

10/9/2024

This post details the development of an automatic remediation system built on Temporal to handle infrastructure failures. It introduces a coordinator for authorization and scheduling of workflows, enforces safety constraints using HCL, and leverages Temporal's Task Routing for efficient execution across different server types and locations. The system is triggered by various sources like Prometheus and alerting systems, enabling automated self-healing of platform components.

4/8/2024

This post details Cloudflare's response to a second major data center power failure in Portland, Oregon, within five months. Following the first incident, the company initiated 'Code Orange' to prioritize high availability of its control plane and analytics infrastructure. The post highlights the successful automated failover of critical services like APIs and Dashboards within minutes during the second outage, contrasting with significant downtime experienced previously. Key improvements include moving configuration databases to a Highly Available (HA) topology, pre-provisioning capacity for automatic failover, making Logpush infrastructure HA with an active failover option in Amsterdam, and enhancing the resiliency of services like Stream and Zero Trust by distributing functionality across more data centers. The time to bring a data center back online was also reduced from 72 hours to approximately 10 hours.

4/4/2024

Introduced Gradual Deployments for Workers and Durable Objects, allowing for phased rollouts of code changes. Enhanced observability for gradual deployments with version filtering in analytics and logs, and version affinity for consistent user experiences. Launched a new Rate Limiting API for Workers, enabling runtime control over rate limits. Added source mapped stack traces to Tail Workers for improved debugging. Released new API SDKs and mentioned updates to Durable Objects.

2/28/2024

This post announces the open-sourcing of Pingora, a Rust framework used internally for building network services. It details Pingora's capabilities, including support for HTTP/1, HTTP/2, TLS, TCP/UDS, customizable load balancing, and zero-downtime graceful restarts. It also provides a practical example of building a simple load balancer using Pingora's programmable APIs and discusses the framework's benefits for security-sensitive and performance-critical applications. The post also outlines future plans, contribution guidelines, and the ongoing collaboration with ISRG's Prossimo project.

5/26/2023

This post introduces Oxy, a Rust framework for building proxies, and details its extensibility through a hook-based system and dependency injection. It allows developers to programmatically modify proxy lifecycles, including connection handling, authorization logic (e.g., using Pre-Shared Keys), and response modification (e.g., adding Proxy-Status headers). This extends the concept of graceful upgrades by providing a flexible and type-safe way to customize proxy behavior at various stages of the connection lifecycle, building upon previous work with libraries like 'tableflip' for Go.

4/20/2023

This post details the implementation of graceful upgrades within the Oxy framework by splitting a large proxy process into smaller, specialized microservices (Splicer, Bumblebee, Fish). It explains how these services leverage Unix Domain Sockets and file descriptor passing (SCM_RIGHTS) to maintain state and connections across process restarts. The post also elaborates on using Tokio's JoinSet for managing concurrent tasks and implementing cancellation for graceful state transfer during upgrades.

4/4/2023

This post details the implementation of a graceful restart mechanism for Oxy, a Rust-based proxy framework. It addresses the limitations of existing solutions like 'tableflip' by leveraging systemd for socket activation and a custom-built WaitGroup equivalent using Tokio's MPSC channels. A key innovation is the use of a Unix socket for coordinating restarts, allowing for configuration validation and feedback to a restarter process, thus improving failure detection and root cause analysis.

3/31/2023

Modified NGINX's upgrade mechanism to control individual workers instead of the entire master process and its workers simultaneously. This involves disabling automatic worker restarts, shutting down one old worker, starting one new worker, and then re-enabling automatic restarts. This process is repeated for all workers, ensuring a stable memory footprint and preventing CPU core contention by carefully managing worker replacement and CPU pinning.

4/5/2022

This post details a specific incident where a missing `pipefail` shell option in a Kubernetes cron job led to corrupted configuration data being written to Quicksilver. This corruption caused the `dosd` service to fail, which in turn caused a slowdown in the Front Line service due to cache misses during code reloads. The fix involved enabling `pipefail` in the Bash script and manually re-running the cron job.

3/19/2020

This post details an optimization where Cloudflare disabled keepalives for connections between the SSL termination layer and the FL (front-end load balancer) at the edge. By forcing new connections for each request, they ensured that requests were always picked up by an idle worker, thereby reducing latency spikes. This was achieved by modifying the behavior to close connections after each request, rather than reusing them.

3/18/2020

This post details the challenges of CPU-intensive tasks within an event-loop architecture, specifically for Cloudflare's Web Application Firewall (WAF). It explores solutions such as increasing worker processes, creating separate services, and offloading to thread pools. The chosen solution involves repurposing NGINX's existing thread pool mechanism to handle WAF processing, thereby freeing up the main event loop. The post highlights the use of the EPOLLROUNDROBIN flag to address load balancing issues and presents performance metrics demonstrating reduced TTFB and improved request acceptance latency after implementing this change.

10/12/2019

This post details the challenges of managing numerous public-facing services on edge servers, each potentially needing to listen on many IP addresses. It explains the evolution of socket binding strategies from a naive one-to-one IP mapping to the more efficient INADDR_ANY wildcard. It further introduces SO_BINDTOPREFIX as a Linux kernel extension to bind wildcard sockets to specific IP prefixes, and TPROXY for scenarios requiring a single service to listen on all ports, highlighting these as crucial advancements in managing complex network service configurations on Cloudflare's edge infrastructure.

9/20/2019

This post significantly deepens the understanding of process management by focusing on the low-level network behavior of TCP sockets. It addresses the problem of 'lingering' TCP sockets, which can prevent graceful connection closure and impact service reliability. The post details the mechanics of TCP timeouts in different connection states (SYN-SENT, SYN-RECV, ESTABLISHED) and explains the crucial roles of TCP keepalives and the `TCP_USER_TIMEOUT` socket option. It provides practical examples and debugging insights, highlighting how these mechanisms ensure that idle or broken connections are eventually terminated, thereby contributing to more robust and resilient system operations, particularly relevant for services like Cloudflare Spectrum.

2/18/2019

This post introduces SOCKMAP, a Linux kernel feature that enables highly efficient TCP socket splicing directly within the kernel. It explores how SOCKMAP can be used to eliminate userspace involvement and syscall overhead for data forwarding, potentially revolutionizing the architecture of data-heavy applications like software proxies. The post details the birthing pains of L7 proxies, compares naive, splice, and io_submit approaches, and then dives into the mechanics and potential of SOCKMAP, including its eBPF integration and initial benchmark results.

11/29/2018

This post introduces the concept and implementation of SCM_RIGHTS (Send/Receive Control Message - Rights) as a mechanism to share file descriptors between processes on Linux. It details how Cloudflare leveraged this to enable a Go process to selectively handle TLS 1.3 connections while an existing nginx process handled older TLS versions on the same port, contributing a specific technical solution to the challenge of managing distinct traffic handling for upgrades or new protocol adoption without service interruption.

10/11/2018

This post introduces the 'tableflip' Go library, designed to implement NGINX-style graceful upgrades for Go applications. It details the challenges of graceful upgrades, including issues with plain `exec()`, `SO_REUSEPORT` race conditions, and the need for a controlled lifecycle. The library allows for inheriting net.Listeners, managing unused sockets, and signaling readiness to the parent process, ensuring that upgrades can occur without service interruption and with minimal risk of dropped connections or failed initializations.

10/29/2015

This post details a specific instance of a deferred deadlock caused by incorrect usage of Go's RWMutex in the RRDNS server. The bug involved a missing RUnlock call in the `getCFAddr` function, which, when combined with a pending Lock call in the table reloading goroutine, caused subsequent RLock calls to block indefinitely. This highlights a common concurrency bug and the importance of careful lock management and potential tooling for static and dynamic analysis of such issues.