TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Process Management & Graceful Upgrades

31 posts · Oct 2015 – Jun 2026

What this trail covers

Cloudflare's process management and graceful upgrades have evolved with the introduction of udpgrm, a lightweight daemon that leverages Linux's SO_REUSEPORT API and eBPF for stateful flow tracking to enable zero-downtime restarts for UDP servers. This addresses the challenges of migrating flow state for stateful UDP protocols like QUIC, WireGuard, and SIP, which were previously difficult to handle gracefully. udpgrm manages the complexity of socket generation, flow dissectors, and eBPF program installation, allowing server processes to focus on their core logic.

All Cloudflare trails
  1. How we built saga rollbacks for Cloudflare WorkflowsThis post introduces saga rollbacks for Cloudflare Workflows, enabling developers to define compensation logic directly within workflow steps. This feature addresses the challenge of handling failures in multi-step processes by allowing for semantic reversal of operations, enhancing the durability and reliability of workflows by ensuring that partial work from completed steps can be undone in a controlled manner.
  2. How we found a bug in the hyper HTTP libraryThis post details the discovery and resolution of a subtle race condition in the hyper HTTP library that affected the Cloudflare Images binding. The bug caused intermittent truncation of image data for larger images, returning a 200 OK status without errors. The investigation involved extensive debugging, including the use of `strace` to analyze kernel syscalls, ultimately leading to a four-line code fix in hyper. This highlights the ongoing effort to ensure the reliability of core network services and their dependencies, even when dealing with complex interactions between services like Workers, internal intermediaries, and external libraries.
  3. How we reduced core unit boot time from hours to minutesThis post details how Cloudflare significantly reduced core unit boot times from hours to minutes by addressing issues with UEFI firmware updates and network boot interface probing. It highlights the optimization of the boot sequence by declaring the correct network boot interface upfront, working with vendors to enable programmatic boot order control, and standardizing network interface strings. This directly contributes to more efficient and less disruptive process management and upgrades of critical infrastructure.
  4. How Cloudflare responded to the “Copy Fail” Linux vulnerabilityThis post details Cloudflare's rapid and effective response to the 'Copy Fail' Linux kernel vulnerability. It highlights their preparedness through a custom Linux kernel release process, automated patching, and advanced behavioral detection systems that identified the exploit pattern within minutes. The post also describes the development and deployment of a runtime mitigation (bpf-lsm) and the subsequent rollout of patched kernels, all while ensuring no customer impact or service disruption.
  5. Code Orange: Fail Small is complete. The result is a stronger Cloudflare networkThis post details the completion of the 'Code Orange: Fail Small' initiative, which significantly enhances Cloudflare's infrastructure resilience. Key contributions include the development and adoption of Snapstone for health-mediated configuration deployments, the implementation of fail-stale/fail-open strategies and system segmentation for reducing failure impact, and the revision of incident management procedures with improved communication and backup authorization pathways. Additionally, an internal Codex has been established and enforced via AI code reviews to codify these improvements and prevent regressions.
  7. Making Rust Workers reliable: panic and abort recovery in wasm‑bindgenThis post introduces significant advancements in the reliability of Rust Workers by detailing the implementation of WebAssembly Exception Handling for panic and abort recovery. It explains how `panic=unwind` support, enabled by the WebAssembly Exception Handling proposal, allows Rust Workers to recover from panics without losing instance state, ensuring destructors run and the WebAssembly instance remains reusable. Furthermore, it addresses abort recovery mechanisms, including the use of exception tags to distinguish between recoverable and non-recoverable errors, and introduces an experimental `--reset-state-function` for wasm-bindgen libraries to enable automatic abort recovery. This work contributes upstream to wasm-bindgen and advances Rust's Wasm support for exception handling.
  8. A one-line Kubernetes fix that saved 600 hours a yearThis post details a specific optimization for Kubernetes pod restarts by identifying and resolving a bottleneck related to `fsGroup` permission changes on large persistent volumes. The core contribution is the identification of the `fsGroupChangePolicy` setting and its application with the `OnRootMismatch` value to significantly reduce pod initialization time, thereby improving the efficiency of restarts for stateful applications like Atlantis.
  9. Shedding old code with ecdysis: graceful restarts for Rust services at CloudflareCloudflare has open-sourced ecdysis, a Rust library for graceful process restarts. It enables zero-downtime upgrades by using a fork-then-exec model with inherited socket file descriptors, ensuring no live connections are dropped and no new connections are refused. The library integrates with Tokio for async support and systemd for lifecycle management. It addresses the challenges of connection refusal and termination during naive restarts by keeping sockets open throughout the transition, providing a safe initialization window for new processes and crash safety if the new process fails.
  10. Code Orange: Fail Small — our resilience plan following recent incidentsThis post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes via the Quicksilver system, mirroring existing Health Mediated Deployment (HMD) for software releases. 2. Reviewing and improving interface contracts between critical services to ensure graceful failure handling, including using validated defaults and allowing traffic to pass with passable classifications when modules fail. 3. Revising 'break glass' procedures and removing circular dependencies to enable faster access to tools during emergencies while maintaining security.
  11. Code Orange: Fail Small — our resilience plan following recent incidentsThis post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes, mirroring software release processes, using an enhanced Quicksilver system. 2. Reviewing and improving failure mode handling between critical services by assuming failure and implementing sane defaults. 3. Revising 'break glass' procedures and removing circular dependencies to expedite emergency response.
  12. Finding the grain of sand in a heap of SaltThis post details the development of an automated root cause analysis system for Salt configuration management failures. It addresses the architectural problem of job results living only on Salt Masters by building a solution that caches job results on minions. This enables faster investigation of failures by correlating them with git commits, external service failures, and ad hoc releases, ultimately reducing release delays and SRE toil.
  13. Cloudflare just got faster and more secure, powered by RustThis post details the transition from Cloudflare's legacy FL1 system (NGINX/LuaJIT) to FL2, a new implementation built in Rust using the Oxy proxy framework. Key contributions include: 1. The architectural shift to Rust and Oxy, addressing performance, safety (memory safety, data races), and maintainability issues of the previous system. 2. The implementation of graceful restarts within Oxy, allowing zero-downtime updates by keeping old instances alive to serve existing connections while new instances start. 3. The integration of systemd socket activation to decouple socket lifetimes from application processes, further improving availability during restarts. 4. The development of custom coordination mechanisms (shellflip) to replace Go libraries for restart coordination, enhancing feedback loops and automation. 5. The introduction of a strict module framework for FL2, enforcing explicit contracts between product logic modules, defining phases, inputs, and outputs to prevent IO within modules and ensure predictable execution order.
  14. QUIC restarts, slow problems: udpgrm to the rescueThis post introduces udpgrm, a new daemon designed to facilitate graceful restarts for UDP servers. It leverages Linux's SO_REUSEPORT socket option and eBPF programs for stateful flow tracking. The core mechanism involves creating 'socket generations' where new instances create a new generation while old instances maintain existing ones. udpgrm manages the eBPF program and the 'working generation' pointer to route new flows, and uses configurable 'flow dissectors' to identify established flows and their corresponding sockets, preserving flow stickiness. It also provides administrative CLI tools for inspecting reuseport groups, sockets, and metrics.
  15. Scaling with safety: Cloudflare's approach to global service health metrics and software releasesThis post introduces Health Mediated Deployments (HMD), a data-driven system for automating software releases across Cloudflare's global network. HMD leverages Prometheus metrics queried via Thanos and Service Level Objectives (SLOs) to monitor service health in real-time, automatically reverting problematic code changes. It also details optimizations for scaling this system, including the use of recording rules and distributed query processing, and its use of R2 for historical data storage and backtesting to ensure rapid response to potential issues.
  16. Improving platform resilience at Cloudflare through automationThis post details the development of an automatic remediation system built on Temporal to handle infrastructure failures. It introduces a coordinator for authorization and scheduling of workflows, enforces safety constraints using HCL, and leverages Temporal's Task Routing for efficient execution across different server types and locations. The system is triggered by various sources like Prometheus and alerting systems, enabling automated self-healing of platform components.
  17. Major data center power failure (again): Cloudflare Code Orange testedThis post details Cloudflare's response to a second major data center power failure in Portland, Oregon, within five months. Following the first incident, the company initiated 'Code Orange' to prioritize high availability of its control plane and analytics infrastructure. The post highlights the successful automated failover of critical services like APIs and Dashboards within minutes during the second outage, contrasting with significant downtime experienced previously. Key improvements include moving configuration databases to a Highly Available (HA) topology, pre-provisioning capacity for automatic failover, making Logpush infrastructure HA with an active failover option in Amsterdam, and enhancing the resiliency of services like Stream and Zero Trust by distributing functionality across more data centers. The time to bring a data center back online was also reduced from 72 hours to approximately 10 hours.
  18. New tools for production safety — Gradual deployments, Source maps, Rate Limiting, and new SDKsIntroduced Gradual Deployments for Workers and Durable Objects, allowing for phased rollouts of code changes. Enhanced observability for gradual deployments with version filtering in analytics and logs, and version affinity for consistent user experiences. Launched a new Rate Limiting API for Workers, enabling runtime control over rate limits. Added source mapped stack traces to Tail Workers for improved debugging. Released new API SDKs and mentioned updates to Durable Objects.
  19. Open sourcing Pingora: our Rust framework for building programmable network servicesThis post announces the open-sourcing of Pingora, a Rust framework used internally for building network services. It details Pingora's capabilities, including support for HTTP/1, HTTP/2, TLS, TCP/UDS, customizable load balancing, and zero-downtime graceful restarts. It also provides a practical example of building a simple load balancer using Pingora's programmable APIs and discusses the framework's benefits for security-sensitive and performance-critical applications. The post also outlines future plans, contribution guidelines, and the ongoing collaboration with ISRG's Prossimo project.
  20. How the Cloudflare global network optimizes for system reboots during low-traffic periodsThis post details the automation of determining data center maintenance windows for server reboots. Previously a manual SRE task, the process now uses sinusoidal wave fitting on historical CPU traffic data to identify low-traffic periods. The implementation uses Python with scipy.optimize.curve_fit, estimating amplitude, frequency, phase, and offset. The system caches these windows in Consul and includes fallback mechanisms for data centers with insufficient data or non-sinusoidal patterns. Goodness of fit is calculated using chi-squared to assess accuracy.
  21. How Oxy uses hooks for maximum extensibilityThis post introduces Oxy, a Rust framework for building proxies, and details its extensibility through a hook-based system and dependency injection. It allows developers to programmatically modify proxy lifecycles, including connection handling, authorization logic (e.g., using Pre-Shared Keys), and response modification (e.g., adding Proxy-Status headers). This extends the concept of graceful upgrades by providing a flexible and type-safe way to customize proxy behavior at various stages of the connection lifecycle, building upon previous work with libraries like 'tableflip' for Go.
  22. Oxy: Fish/Bumblebee/Splicer subsystems to improve reliabilityThis post details the implementation of graceful upgrades within the Oxy framework by splitting a large proxy process into smaller, specialized microservices (Splicer, Bumblebee, Fish). It explains how these services leverage Unix Domain Sockets and file descriptor passing (SCM_RIGHTS) to maintain state and connections across process restarts. The post also elaborates on using Tokio's JoinSet for managing concurrent tasks and implementing cancellation for graceful state transfer during upgrades.
  23. Oxy: the journey of graceful restartsThis post details the development of Oxy, a Rust-based proxy framework, and its approach to graceful restarts. It highlights the limitations of traditional upgrade processes and the challenges of long-running connections. The post explains how Oxy leverages systemd for socket management, allowing sockets to remain open during application restarts. It also describes the creation of a custom WaitGroup equivalent in Rust using Tokio's MPSC channels and a novel restart coordination mechanism via Unix sockets to improve failure detection and reporting, ensuring that configuration errors are caught early and systems do not enter inconsistent states. The technique is compared to using HTTP endpoints, emphasizing the security benefits of Unix sockets due to their permission model.
  24. Upgrading one of the oldest components in Cloudflare’s software stackThis post details a custom NGINX upgrade mechanism for the FL proxy that avoids doubling memory requirements during releases. The mechanism involves disabling automatic worker restarts, shutting down one worker at a time from the old instance, starting a new worker for the new instance, and then re-enabling automatic restarts. This process is repeated for all workers, ensuring that memory usage remains stable and that CPU pinning is maintained throughout the upgrade. This custom upgrade mechanism enables reliable releases of large applications like FL regardless of available memory on edge servers.
  25. PIPEFAIL: How a missing shell option slowed Cloudflare downThis post details a specific incident where a missing `pipefail` shell option in a Kubernetes cron job led to corrupted configuration data being written to Quicksilver. This corruption caused the `dosd` service to fail, which in turn caused a slowdown in the Front Line service due to cache misses during code reloads. The fix involved enabling `pipefail` in the Bash script and manually re-running the cron job.
  26. Keepalives considered harmfulThis post details an optimization where Cloudflare disabled keepalives for connections between the SSL termination layer and the FL (front-end load balancer) at the edge. By forcing new connections for each request, they ensured that requests were always picked up by an idle worker, thereby reducing latency spikes. This was achieved by modifying the behavior to close connections after each request, rather than reusing them.
  27. The problem with thread^W event loopsThis post details the challenges of event loop architectures, particularly when faced with CPU-intensive tasks like WAF processing. It explains how Cloudflare addressed this by leveraging NGINX's existing thread pool mechanism to offload these tasks, improving request latency and server accept speed. The post also touches on the importance of kernel patches like EPOLLROUNDROBIN for load balancing in such hybrid architectures.
  28. It's crowded in here!This post details the challenges of managing numerous public-facing services on edge servers, each potentially needing to listen on many IP addresses. It explains the evolution of socket binding strategies from a naive one-to-one IP mapping to the more efficient INADDR_ANY wildcard. It further introduces SO_BINDTOPREFIX as a Linux kernel extension to bind wildcard sockets to specific IP prefixes, and TPROXY for scenarios requiring a single service to listen on all ports, highlighting these as crucial advancements in managing complex network service configurations on Cloudflare's edge infrastructure.
  29. When TCP sockets refuse to dieThis post significantly deepens the understanding of process management by focusing on the low-level network behavior of TCP sockets. It addresses the problem of 'lingering' TCP sockets, which can prevent graceful connection closure and impact service reliability. The post details the mechanics of TCP timeouts in different connection states (SYN-SENT, SYN-RECV, ESTABLISHED) and explains the crucial roles of TCP keepalives and the `TCP_USER_TIMEOUT` socket option. It provides practical examples and debugging insights, highlighting how these mechanisms ensure that idle or broken connections are eventually terminated, thereby contributing to more robust and resilient system operations, particularly relevant for services like Cloudflare Spectrum.
  30. SOCKMAP - TCP splicing of the futureThis post introduces SOCKMAP, a Linux kernel feature that enables highly efficient TCP socket splicing directly within the kernel. It explores how SOCKMAP can be used to eliminate userspace involvement and syscall overhead for data forwarding, potentially revolutionizing the architecture of data-heavy applications like software proxies. The post details the birthing pains of L7 proxies, compares naive, splice, and io_submit approaches, and then dives into the mechanics and potential of SOCKMAP, including its eBPF integration and initial benchmark results.
  31. Know your SCM_RIGHTSThis post introduces the concept and implementation of SCM_RIGHTS (Send/Receive Control Message - Rights) as a mechanism to share file descriptors between processes on Linux. It details how Cloudflare leveraged this to enable a Go process to selectively handle TLS 1.3 connections while an existing nginx process handled older TLS versions on the same port, contributing a specific technical solution to the challenge of managing distinct traffic handling for upgrades or new protocol adoption without service interruption.
  32. Graceful upgrades in GoThis post details the challenges and trade-offs of implementing graceful process upgrades in Go, contrasting simple `Exec()` and `SO_REUSEPORT` approaches with the more robust NGINX model. It introduces Cloudflare's open-sourced Go library, `tableflip`, designed to replicate NGINX-style graceful upgrades by managing inherited file descriptors and ensuring a clear lifecycle for upgrade processes, preventing multiple simultaneous upgrades and ensuring old code is not retained after a successful upgrade.
  33. Creative foot-shooting with Go RWMutexThis post details a specific instance of a deferred deadlock caused by incorrect usage of Go's RWMutex in the RRDNS server. The bug involved a missing RUnlock call in the `getCFAddr` function, which, when combined with a pending Lock call in the table reloading goroutine, caused subsequent RLock calls to block indefinitely. This highlights a common concurrency bug and the importance of careful lock management and potential tooling for static and dynamic analysis of such issues.