Oxy: the journey of graceful restarts

This post details the development of Oxy, a Rust-based proxy framework, and its approach to graceful restarts. It highlights the limitations of traditional upgrade processes and the challenges of long-running connections. The post explains how Oxy leverages systemd for socket management, allowing sockets to remain open during application restarts. It also describes the creation of a custom WaitGroup equivalent in Rust using Tokio's MPSC channels and a novel restart coordination mechanism via Unix sockets to improve failure detection and reporting, ensuring that configuration errors are caught early and systems do not enter inconsistent states. The technique is compared to using HTTP endpoints, emphasizing the security benefits of Unix sockets due to their permission model.