Technical deep dive: Truss live reload

This post details the technical design and implementation of Truss's live reload feature. It introduces a Control Proxy that runs alongside the inference server, handling patch computation and application. The mechanism relies on tracking Truss content hashes and signatures to generate patches, which are then applied via a check-and-set operation to prevent race conditions. The system is designed for simplicity, persistence, and the ability to recover from errors, aiming to provide an identical prediction interface to published production models without increasing build times. The implementation on Baseten involves the API server managing patches and syncing them to development deployment pods, with a persistent queue for patch management and compaction for efficient recovery.