TechPrufer
Get started
Baseten
Baseten logoBaseten · Feature trail

Truss Live Reload

2 posts · Dec 2022 – Feb 2023

What this trail covers

Baseten's Truss model packaging format now includes a live reload feature for development deployments, significantly reducing the development loop time for ML models from minutes to seconds. This allows for near-instantaneous updates to deployed models without requiring a full rebuild and redeploy cycle, improving developer productivity and flow state.

All Baseten trails
  1. Technical deep dive: Truss live reloadThis post details the technical design and implementation of Truss's live reload feature. It introduces a Control Proxy that runs alongside the inference server, handling patch computation and application. The mechanism relies on tracking Truss content hashes and signatures to generate patches, which are then applied via a check-and-set operation to prevent race conditions. The system is designed for simplicity, persistence, and the ability to recover from errors, aiming to provide an identical prediction interface to published production models without increasing build times. The implementation on Baseten involves the API server managing patches and syncing them to development deployment pods, with a persistent queue for patch management and compaction for efficient recovery.
  2. Accelerating model deployment: 100X faster dev loops with development deploymentsIntroduced 'development deployments' for Truss, enabling live reload of model code. This feature allows for hot-swapping code changes (e.g., pre-processing functions) without restarting the model serving environment, reducing the dev loop from minutes to seconds. Development deployments function like published deployments with their own API endpoints and special autoscaling settings for cost savings during experimentation, and can be promoted to production.