TechPrufer
Get started
Replicate
Replicate logoReplicate · Feature trail

Torch Compile Caching

2 posts · Sep 2023 – Sep 2025

What this trail covers

Replicate has implemented caching for `torch.compile` artifacts to significantly reduce model boot times for PyTorch-based models. This optimization caches compiled code across model container lifecycles, leading to 2-3x faster cold starts for models like those in the FLUX family. The system keys cache files on model version and stores them near GPU nodes, updating the cache upon graceful container shutdown. This has been further extended to dramatically improve cold boot times for fine-tuned models.

All Replicate trails
  1. Torch compile caching for inference speedThis post details the implementation of caching for `torch.compile` artifacts within Replicate's infrastructure. It explains how compiled code is cached across model container lifecycles, leading to 2-3x faster cold starts for PyTorch models. The caching mechanism involves keying cache files on model version, storing them near GPU nodes, and updating the cache upon graceful container shutdown. Performance improvements are quantified with specific examples of reduced boot times for models like flux-kontext-dev, flux-schnell, and flux.1-dev-lora.
  2. FLUX fine-tunes are now fastThis post details optimizations to make FLUX fine-tunes run as fast as base models. This was achieved by quantizing LoRAs to fp8, merging weights into the base model, and automatically increasing `lora_scale` by 1.5x when `go_fast=true`. The underlying optimization builds upon previous work using `torch.compile` and fast CuDNN attention kernels. Additionally, support for loading LoRAs from external sources like Hugging Face and Civitai has been introduced.
  3. Fine-tuned models now boot in less than one secondThis post details a further optimization to the `torch.compile` caching mechanism, specifically targeting fine-tuned models. The new approach achieves sub-second cold boot times for fine-tuned language models (Llama 2 variants) and image models (SDXL). This improvement is available for new fine-tuned models created after the post date, with ongoing work to extend it to all models.