Blogs›Replicate›Torch Compile Caching
Replicate has implemented caching for `torch.compile` artifacts to significantly reduce model boot times for PyTorch-based models. This optimization caches compiled code across model container lifecycles, leading to 2-3x faster cold starts for models like those in the FLUX family. The system keys cache files on model version and stores them near GPU nodes, updating the cache upon graceful container shutdown. This has been further extended to dramatically improve cold boot times for fine-tuned models, reducing them to under one second by optimizing the loading and initialization process for models like Llama 2 and SDXL.
9/8/2025
Introduced caching for `torch.compile` artifacts to reduce inference boot times. This involves storing compiled code across model container lifecycles, keyed by model version and stored near GPU nodes. The system reuses cached artifacts instead of recompiling and updates the cache on graceful container shutdown. This resulted in significant cold boot time reductions for specific FLUX models.
9/6/2023
This post details a significant improvement in cold boot times for fine-tuned models on Replicate, reducing them to under one second. This is achieved by optimizing the model loading and initialization process, specifically targeting large language models like Llama 2 and image models like SDXL. The improvements are currently available for new fine-tuned models and are expected to be rolled out to all models in the future.