Replicate

Fine-tuned models now boot in less than one second

Sep 6, 2023
Fine-tuned models now boot in less than one second
SummaryAI, from the source post

This post details a further optimization to the `torch.compile` caching mechanism, specifically targeting fine-tuned models. The new approach achieves sub-second cold boot times for fine-tuned language models (Llama 2 variants) and image models (SDXL). This improvement is available for new fine-tuned models created after the post date, with ongoing work to extend it to all models.

More from the Torch Compile Caching trail