How to serve 10,000 fine-tuned LLMs from a single GPU

This post details the integration of LoRA swapping with TensorRT-LLM on Baseten, enabling the serving of thousands of fine-tuned LLMs from a single GPU. It explains the mechanics of LoRA, how TensorRT-LLM compiles models, and the caching strategies for LoRAs across GPU memory, host memory, disk, and network. Key technical contributions include demonstrating that LoRA swapping is compatible with in-flight batching (each request can use a different LoRA) and has minimal impact on latency (1-2ms load time from CPU memory). It also quantifies the theoretical capacity of an H100 GPU to hold over ten thousand 16MB LoRAs in its host memory.