Driving model performance optimization: 2024 highlights

This post details Baseten's 2024 advancements in AI model performance optimization. Key contributions include adopting TensorRT-LLM as the core inference framework, integrating NVIDIA's Hopper architecture (H100, H200, GH200) with FP8 precision and MIG, and implementing speculative decoding techniques. The platform now offers featureful inference servers with guaranteed structured output, function calling, LoRA inference, and optimizations for long context. Developer tooling was enhanced with an automated Engine Builder and a custom C++ inference server. Significant performance gains were achieved across various modalities, including custom LLMs, real-time AI phone calls, and Whisper ASR. A new in-house weight distribution system was developed to reduce large model cold start times to under a minute. Future work will focus on expanding speculative decoding, optimizing embedding models, supporting new GPU architectures and quantization methods, and implementing disaggregated serving.