The Baseten Inference Stack at NVIDIA Dynamo Day

This post details Baseten's integration and contribution to NVIDIA Dynamo, an inference framework for scaling generative AI workloads. It highlights how Baseten uses Dynamo's disaggregated serving (prefill/decode), KV cache-aware routing, and KV cache offloading to achieve significant performance gains (e.g., 50% reduction in time to first token, 34% reduction in time per output token, 61% higher throughput). The post also touches on Dynamo's support for multimodal model serving and KV cache offloading mechanisms like NIXL. Baseten actively contributes to Dynamo and related OSS projects like TensorRT-LLM, SGLang, and vLLM, hardening them for production and sharing improvements back to the community.