In-House LLM Serving at Netflix

What this post added

This post details Netflix's in-house LLM serving infrastructure, focusing on the technical decisions and production learnings. Key contributions include the selection of vLLM as the primary inference engine due to its extensibility and debuggability, the integration of vLLM into Triton Inference Server using the vLLM backend for dynamic I/O spec generation, and the addition of an OpenAI-compatible HTTP frontend to leverage the existing LLM ecosystem. It also elaborates on deployment strategies (Red-Black and Versioned) and their trade-offs, particularly concerning I/O schema changes and GPU cost. Operational notes cover model caching strategies and the boot sequence for vLLM-on-Triton instances.