In-House LLM Serving at Netflix

This post details the evolution of Netflix's in-house LLM serving infrastructure. Key contributions include: the selection of vLLM as the paved-path inference engine due to its extensibility and debuggability; the architectural choice to use Triton's vLLM backend for dynamic I/O tensor spec generation, decoupling models from frontend upgrades, while acknowledging the Python backend as an escape hatch for custom logic; the adoption of an OpenAI-compatible HTTP frontend alongside gRPC for ecosystem compatibility and seamless experimentation-to-production paths; and the discussion of deployment strategies (Red-Black vs. Versioned) with a recommendation for Red-Black by embedding variable configurations into models. Operational notes on model caching via Amazon FSx and the boot sequence for embedded vs. standalone Triton are also provided.