TechPrufer
Sign inGet started

Blogs›Netflix›LLM Serving Infrastructure

LLM Serving Infrastructure

LLM Serving Infrastructure

3
posts
2026

Netflix is building a comprehensive in-house infrastructure for serving Large Language Models (LLMs) and applying them to core product features. This involves developing a full stack solution that integrates model deployment, inference, and serving within their existing production environment. Key efforts include selecting and integrating performant LLM inference engines like vLLM, managing model packaging and deployment strategies, and exposing an operational interface. GenPage represents a significant step in this direction, demonstrating the application of LLMs to recommendation systems by developing GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec shows that an LLM-based ranker can match or exceed a mature production system while relying on far fewer labeled examples and input signals. It verbalizes user histories, item metadata, and context as text, post-trains a Netflix-adapted foundation LLM for ranking, adds a catalog-aware scoring head, uses reward signals to align with long-term member value and business goals, and runs in prefill-only mode on Netflix's LLM serving stack for cost efficiency.

2026

GenRec: Towards LLM-Native Recommendation at Netflix

7/30/2026

This post introduces GenRec, an LLM-backed recommendation ranker that leverages Netflix's LLM serving infrastructure. It details the two-phase training framework (Netflix-adapted foundation LLM and GenRec post-training), the conversion of interaction logs into conversational data for training, and the crucial 'context engineering' techniques used to manage prompt token budgets. It also outlines the multi-objective loss function, combining catalog-aware ranking, language modeling, and reward-weighted loss for alignment with business goals and long-term member satisfaction. The post highlights the use of vLLM in prefill-only mode for cost efficiency.

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

7/22/2026

This post introduces GenPage, a novel approach to Netflix homepage construction using a single generative transformer model. It shifts from a multi-stage pipeline to an end-to-end autoregressive generation process, treating user context as a prompt and the homepage as a tokenized sequence. Key technical contributions include custom domain-specific tokenization for computational efficiency and product control, representing both context and the structured homepage layout as tokens, and exploring whole-page optimization via reinforcement learning. The post also details challenges in production, such as serving latency, entity cold start, and enforcing business rules, and presents production impact and offline findings.

In-House LLM Serving at Netflix

7/17/2026

This post details Netflix's in-house LLM serving infrastructure, focusing on the technical decisions and production learnings. Key contributions include the selection of vLLM as the primary inference engine due to its extensibility and debuggability, the integration of vLLM into Triton Inference Server using the vLLM backend for dynamic I/O spec generation, and the addition of an OpenAI-compatible HTTP frontend to leverage the existing LLM ecosystem. It also elaborates on deployment strategies (Red-Black and Versioned) and their trade-offs, particularly concerning I/O schema changes and GPU cost. Operational notes cover model caching strategies and the boot sequence for vLLM-on-Triton instances.