TechPrufer
Get started
Netflix
Netflix logoNetflix · Feature trail

LLM Serving Infrastructure

5 posts · Apr 2022 – Jul 2026

What this trail covers

Netflix is building a comprehensive in-house infrastructure for serving Large Language Models (LLMs) and applying them to core product features. This involves developing a full stack solution that integrates model deployment, inference, and serving within their existing production environment. Key efforts include selecting and integrating performant LLM inference engines like vLLM, managing model packaging and deployment strategies, and exposing an operational interface. GenPage represents a sig. This post details the evolution of the LLM serving platform, including the selection of vLLM as the paved-path engine, the integration of vLLM into Triton using the vLLM backend for dynamic I/O spec generation, and the adoption of an OpenAI-compatible HTTP frontend alongside gRPC for ecosystem compatibility. It also discusses deployment strategies like Red-Black and Versioned, and operational notes on boot sequences and unified metrics endpoints.

All Netflix trails
  1. GenRec: Towards LLM-Native Recommendation at NetflixThis post introduces GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec verbalizes user histories, item metadata, and context into natural-language prompts, then uses a catalog-aware scoring head to rank items. It employs a two-phase training framework: Phase 1 adapts a foundation LLM on Netflix corpora, and Phase 2 post-trains it for ranking. Context engineering is used to manage the LLM's token budget. The training objectives include a catalog-aware ranking objective, a language modeling objective, and reward-weighted loss for alignment with long-term member value and business goals. GenRec runs in prefill-only mode on vLLM for cost efficiency and achieved statistically significant improvements in a large-scale A/B test.
  2. GenPage: Towards End-to-End Generative Homepage Construction at NetflixThis post introduces GenPage, a novel approach to Netflix homepage construction that leverages a single generative transformer model to autoregressively build the entire homepage. It replaces complex multi-stage recommender systems with an end-to-end model that treats user context as a prompt and generates the homepage as a sequence of tokens. Key technical contributions include a domain-specific tokenizer for computational efficiency and product control, a data representation scheme for context and page tokens, and the application of reinforcement learning for whole-page optimization. The post details trade-offs such as using handcrafted summaries for long data sources and discusses challenges in real-time serving latency, entity cold start, and enforcing business rules. GenPage has demonstrated significant user engagement gains and reduced serving latency in production.
  3. In-House LLM Serving at NetflixThis post details the evolution of Netflix's in-house LLM serving infrastructure. Key contributions include: the selection of vLLM as the paved-path inference engine due to its extensibility and debuggability; the architectural choice to use Triton's vLLM backend for dynamic I/O tensor spec generation, decoupling models from frontend upgrades, while acknowledging the Python backend as an escape hatch for custom logic; the adoption of an OpenAI-compatible HTTP frontend alongside gRPC for ecosystem compatibility and seamless experimentation-to-production paths; and the discussion of deployment strategies (Red-Black vs. Versioned) with a recommendation for Red-Black by embedding variable configurations into models. Operational notes on model caching via Amazon FSx and the boot sequence for embedded vs. standalone Triton are also provided.
  4. ML Observability: Bringing Transparency to Payments and BeyondThis post details Netflix's approach to ML observability, focusing on its application in payment processing. It introduces a framework for ML observability encompassing logging, monitoring, and explaining ML model decisions. Key contributions include defining a data schema for logging ML events, emphasizing stakeholder-focused online metrics for monitoring (e.g., traffic routing, regional performance), and detailing the use of SHAP values for model explainability to understand the 'why' behind ML decisions at both aggregate and individual levels. The post highlights the importance of transparency and trust for business stakeholders by translating technical ML performance into business outcomes.
  5. MediumThis post details the evolution of Netflix's ML Fact Store, a system designed to store and serve machine learning features for various applications. It covers the challenges of scaling the store, improving data freshness, and enabling efficient feature retrieval. The evolution includes architectural changes to handle increasing data volumes and query loads, as well as optimizations for feature computation and serving.