Multimodal Asset Personalization

3 posts · Aug 2022 – Aug 2026

What this trail covers

Netflix is evolving its personalization systems by incorporating multimodal embeddings (e.g., from CLIP) into asset representations. This allows models to understand the visual content of assets like artwork and video previews, significantly improving personalization for new titles (addressing the cold-start problem) and enabling model consolidation across different asset canvases. The system uses reward-based weighting for training data and inverse propensity scoring for offline evaluation to e

All Netflix trails

Aug 2026MAPS: Netflix’s Multimodal Asset Personalization at ScaleThis post details the integration of CLIP embeddings into Netflix's artwork personalization system to address the cold-start problem for new titles. It explains how concatenating CLIP image embeddings with learned ID embeddings allows for immediate personalization based on visual themes, talent, and color palettes, enabling knowledge transfer across titles and canvases. The post also describes model consolidation from five per-canvas models to one unified model due to CLIP's crop invariance. It introduces reward-based weighting for mixing training data across disparate canvases, optimizing for long-term member satisfaction, and discusses the use of inverse propensity scoring on exploration traffic for offline evaluation to mitigate bias from the production policy.
Aug 2022MediumThis post details the application of reinforcement learning to budget-constrained recommendations, specifically focusing on optimizing the trade-off between recommendation quality and the cost of serving those recommendations. It introduces a novel approach using a deep Q-network (DQN) to learn a policy that balances these competing objectives, aiming to maximize member engagement while staying within a defined budget. The system addresses the challenge of sparse rewards and the need for efficient exploration in a large recommendation space.
Advertisement