LinkedIn Feature Trails

This post details engineering techniques to optimize the training efficiency of Generative Recommenders (GRs) at LinkedIn, specifically focusing on multi-teacher distillation for AI-Powered Job Search. It addresses challenges related to skewed sequences, large vocabularies, custom attention masks, incremental training, and I/O bottlenecks. Optimizations include a C++-based fused data loader, improved attention kernels (FlashAttention-3, FlexAttention), fused metrics calculation, fused optimizer operations, and fused embedding table lookups. The system was re-architected to support distributed training across heterogeneous GPU clusters, a multi-teacher distillation system coordinating specialized models, and a caching and streaming infrastructure for rapid experimentation. This resulted in an 8X speedup for the multi-teacher distillation pipeline, reducing training time from over two days to under 5 hours. Key infrastructure improvements include scaling teacher fine-tuning with PyTorch's FSDP and FSDP2, HSDP for multi-node setups, and leveraging high-throughput inference engines with continuous batching and paged attention for teacher inference. The system now treats teachers as pluggable components, allowing for easier addition and swapping of models.

This post introduces the concept and implementation of an AI-Driven Autonomous Testing system, referred to as the 'Quality Assurance (QA) Agent'. This system aims to reimagine software quality by leveraging generative AI and Vision-Language Models (VLMs) to create autonomous agents capable of navigating and testing mobile and web applications. The agent uses a hybrid architecture combining deterministic replay (System 1) for stable flows and VLM-based planning (System 2) for dynamic scenarios. It employs a multi-model approach for planning, reasoning, and visual grounding, with evaluators acting as guardrails. The system also enables a no-code approach for creating tests by recording user interactions and synthesizing them into natural language instructions.

LinkedIn has developed the Product Configuration Center (PCC) to address the fragmentation and manual management of business configurations across its monetization ecosystem. PCC centralizes configuration management through Change Requests, SAGA-based cross-domain orchestration via Change Transport Management (CTM) and Temporal, and deterministic identifier generation. This has significantly accelerated go-to-market velocity, reducing launch cycles from weeks to hours, and improved reliability by eliminating drift and manual errors. The platform provides a unified interface for managing products, prices, and offers, and lays the groundwork for AI-assisted configuration and advanced analytics through a unified configuration graph.

LinkedIn has developed a new semantic search stack leveraging Large Language Models (LLMs) to provide a more intuitive, personalized, and predictive search experience. This involves transforming the search infrastructure to incorporate LLM-based components for query understanding, embedding-based retrieval (EBR), and cross-encoder ranking. The system aims to move beyond keyword matching to interpret natural language and infer user intent, improving relevance and efficiency at scale. Key technical advancements include the development of the MUSE (Member Understanding Semantic Embeddings) platform, which uses a dual-tower Siamese architecture with Matryoshka embeddings for efficient retrieval and ranking. The MUSE Teacher, an open-weight reasoning model, was developed to generate large-scale relevance labels by replicating an expert LLM judge, improving the quality of supervision for the embedding model. This system is applied to LinkedIn's Hiring Assistant to semantically match recruiter queries with candidate profiles.

This post details the process and challenges of internationalizing LinkedIn's Hiring Assistant, an agentic AI product, for French and German markets. It highlights the difficulties of on-the-fly text generation compared to traditional software localization, focusing on linguistic nuances like gender agreement, formal/informal address, noun capitalization, and number/date formats. The solution involves a shared rubric framework to capture language-specific rules, per-language Low-rank Adaptation (LoRA) adapters for efficient model adaptation, and an automated prompt transformation pipeline. The post emphasizes the need for explicit handling of language nuances, scaling linguist annotation for measurement, and managing the complexity of prompt evolution across multiple languages and sub-agents.

This post details the engineering effort behind building LinkedIn's Connected TV (CTV) Ads solution. It covers the core design principles, technical frameworks, and infrastructure leveraged to deliver a scalable, reliable, and measurable advertising experience. Key aspects include integrating with CTV publishers via Real-Time Bidding (RTB) through a deal-driven private marketplace, building a supply management system for onboarding and dynamically controlling PMP deals, and adapting supply prioritization algorithms and relevance models for CTV-specific dynamics. The post also addresses brand suitability by integrating with DoubleVerify for quality insights and proactively filtering problematic apps and invalid traffic. A significant engineering challenge discussed is the cross-device identity approach to associate CTV impressions with LinkedIn members for professional targeting, while respecting privacy. Furthermore, it details the technical adjustments made to meet stringent creative specifications from premium CTV publishers, including enhanced video processing pipelines, extended media asset management, and creative validation systems. Finally, it touches upon measuring impact beyond impressions with robust reporting and partnerships for independent validation.

LinkedIn has developed a unified integrations platform to standardize, reconcile, and deliver hiring data at scale. This platform supports both partner-push (BuildIn) and LinkedIn-owned pull/push (BuildOut) integration models, addressing challenges of semantic variability, entity interdependency, and operational scale. Key design principles include security, isolation, observability, unified data contracts, and idempotent, replayable, and evolvable data paths. The architecture features gateway adapters for partner-facing variability and a centralized integration core. BuildIn handles partners with less mature APIs, while BuildOut provides stronger guarantees for partners with robust APIs, leveraging Temporal for workflow orchestration and Kafka for streaming. Smart throttling is implemented for quota-aware freshness control.

This post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique values, and optimizing data serialization formats. The author also describes experimenting with sampling training data for feed recommendation algorithms, working with datasets of 500 million rows and using big data technologies like Spark and Hadoop to achieve business metric gains. Additionally, the post touches upon on-call responsibilities for the LinkedIn feed worldwide, involving immediate problem-solving for data generation pipelines to ensure member experience.

Northguard is a new log storage system designed for improved scalability and operability, replacing Kafka at LinkedIn. It features a data model based on records, segments, ranges, and topics, with log striping for load balancing. Ranges are used instead of indexed partitions to minimize interruption and facilitate stream processing. Metadata is managed by vnodes, which are Raft-backed replicated state machines.