The training infrastructure behind AI-Powered Job Search: 8X faster multi-teacher distillation

What this post added

This post details the engineering effort to accelerate the training infrastructure for LinkedIn's AI-Powered Job Search, specifically focusing on multi-teacher distillation. It describes the challenges of coordinating multiple specialized teacher models (relevance, engagement, embedding) with a student SLM, and the infrastructure re-architecture to address these. Key contributions include: 1. Distributed training enhancements using PyTorch's FSDP and FSDP2 with HSDP for multi-node setups, improving teacher fine-tuning throughput and hardware FLOPs utilization. 2. A multi-teacher distillation system that treats teachers as pluggable components, using high-throughput inference engines with continuous batching and paged attention for teacher inference, and overlapping teacher inference with student computation. 3. A caching and streaming infrastructure to enable rapid experimentation. These improvements led to an 8X speedup in the multi-teacher distillation pipeline, reducing training time from over two days to under 5 hours.