How Baseten makes pyannote’s diarization models 9.6x faster

This post details specific optimizations applied to pyannote's speaker diarization models (Community-1 and Precision-2) to improve their inference efficiency. Key contributions include: 1. Quality-aware quantization and mixed precision (FP8, BF16, FP32, FP64) to boost throughput by 3.2x for the proprietary model with minimal DER penalty. 2. An indexing strategy for clustering high-dimensional embeddings, enabling diarization of 20 hours of audio in two minutes on a single GPU, addressing the memory explosion and curse of dimensionality for long audio. 3. Scheduling and batching optimizations, CPU-GPU memory transfer reduction, and pipeline fusion for up to 9.6x speedup in long-audio latency and 3x throughput increase. The post also highlights the trade-offs between different numerical precisions for various pipeline stages and the quadratic cost growth of naive embedding pair comparison versus an indexed clustering approach.