The fastest Whisper — with streaming and diarization

This post details significant performance and cost optimizations for the Whisper transcription pipeline, achieving a 2x speed improvement (2400x RTF) and 78-98% cost reduction. It introduces real-time streaming transcription with configurable update cadences and consistent low latency. Additionally, it adds speaker diarization with a custom speaker assignment algorithm and a sliding-window approach for real-time, audio-only processing, achieving lower Diarization Error Rates (DER) and higher throughput than competitors. The pipeline is built on Baseten Chains, allowing for decoupled components and flexible GPU configurations.