Generally Available: The fastest, most accurate and cost-efficient Whisper transcription

This post details the technical optimizations made to the Whisper model for production speech-to-text workloads, resulting in a highly performant and accurate transcription pipeline. Key contributions include: 1. Voice Activity Detection (VAD) for audio chunking to remove silences and process arbitrary lengths. 2. Utilization of Baseten Chains to create a multi-step inference pipeline with independent scaling for VAD (CPU/low-power GPU) and Whisper (powerful GPU) Chainlets. 3. Optimization of the Whisper runtime by migrating from a TensorRT-LLM Python runtime to a C++ executor for true multithreading and reduced GIL overhead. 4. Transitioning from dynamic batching to in-flight batching to immediately process requests and group them on-the-fly, significantly reducing latency. These optimizations collectively achieve over 2400x real-time factor and the lowest word error rate on benchmarks.