Transcribe speech 100x faster and 100x cheaper with open models

This post details the implementation and optimization of a batch transcription service on Modal using NVIDIA's open ASR models (Parakeet and Canary). It compares the performance and cost against a proprietary API, demonstrating 100x faster or 100x cheaper transcription. Key technical contributions include: architecting a distributed batch processing system on Modal GPUs using the NeMo framework, optimizing data transfer and cold start times for end-to-end throughput, and comparing this to a CPU-based approach for saturating a proprietary API. The post highlights the engineering tradeoffs in balancing batching efficiency, memory packing, and time utilization for distributed workloads.