Canopy Labs selects Baseten as preferred inference provider for Orpheus TTS models

This post details the optimization of the Orpheus TTS model for production inference on Baseten's platform, leveraging TensorRT-LLM for performance enhancements. It introduces key performance metrics for TTS models, such as simultaneous real-time streams and time-to-first-byte (TTFB), and presents benchmark results showing significant improvements (up to 3.5x more streams, <150ms TTFB). The technical details include applying FP8 quantization, chunked context, and Llama-specific CUDA optimizations via TensorRT-LLM. It also provides sample client code demonstrating efficient real-time speech synthesis calls and highlights the benefits of session re-use. The post emphasizes the cost-efficiency of using H100 MIG GPUs for Orpheus inference and the ability to scale throughput for non-real-time tasks.