It’s All Just Wiggly Air: Building Infrastructure to Support Audio Research

This post introduces Klio, a new framework for building smarter data pipelines for audio and other media processing. Klio is built on Apache Beam for Python, aiming to provide a more ergonomic and Python-native experience. Key features include streamlined Apache Beam usage, an open graph of job dependencies, integration with cloud processing engines, containerization of custom dependencies, and support for both batch and streaming pipelines. The post highlights Klio's efficiency gains, demonstrated by a six-day catalog downsampling task that previously took weeks, and a four-fold cost reduction.