The AI Technologies Behind Speech Editing at Loom

This post details the AI model architecture behind speech editing at Loom, focusing on zero-shot learning and Masked Acoustic Modeling (MAM). It explains the end-to-end workflow, including waveform to MelSpectrogram conversion, text normalization using NeMo-text-processing, and phonemization with Espeak. Challenges with accents and names are discussed, along with the use of a custom phonetic spelling lookup table. The core of the system is a universal model capable of generalizing to new voices and acoustic environments without user-specific fine-tuning, which simplifies management, reduces inference delays, and lowers costs.