Audio-native semantic speaker-change detection | Decagon

This post introduces audio-native semantic speaker-change detection as a critical component for voice AI agents. It details the problem of distinguishing relevant speaker changes from background noise or non-interventional speech, and proposes solutions involving speaker embeddings (e.g., TitaNet) and multimodal post-training of LLMs. The post outlines benchmark construction with conversation-level false-positive rate (cFPR) and recall metrics, discusses the limitations of diarization, and explores different methodologies for using speaker embeddings (mean dissimilarity, maxsplit, graph methods). It also details synthetic dataset generation strategies and the supervised fine-tuning (SFT) objective for multimodal models, demonstrating significant improvements in recall at a fixed cFPR.