How speech models fail where it matters the most and what to do about it

This post introduces the SF Streets and US Streets benchmarks to highlight the failure modes of state-of-the-art ASR models on named entities, particularly for non-English primary speakers, showing an average 39% transcription error rate and a 2.40-mile deviation for these speakers. It proposes and demonstrates a synthetic data generation technique called 'cross-lingual style transfer' using the XTTS model to create diverse phonetic pronunciations of English street names in foreign languages. Fine-tuning a base model on fewer than 1,000 synthetic utterances improved accuracy by 60% relative to the base model, with significant gains for non-English speakers. The post also discusses the economic impact of these errors and the release of the SF Streets and US Streets datasets for community research.